The function used for interpolation between keys depends on the type of controller. In general, 3ds Max uses cubic (polynomial of degree three) splines for interpolation. Rotations are done using spherical linear interpolation (slerps). Registered developers who are interested in the exact details of keyframe interpolation have the source code 3ds Max uses internally as part of the Debug SDK. The bezier interpolation code is available in <MaxDebugRoot>\Core\INTERP.CPP and the TCB interpolation is in <MaxDebugRoot>\Core\TCBINTRP.CPP. This code is not available for non-registered developers. See the Advanced Topics section on Debugging for more details on the Debug SDK.

Several developers have tried to use the IKeyControl interface and then operate on the keys to get and interpolate them in 'world space'. This is really a problematic thing to try do because the keys are interdependent. For instance, consider scale keys. In world space scale keys are dependent on the position and rotation of the controller as well as the parent transformation. This is because first an object is scaled, then it is rotated, then its position is applied, then the parent transformation is applied. So the scale is rotated. If you scale an object along its X axis and then rotate it the scale becomes rotated. The scale is no longer rotated about the world X axis, its rotated about some other axis. This is normally what you want, i.e. you don't want the object to rotate through the scaled spaced (this would cause it to skew). So to talk about the scale in world space is a strange concept.

Further, the interpolation of keys that are in world space is equally problematic. If keys are put into world space, then interpolated in world space, the resulting animation will be very different that what would be seen inside 3ds Max. For example, consider an object that is rotating 360 degrees about its local Z axis and is tilted at some arbitrary angle. If you interpolate the way 3ds Max does, all the interpolation is done in the local coordinate system. Thus the object would rotate about its local Z axis (which is tilted). However if you were to try to interpolate about in world space the rotation would occur about the world Z axis. The object will still end up in the same place, but it will be a very different animation than what 3ds Max would have.

The bottom line is that developer need to understand the way 3ds Max computes its node transformation based on the keyframes and use the information to interpolate between keys accordingly.