A windows procedure must have the following signature:

LRESULT CALLBACK MyWndProc(HWND, UINT, WPARAM, LPARAM);

A dialog procedure must have the following signature:

INT_PTR CALLBACK MyDlgProc(HWND, UINT, WPARAM, LPARAM);

If you need to cast your windows or dialog procedure then you should double-check the signature. If it is correct, there should be no need to cast unless you are attempting to put it into a LONG_PTR.

In x64, INT_PTR and LRESULT are essentially the same; but in Win32 with /Wp64, they are treated differently.

Along the same lines as DLSetWindowLongPtr and GetWindowLongPtr (in maxsdk\include\3dsmaxport.h), there are also DLGetWindowProc and SetWindowProc. DLGetWindowLongPtr and SetWindowLongPtr also have overloaded versions that will accept a WNDPROC properly. If you want to put a WNDPROC into GWLP_USERDATA, you can explicitly specify it, otherwise it is routed to GWLP_WNDPROC.

In versions of 3ds Max previous to version 9 there were cases where windows procedures return INT_PTR instead of LRESULT, and these were being routed quietly to GWLP_USERDATA in Win32 builds. This changed the pointer to MyClass* with a pointer to a function which is undesirable. Since 3ds Max there is an overloaded version which accepts this signature and routes it to GWLP_WNDPROC as well.