While regular displacement maps specify the amount of displacement as a single height value, vector displacement maps specify displacement as a vector with XYZ components. This means that they can control not just the amount of displacement but also the direction. This makes it possible to create overhangs and more complex shapes.
A vector displacement map applied to a grid to create an ear
Vector displacement maps are typically created using sculpting software such as Autodesk Mudbox™ or other programs. When you import an FBX file with a VDM, either manually or using the Send To commands, the Vector Displacement Map shader gets applied in the render tree.
For more information about using Softimage and Mudbox, see Interoperability with Mudbox.