The Maya interface is built using the Qt framework. You can create custom UI for Maya using Qt Designer. For Windows and Linux users, Qt Designer is installed with Maya. For Mac OS X users, you can find Qt Designer directly at the Qt Development Tools website; or, you can build Qt from source. Maya 2013 uses a version of Qt 4.7.1 that has been modified by Autodesk to include some changes needed by the Maya UI, and you can find this modified version at http://www.autodesk.com/lgplsource. For detailed instructions, see Configuring Qt for use in Maya plug-ins in the Setting up your build environment chapter of the API Guide.
You can load your custom UI into Maya using the loadUI command.