The Maya interface is built using the Qt framework. You can create custom UI for Maya using Qt Designer, which can be found at the Qt Development Tools website.
You can load your custom UI into Maya using the loadUI command.