You can use the renderNodeTypeTreeInitializeUserProc optionVar to add a custom render node or node category to the Hypershade, Assign New Material window, and Create Render Node window. See renderCreateBarUI.mel for more details.
This feature is most useful if you write a plug-in with custom nodes for rendering.
Note: This feature is not part of Maya's official API and is subject to change.