You can use the renderNodeTypeTreeInitializeUserProc optionVar to add a custom render node or node category to the Hypershade, Assign New Material window, and Create Render Node window. See renderCreateBarUI.mel for more details.

This feature is most useful if you write a plug-in with custom nodes for rendering.