The Render Manager > mental ray (Global Renderer)
The Render Manager > (expand a pass) > mental ray
Set the current pass and choose Render > Render > Renderer Options from the Render toolbar.
Tab: Render Type [Properties Reference]
Options: Memory Limit (mental ray)
If you are having trouble getting a very large scene to render, there are a number of parameters that you can adjust to increase scalability and ensure that mental ray does not run out of memory.
The general approach to adjusting these parameters, is not to increase rendering speed, but to ensure that extremely demanding scenes do in fact render. This is more art than science, and you'll likely need to do a fair amount of tweaking to get a satisfactory result.
To optimize a large scene for scalability
Reduce the Mesh Splitting Factor value. This breaks large objects into smaller pieces that mental ray can more easily load in memory. For more information about the mesh splitting factor, see Mesh Splitting.
Change the Tile Order to Hilbert. Hilbert tiling order is more efficient than the Spiral tiling order, particularly in cases where a large mesh's triangle density is unevenly distributed.
Reduce the Tile Size. A smaller tile size means that less of the scene needs to be loaded into memory to render a given tile. This is another optimization that won't make or break the render, but may help to slightly reduce render times.
You can make all of the above changes simultaneously by clicking the Optimize button in the For Heavy Scenes section. This is often a good starting point, after which you can continue to tweak individual parameters until you can get the scene to render.