When I render in mental ray, the quality of Fur is very poor compared to the Maya software renderer.

Since mental ray is a point sampler it has to be tuned with higher sampling settings and lower contrast values to catch very fine details and thin geometry, like hairs.

Solution: Use the Production: Rapid Fur Quality Preset option in the Quality tab in Render Settings: mental ray tabs as a good starting point, then increase the Number of Samples on the Fur Feedback node; make sure that the overall sampling range is as large as the sampling setting in the fur feedback, otherwise these per object values will be clipped. Use the Production: Fine Trace Quality Preset option if you need to raytrace.