The jagged edges that can occur along diagonal and curved lines in geometry are caused by aliasing, or insufficient spatial sampling of the image. The process of minimizing jagged edges by increasing the sampling rate is called anti-aliasing.
Use the Anti-alias field in the Rendering Setup menu to specify an anti-aliasing factor for geometry. When the value is set to 1, no anti-aliasing occurs. Best results are obtained with a value of 4.