Create a paint mask using the cavity ambient occlusion filter

The Cavity Ambient Occlusion filter creates shaded effects in the cracks and crevices of your sculpted model. While its not currently possible to bake the occlusion effect as an image map, you can create a stencil mask that lets you paint in the crevices and cracks to recreate the effect in an image map.

Use a model that has some sculpted detail that is enhanced by the Cavity Ambient Occlusion filter. (That is, either sculpt some fine details directly onto the model, or use a stamp or stencil to create the details.)

Create an cavity ambient occlusion mask

1. Dolly, track, or tumble the view to obtain a viewpoint you can paint from.
   Note Once you establish a camera position, do not move the camera again until you finish painting with the stencil mask. Otherwise the model and stencil mask will become misaligned.
2. Turn off the grid in the 3D View (so it does not appear in the stencil mask) by selecting Display > Grid. (You can also right-click an empty area within the 3D View and turn off the Grid in the pop-up menu that appears.
3. Turn on Flat Lighting by selecting Display > Flat Lighting. (You can also right-click an empty area within the 3D View and turn on Flat Lighting from the menu that appears.)

   The model appears as a flat 2D shape without any depth shading.

4. Right-click the model and select Edit Material from the pop-up menu that appears.

   The Material window appears.

5. In the Material window, adjust the diffuse color so it is fully white.

   Adjusting the material to white ensures that the stencil mask you create will have both transparent and opaque regions.

6. Click the Viewport Filters tab and turn on the Cavity Ambient Occlusion filter for the 3D View.

   The cracks and crevices on the model appear with darker shading.

7. In the Viewport Filters window, click the Cavity Ambient Occlusion filter name to display its properties in the Properties window.
8. In the Properties window, increase the Strength property so the dark shaded regions appear even darker.
9. Set the Quality property to Best.
10. To capture an image of the 3D View do the following:
    • (Windows and Linux) Press Alt + P.
    • (Mac OS X) Press Command + Shift + 3.

    An image of the view is automatically saved to the following directory on your computer for your black and white stencil mask to the following directory:

    (Windows) <drive>:\My Documents\Mudbox\2011.5

    (Mac OS X) /Users/<user name>/Desktop

    (Linux) /home/<username>/Mudbox/2011.5

Apply the cavity ambient occlusion mask

1. To load the stencil mask into the 3D View so you can paint, switch to the Image Browser.
2. In the Image Browser, click the Open Directory icon image, and using the file browser, navigate to the directory where the image was captured and click OK.
3. In the thumbnail list, select the file named screenshot0001.png. (The name assigned to the screen capture may be different depending on which platform you are using.)
4. In the Image Browser, click the Set Stencil icon image to load screenshot0001.png as the stencil mask.

   The stencil image is assigned as a component of the active camera view, in this case, the perspective view.

   NoteIf you reposition either the camera or model in the 3D View at this point, the model and stencil mask will not align and the paint mask will not register accurately when painting. While you can manually reposition the stencil, its simpler to remember to not move either the camera or model while painting.
5. Switch to the 3D View.
6. Turn off Flat Lighting by selecting Display > Flat Lighting.
7. In the Viewport Filters window, turn off the Cavity Ambient Occlusion filter.
8. In the Paint Tools tray select the Paint Brush.
9. In the Paint properties window, edit the Color property to the color you want to paint in the crevices.
10. Using the Paint Brush, stroke across the model.

    The black regions in the stencil mask resist paint while the white regions allow paint to pass through because they are transparent.

    NoteIf you want to invert the stencil mask so that transparent areas become opaque and vice versa, click the perspective camera in the Object List window, then click the stencil node to view its properties in the Properties window. In the stencil’s Advanced properties section, turn on the Invert property.