Texture Baking

by Florian Fernandez

1. Introduction
2. Map Options in Mudbox
3. General Description of Map Format
4. Map Extraction Process
5. Extraction Settings and Recommendations
6. Working with Float Maps
7. Common 32 Bit Extraction Problems
8. Maya Render Setup
9. Rendering Float Maps in Renderman for Maya
10. Rendering Float Maps in MentalRay for Maya

1. Introduction

This tutorial will:

• give you a quick start on the displacement map extraction process in Mudbox using the Texture Baking tool.
• show you some possible options to render your 32 bit displacement maps in Mental Ray and Renderman for Maya.

I tried to keep it short and simple but giving enough background information at the same time. Hopefully this little insight in the Mudbox extraction workflow and philosophy helps to get you good maps in little time.

This is written for Mudbox Beta 1213 and will be updated with upcoming releases and features.

2. Map Options in Mudbox

Displacement map extraction in Mudbox comes in three different flavors. These are 8 bit, 16 bit and 32 bit floating point precision maps. On the following pages I’ll shortly outline the different characteristics of each format and how it’s used in Mudbox.

3. General Description of Map Formats

Here is an overall comparison of 8, 16 and 32 bit maps. I’m using the meshDisplace feature to apply the maps on actual highRes geometry. You will notice that the artifacts in 8bit and 16bit map look almost the same, even though 16bit should look quite a bit better. I think that’s a current Mudbox bug

3. Map Extraction Process

The extraction process is handled as a raytracing operation between the lowRes and the highRes surface. The difference between these two arbitrary meshes is written in the UV texture space of the lowRes mesh. Therefore the extracted texture map contains the difference between these two surfaces.

At render time the values of the displacement map will deform the low resolution Mesh along it’s faces normal direction according to the height information in the map. The rendered result should look just like the highRes surface the extraction was taken from.

In Mudbox you have explicit control on the behavior of the lowRes and the highRes surface at extraction time. This is crucial for a professional result and to get good maps under different circumstances.

Mudbox allows you to extract maps between poly and subdivision surfaces in any combination. For best results the lowRes and highRes will be treated as subd surface at extraction time. The resulting maps will line up when your renderer applies the Catmull-Clak subd scheme to your lowRes polygon at render time.

4. Extraction Settings and Recommendations

The Texture Baking tool presents you with options for the subdivision of the lowRes and highRes mesh, extraction settings and map format. I will briefly go over the most important setting for 32 bit floating point map extraction.

Hovering over each input field or menu will pop up a short explanation of each operators function.

Low Resolution Mesh

In the Subdivision menu you set the subdivision behavior of the lowRes mesh at extraction time.

For displacement maps you normally want the lowRes to be treated as a subd surface. But in real life you don’t really need to do the calculations fully on the limit surface. Bringing the lowRes very close to the limit surface will give you the same result but save a lot of render time and memory requirement . Here is how the different setting affect the lowRes mesh.

In production the Poor setting will give excellent results. Only in extreme cases with very low resolution base meshes you might need to switch to the Good setting. Normally the resulting map between Poor and Good mode is pretty much identical.

Here is a 100% close up on a 2k map extracted in the different lowRes smoothing modes. I applied an edgeDetect filter in Shake to highlight any visible faces in the maps. They all look absolutely identical.

Now I apply a difference filter to highlight the differences in pixel values between all the maps. This is another close up on the upper right corner, as I could find a slight difference in the 8 pixel bleeding part when using full mode. It actually looks like an artifact to me and could be introduced by the splitting of the mesh at extraction time to reach the full-mode quality. This bug is submitted to the Team

Conclusion : Save render time and memory and go with poor mode.

High Resolution Mesh

Here you simply link the extraction tool to your highRes mesh. This can either be in the scene or on disk. If you work in the high polycount realms, it’s a good idea to save out the highest level on disk as *.bio (binary *.obj file, the improved, streaming *.obj format of Mudbox).

Sometimes it’s impossible to display a really highRes mesh due to memory limitations, saving it out on disk overcomes this problem.

Map Format

Set your preferred map dimensions and bit depth. I’m 2k resolution and 32 bit float in my example.

The edge bleed value sets the bleeding part of the map. Within those pixels you will extract information of the neighbor tiles as additional bleeding information outside the UV shells.

The bleeding and the filling part of the map are important at render time. Most renderes like Prman sample bigger chunks of texture when the object is further away from the camera to get faster and more efficient results. The bleed area will assure that even bigger sampling won’t introduce seams between your individual UV tiles.

Right now Mudbox will not fill your map to the texture borders. This will be updated in a future release.

Ray

The Ray Range is the distance the ray is allowed to travel, to look for detail between the two meshes. It will only travel as far as you allow it to do.

If you make the range too small you run the risk to cut off detail or to extract empty, black maps as the ray couldn’t find any detail within the given ray range. Keep the ray range close enough to the mesh so you keep the extraction more efficient.

Always choose a ray range higher then the you actually need so you don’t cut off any detail (see picture bellow).

The range is really important when you use 8bit or 16 bit maps. Due to the normalized nature of these two formats, both are dependent on the ray range as displacement height in the final render.

As there is no real measurement tool or function in Mudbox right now, you can use the brush size or brush magnitude manipulators to get an idea about your scene size.
Both of these values are listed in world space units so they can give you rough starting point for the maximal distance between low and highRes mesh.

The Smooth Level defines how the highRes get’s treated at render time. Use high on this setting for the best result.

Overview of the displacement map extraction settings I’m using :

5. Working with Float Maps

To clean up or manipulate your float maps you need a tool that’s able to process 32 bit images. In future versions, Mudbox will allow you to do so, but in the meantime you have to use a compositing application or a 32 bit painter of your choice to paint or mask out any smaller problems in the map. Make sure your pipeline can handle 32 bit float images. Most compositing and render applications can handle float images but sometimes it can be tricky to work with them.

On the render side of things, using worldSpace units makes it very easy to control the displacement height at render time compared to normalized maps. The shader displacement height is normally set to 1 and it will read the height information right from the pixel values of the map. There is no guessing involved about the height of your extracted displacement maps and it will look just like your highRes sculpt. As the maps are in worldSpace units, scaling your object will break this relationship.

If you have to scale your object from the initial extraction size, you need to multiply the height by your scale factor. For example, if you lowRes got scaled by 3 after extracting the maps, you need to change the displacement height from 1 to 3 as well.

6. Common 32 Bit Extraction Problems

Here is a little troubleshooting list for common extraction problems. I’m sure there is more to come soon.

7. Maya Render Setup

The current Mudbox .tif format is not recognized by most applications. This is a matter of change but for now you need to save out the maps in a different .tif format from an application like Shake or the free HDRShop.

I’m not using any of Maya’s normal or feature based displacement functions as it can’t provide a decent render quality for displacements. That’s why I will outline two possible ways of rendering the maps in Mental Ray or Renderman for Maya on the following pages.

Let’s start with some general shader setup needed for both ways.

In Maya, create a surface shader of you choice and attach a displacement shader to it.

One of many ways to do so is by stepping up one node to the shading group and to connect a displacement texture.

Next, click on the file input icon of the displacement node to attach a file texture.

Create a file node and make sure it’s set to normal. This will assure the mesh’s UV information will be used as texture coordinates.

In the file node, load your displacement texture.

At this point Maya will give you a warning. Don’t worry, it’s just complaining that it can’t read the 32 bit file (the render can read it and that’s all that matters). For the same reason the Texture Sample icon stays black as Maya can’t create a preview.

Assign the surface shader to the lowRes mesh.

8. Rendering Float Maps in Renderman for Maya

Renderman is great for rendering subdivision surfaces and displacements. At render time everything gets converted to so called mircopolygons, which means no face of your geometry is bigger then one pixel.

This makes every small detail in your maps show up plus it’s extremely fast !

First off you need to add a subd attribute to the lowRes poly surface so it renders as subdivision surface at render time. Select the lowRes, open the Attribute Editor and assign Subdiv Scheme from Attributes - RenderMan.

Now you need to add another Renderman specific Attribute to the displacement node of the shader.

In Hypershade, select your displacement node and add the Renderman Displacement Attribute from the Attributes list.

This will add Displacement Bound controls in the Extra Attributes section of the shader.

Adjust the bounding box size so it’s slightly bigger then the highest point in your displacement map to keep memory requirements as low as possible.

These are worldSpace units again and you’ll notice black artifacts if you run outside your bounding box.

As a little helper, RenderMan will print out a warning in the command line if your displacement exceeds the bound value.

Renderman reads the 32 bit float pixel value height information so you don’t need to adjust the displacement height in alpha gain or offset. Alpha Gain stays at a default 1and offset at default 0.

Alpha Gain acts as a multiplier for the displacement height. If your model got scaled you can enter the scale factor to adjust the maps accordingly.

By default the sampling rate is very low (high number). For best render results with really crisp and accurate displacements you should lower the sampling rate to 1 or lower.

The lower you go the more expensive your render gets.

Final Render : Shading Rate 0.5; 8 seconds.

9. Rendering Float Maps in MentalRay for Maya

Ok, now let’s have a look on the Mental Ray displacement rendering.

Different to Renderman‘s micropolygon technology, Mental Ray uses a special kind of tessellation called fine approximation to optimize the mesh for displacement rendering.

This function will break your mesh up into independent sub-objects before it’s tessellated. Basically it creates an optimized mesh that uses less memory instead of doing it on the whole mesh. This allows you to render high poly counts.

Final rendering can take a long time so sometimes it’s a good idea to apply high frequency details as additional bump map.

Subdivision Approximation

To convert the poly mesh to subdivision surface at render time, you need to add a Subdivision Approximation to the geometry. In Maya go to Windows> Rendering Editors> Mental Ray> Approximation Editor. Select your lowRes mesh and create a Subdivision Approximation. Having the geo selected when you create the Approximation will assign in directly to the mesh.

Click the Edit button to tune your Subdivision Approx.

Mental Ray give you a couple of options how to handle the subdivision of your lowRes at render time.

I’m using the Spatial setting but feel free to experiment with different settings on the Subdivision Approximation. Parametric give you very similar results and can be faster.

Supposedly Fine Approximation only works in Spatial Mode in Maya 7.

The Length Attribute is relating to pixels on screen if View in On, otherwise it’s in object space.

No triangle in the final render will be longer then the entered value in pixel.

These settings are a good starting point.

Min and Max is the amount a triangle can be subdivided in the final image. The higher the number, the finer the detail will be but also the more Mental Ray has to tessellate so the render gets more expensive. Have a careful eye on the memory usage !

Turn off Mays’s feature displacement on the Shape Node so it won’t be picked up at render time.

The displacement value that’s applied to the surface comes from the color balance alpha gain.

Mental Ray reads the 32 bit float pixel value height information so you don’t need to adjust the displacement height in alpha gain or offset. Alpha Gain stays at a default 1and offset at default 0.

Alpha Gain acts as a multiplier for the displacement height. If your model got scaled you can enter the scale factor to adjust the maps accordingly.

Final Render : Parametric Mode on 7; 48 seconds.

Just like Maya’s .bot and Renderman’s .tex format, Mental Rays works much more efficient using it’s own .map file format (memory mapped textures).It will also display the texture preview icon when using .map files.

Working on big scenes with many texture calls or high resolution maps that cause memory problems, it’s a good idea to convert to .map.

Use the imf_copy utility that comes with Maya. To perform the convertion use this syntax in your systems command shell:

imf_copy -p inputFile.tif outputFile.map

Ok, so this got a little bit longer than expected. I hope you found a bit of useful information in these pages.

Thanks for reading!