Optimize the Maya Hotbox and marking menus on Linux

Modifications to operating system and window manager settings are recommended to optimize the performance of Maya’s hotbox and marking menus on Linux systems.

There are three display methods that the hotbox and marking menus can use: RGB Overlay, X Compositing, and Internal Software Compositing.

In general, you should try using RGB Overlay and X Compositing to see which method works best in practice. The Internal Software Compositing method is the slowest method and should be avoided if possible.

Certain limiting factors may determine which method you can use:

RGB Overlay
Use this method if you intend to use the Stereo feature in Maya. Do not use this method with AMD graphics cards.
X Compositing
Use this method with AMD graphics cards (the Linux drivers with AMD graphics do not support RGB Overlay).
Internal Software Compositing
Only use this method if both RGB Overlay and X Compositing Mode are unsuitable. This is the Maya default.

Using RGB Overlay

The RGB Overlay method uses the hardware RGB overlay to display the hotbox and marking menu. This method removes flickering that occurs with Internal Software Compositing.

To turn on RGB Overlay

Open the /etc/X11/xorg.conf file.

Replace the "CIOverlay" option (Color Index Overlay) as follows:

Section "Device"
   Option "Overlay" "on"
EndSection

OR
Section "Screen"
   Option "Overlay" "on"
EndSection

Turn off X Compositing as follows:

Section "Extensions"
    Option "Composite" "false"
EndSection

Save the xorg.conf file.

Using X Compositing

To use the X Compositing method, the X Composite Extension must be enabled, and your window manager must have compositing enabled. This method removes flickering that occurs with Internal Software Compositing.

To turn on X Compositing

NoteX Compositing is turned on by default on Red Hat 6 and Fedora 14 systems, so you do not have to modify the xorg.conf file.

Open the /etc/X11/xorg.conf file.

Turn on X Compositing as follows:

Section "Extensions"
    Option "Composite" "true"
EndSection

Save the xorg.conf file.

To turn on window manager compositing

The following table outlines how to turn on window manager compositing.

Window Manager	To turn on compositing…
KDE	Select System Settings > Desktop Effects, and under Activation, turn on Enable desktop effects at startup.
Metacity	Open the Configuration Editor, then go to Metacity > General, and turn on compositing_manager.
Others	Consult the window manager’s Help for information.

Using Internal Software Compositing

Internal Software Compositing takes a screen shot and draws the hotbox and marking menus on top of that image to simulate the transparency.

This is a fallback method and should only be used if RGB Overlay and X Compositing methods are not suitable. You may find this method is slow and is likely to cause flickering when the hotbox or marking menu disappears from the screen.

Other Recommended Settings

Focus Settings

To further improve interaction with the hotbox and marking menus in Maya, use the following focus policy settings for the window manager you are using.

Window Manager	Recommended setting
KDE	Use the Focus follows the Mouse Pointer policy. Turn off Click raises active window. For Inactive Inner Window, set Activate & Pass Click for LMB, MMB and RMB.
Metacity (with Gnome)	Set focus_mode to sloppy. Set raise_on_click to false.

Environment Variables

The following adjustments to Maya environment variables are also recommended to improve interaction with the hotbox.

In your Maya.env file, set the following:

MAYA_SET_PARENT_HINT=0
Controls the automatic parenting of windows, where setting it to 0 disables the default parenting. (The behavior is similar to Maya 2010 on Linux systems.)
MAYA_HBFreeFocus=1
When set to 1, does not force the focus back to the window that had focus prior to the hotbox appearing on screen. (This is an additional control when MAYA_SET_PARENT_HINT=0 is set on Linux systems.)
MAYA_HBFreeRaise=1
When set to 1, does not force the previously focused window to be raised when the hotbox disappears from screen. (This is an additional control when MAYA_SET_PARENT_HINT=0 is set on Linux systems.)