Blending the Lip-Sync Animation

To view the speech clip

• Click the Mixer tab in the Lip Sync view.

  or

• Press Ctrl+0 to open the mixer in a floating window.

If you don't see the speech clip in the mixer, select the head and click the Refresh icon in the mixer's command bar.

For general information on the mixer, see Overview of the Animation Mixer.

The Speech Blend Property Editor

1. Open the Speech Blend property editor.

   This property editor appears automatically when you generate the speech clip, but you can also open it at any time by choosing Inspect Speech Op on the Lip Sync tab in the Lip Sync view.

2. Set the following options in the Speech Blend property editor:

   

   A	Select the language in which you want to display the phonemes in the Lip Sync view: English or Japanese.
   B	Set the General > Minimum Silence and Phoneme Length for all phonemes in the Lip Sync view—see Setting the Minimum Silence and Phoneme Lengths.
   C	Enable or disable each channel on its respective tab (Lips, Jaw, Tongue)—see Activating Each Channel.
   D	Set the Weight Scale Factor for each channel on its respective tab (Lips, Jaw, Tongue)—see Weighting the Lip-Sync Animation.
   E	Set the Time Offset for each channel on its respective tab (Lips, Jaw, Tongue)—see Setting a Time Offset for Each Channel.
   F	Set the Blend and Falloff for each channel on its respective tab (Lips, Jaw, Tongue)—see Blending the Lip-Sync Animation between Phonemes.
   G	Set the Phoneme Type parameters for each channel on its respective tab (Lips, Jaw, Tongue)—see Preserving the Viseme Pose When Blending Phonemes.
   H	If you have created any corrective shapes for the visemes, you can blend their weight in using the Corrective Shapes parameters on the Lips tab—see Blending in Corrective Shapes Globally.

Setting the Minimum Silence and Phoneme Lengths

The General parameters appear on each of the three tabs, but apply generally to all three channels (lips, jaw, tongue) in the speech clip.

• The Min(imum) Silence Length is the length (in seconds) at which a gap between phonemes (silence) is included in the blending.

  If there is a silence gap that is smaller than this value, it is ignored when blending, and Face Robot instead blends into its neighboring phonemes.

  If the silence is larger than this value, it is treated like a phoneme for blending (when computing how far to blend into the neighboring phonemes). However, very long silences are not treated like very long phonemes. Blending a long silence is "capped" based on the length of the next phoneme and how much there is left to blend.

• The Min(imum) Phoneme Length is the length (in seconds) at which a phoneme is included in the blending.

  If a phoneme is smaller than this value, it is ignored when blending. This can be useful for preventing animation "pops" where the blending hits the short phonemes, but doesn't have enough time to properly ease into and out of the neighboring phonemes.

Activating Each Channel

You can toggle the Enable option on each channel's tab (Lips, Jaw, Tongue) in the Speech Blend property editor to determine whether or not the animation controls for that channel are used in the lip-sync animation. For example, if you have mocap controlling the jaw instead of the lip-sync animation, you could deactivate the Enable option for the jaw for the duration of the dialogue.

You can also animate this option to have a channel be active for only certain frames of the dialogue. For example, if you wanted to use keyframed animation on the tongue for certain phonemes, you could deactivate the Enable option for the tongue over the frames for those phonemes so that the keyframe animation takes over, then activate it again when you want the lip-sync animation to be used.

Setting a Time Offset for Each Channel

Set the Time Offset (seconds) on each tab (Lips, Jaw, Tongue) in the Speech Blend property editor to offset that channel's blend time.

For example, you can have only the lips and jaw channels lead the audio by an eighth of a second, which may generally look better than having the visemes sync exactly with the audio. Or you can have only the tongue start moving a quarter of a second before the audio.

Blending the Lip-Sync Animation between Phonemes

Sometimes you need to change the way in which the lip-sync animation is blended between phonemes, such as to increase enunciation or to slur over some of the phonemes so that the mouth movement looks more natural (most people don't enunciate every phoneme distinctly). You can do this using the Blend parameters on each tab (Lips, Jaw, Tongue) in the Speech Blend property editor. These parameters determine the number of phonemes over which each channel (lips, jaw, tongue) is blended and how smoothly the mouth eases in and out (the falloff) of its viseme pose for each phoneme.

The higher the number of phonemes used for blending, the smoother the blend between the viseme poses. However, if this value is set too high, it can make all the viseme poses look slurred. Setting the value low is useful for keeping the viseme poses well defined, such as for well-enunciated passages, but too little blending can makes the mouth snap too much between phonemes.

You can animate the Blend value so that each channel's blend changes over the duration of the lip-sync animation.

Tweaking the Blend In and Out Values Per Phoneme

Once you have set the Blend values as described in the previous section, you can tweak how the lip-sync animation blend happens going into and coming out of each phoneme. The separate Blend In and Blend Out commands give you lots of control when you need it, such as where there is a change in speech behavior within a phoneme. For example, if a commander is calling his troops to attention ("Atteeeen-tion!"), you can blend the lip-sync animation smoothly going into the phoneme, then clip it short on its way out.

The Blend In and Out values per phoneme are applied as a scale factor "on top" of the global Blend values in the Speech Blend property editor. Their result always depends on the global Blend values which are applied first. For example, if you have the global Blend Before/After value set to 2 phonemes, you could set the Blend In value to 0.5 and the Blend Out value to 2.0 so that the lip-sync animation blends in 1 phoneme before and 4 phonemes after that phoneme.

You may notice that the blend in/out changes you make per phoneme are quite subtle. This is because the phoneme weight drops quickly due to any falloff that is set first with the global Blend Falloff value.

Preserving the Viseme Pose When Blending Phonemes

If you have a high Blend Before/After value, you will notice that some of the viseme poses may look too slurred. While this is okay for some visemes, there are others whose mouth shape needs to be properly formed to be identified properly as that phoneme. These are sometimes referred to as landmark visemes. For example, for the bilabial phonemes (M, P, and B), the lips in the viseme pose must come together and compress to some degree to be identified as these phonemes, as is shown in the P viseme here.

You can preserve the mouth shape for the visemes according to which part of the mouth—lips, tongue, or jaw—is being used. To see how each viseme/phoneme is classified for preservation, open the USEnglish.csv file in your current Viseme library (for Japanese, this is the same file). Each phoneme in the library is classified according to the type of mouth controls it involves: all mouth controls, jaw+tongue, or mainly the tongue. For example, the Jaw+Tongue protected phonemes would allow the lip controls to be blended, but would ensure that the jaw and tongue controls would always form the correct shapes.