Two Undocumented Lip sync commands in Softimage 2011 SP1
June 21st, 2010 by Luc-Eric - Viewed 5767 times - Popularity: 13% [?]Hi there! It’s been a while..
In Softimage 2010, a new lip sync feature was added to Face Robot (here’s the doc). In 2011 SP1, there are three new commands to get the data out into your own tools. One of them doesn’t even require the use of Face Robot and calls the speech recognition library directly.
The commands return VBScript-compatible multi-dimensional arrays. This is transparent in VBScript and Python, but it requires a bit of magic in JScript, so check the example in the first command if that’s what you’re using.
Getting Phonemes without Face Robot
SIExtractPhonemes
Description
Scripting Syntax
oReturn = SIExtractPhonemes( Language, FileName, GuideText ); |
Return Value
Returns a multi-dimentional array of results. Each column will contain
1) The Unicode IPA code of the phoneme
2) The Start Time of the phoneme, in seconds
3) The End Time of the phoneme, in seconds
4) A human-readable hint of phoneme, useful for debugging
Parameters
| Parameter | Type | Description |
|---|---|---|
| Language | String | Language to use. “en” for English, or “jp” for Japanese |
| FileName | String | Path to an audio file, for example a .wav or an .avi file. Mono and 16000k is recommend. Same audio formats as Audio in Softimage; MP3 not supported. |
| GuideText | String | Text that match the spoken words in the audio file. Avoid any unusual punctuations or strange characters that may throw off the engine |
Examples
JScript Example
//returns a VBArray object, which require using the .getItem method in Jscript because they are not
//native objects there.
//However, they work transparently like multi-dimentionnal arrays in VBscript and Python
var res = SIExtractPhonemes ("en", "s:\\here_be_dragons.wav", "Here be dragons, the map says");
arraysize = res.ubound( 1 )+1;
logmessage( arraysize + "phonemes found" );
for ( i=0; i < arraysize; i++)
{
logmessage( "IPA#" + res.getItem(i,0)
+ "(" + res.getItem(i,3) + ") "
+ "Start Time(sec) " + res.getItem(i,1)
+ "End Time(sec) " + res.getItem(i,2) );
}
|
Getting Lip Sync Stuff Out of Face Robot
This allows the artists to tweak the phonemes with the Face Robot UI, but used the data outside of Face Robot, perhaps in a game engine that does phoneme blending already.
The second command even allows you to get the result of the blend algorithm of Face Robot, so you can make a simplified playback environment.
GetPhonemeFromSpeechClip
Description
Returns the data of the Face Robot speech operator, as shown on the tracks of in the Lip Sync view. The data starts at time zero, and needs to be offsetted by the offset of the Speech Clip.
Scripting Syntax
oReturn = GetPhonemeFromSpeechClip( SpeechClip ); |
Return Value
Returns a multi-dimentional array of results. Each column will contain
1) The Unicode IPA code of the phoneme
2) The Start Time of the phoneme, in seconds
3) The End Time of the phoneme, in seconds
4) A human-readable hint of phoneme, useful for debugging
5) weight for this phoneme
6) in falloff for this phoneme
7) out falloff for this phoneme
8) shape weight for this phoneme
9) shape in falloff for this phoneme
10) shape out falloff for this phoneme
11) blend start scale factor
12) blend end scale factor
13) phoneme variation
Parameters
| Parameter | Type | Description |
|---|---|---|
| SpeechClip | String | The Speech Clip from the animation mixer |
Examples
VBScript Example
phonemes = GetPhonemesFromSpeechClip ("Face.Mixer.Phoneme_Speech.SpeechAction_Clip")
LogMessage "phoneme data"
for i = 0 to UBound( phonemes )
LogMessage "IPA # " & phonemes(i, 0) & "(" & phonemes(i, 3) & ")" & " Start (sec)" & phonemes(i, 1) & " End (sec)" & phonemes(i, 2)
next
|
GetPhonemeBlendFromSpeechClip
Description
Queries the Face Robot speech operator for the list of phonemes and weights at a given time, as evaluated by its blend algorithm. At a given time there generally is the current phoneme, as shown in the lipsync view, but also some of the previous and the next one. With its different parameters, the speech operator could evaluate a different blend for the lips, jaw and tongue.
Scripting Syntax
oReturn = GetPhonemeBlendFromSpeechClip( SpeechClip, [Channel], [Time] ); |
Return Value
Returns a multi-dimentional array of results. Each column will contain
1) The Unicode IPA code of the phoneme. Value zero is the rest pose
2) A human-readable hint of phoneme, useful for debugging
3) The weight of the viseme at this time
4) The viseme variation selected by the user
Parameters
| Parameter | Type | Description |
|---|---|---|
| SpeechClip | String | The Speech Clip from the animation mixer |
| Channel | Integer | 0 for Lips, 1 for Jaw, 2 for TongueDefault Value: 1 |
| Time | Integer | Frame at which to evaluate the blendDefault Value: Current frame. |
Examples
VBScript Example
phonemes = GetPhonemeBlendFromSpeechClip ("Face.Mixer.Phoneme_Speech.SpeechAction_Clip")
LogMessage "phoneme blend"
for i = 0 to UBound( phonemes )
LogMessage "IPA # " & phonemes(i, 0) & "(" & phonemes(i, 1) & ")" & " Weight " & phonemes(i, 2 )
next
|
Popularity: 13% [?]
Wow!! thanks for that!
Thank’s for the new lip sync feature for Face Robot! It made the usage of the program much more convenient!