In Softimage 2010, a new lip sync feature was added to Face Robot (here’s the doc).  In 2011 SP1, there are three new commands to get the data out into your own tools.  One of them doesn’t even require the use of Face Robot and calls the speech recognition library directly.

The commands return VBScript-compatible multi-dimensional arrays.    This is transparent in VBScript and Python, but it requires a bit of magic in JScript, so check the example in the first command if that’s what you’re using.

Getting Phonemes without Face Robot

SIExtractPhonemes

Description

Scripting Syntax

oReturn = SIExtractPhonemes( Language, FileName, GuideText );

Return Value

Parameters

Examples

//returns a VBArray object, which require using the .getItem method in Jscript because they are not
//native objects there.
//However, they work transparently like multi-dimentionnal arrays in VBscript and Python
var res = SIExtractPhonemes ("en", "s:\\here_be_dragons.wav", "Here be dragons, the map says");
arraysize = res.ubound( 1 );
logmessage( arraysize + "phonemes found" );
for (  i=0; i < arraysize; i++)
{
logmessage( "IPA#" + res.getItem(i,0)
         + "(" + res.getItem(i,3) + ") "
         + "Start Time(sec) " + res.getItem(i,1)
         + "End Time(sec) " + res.getItem(i,2) );
}

Getting Lip Sync Stuff Out of Face Robot

This allows the artists to tweak the phonemes with the Face Robot UI, but used the data outside of Face Robot, perhaps in a game engine that does phoneme blending already.

The second command even allows you to get the result of the blend algorithm of Face Robot, so you can make a simplified playback environment.

GetPhonemeFromSpeechClip

Description

Scripting Syntax

oReturn = GetPhonemeFromSpeechClip( SpeechClip );

Return Value

Parameters

Examples

phonemes = GetPhonemesFromSpeechClip ("Face.Mixer.Phoneme_Speech.SpeechAction_Clip")
LogMessage "phoneme data"
for i = 0 to UBound( phonemes )
	LogMessage "IPA # " & phonemes(i, 0) & "(" & phonemes(i, 3) & ")" & " Start (sec)" & phonemes(i, 1) & " End (sec)" & phonemes(i, 2)
next

GetPhonemeBlendFromSpeechClip

Description

Scripting Syntax

oReturn = GetPhonemeBlendFromSpeechClip( SpeechClip, [Channel], [Time] );

Return Value

Parameters

Examples

phonemes = GetPhonemeBlendFromSpeechClip ("Face.Mixer.Phoneme_Speech.SpeechAction_Clip")
LogMessage "phoneme blend"
for i = 0 to UBound( phonemes )
	LogMessage "IPA # " & phonemes(i, 0) & "(" & phonemes(i, 1) & ")" & " Weight " & phonemes(i, 2 )
next

Well, a colleague of mine, Etienne Pellerin, and myself are starting to work in C++ in XSI so the Jitter tool was rewritten.

New in Jitter v1.5:

• Installation is smoother – just drop it into your favourite Application/Plugins folder
• Performance is increased when you have large numbers of Jitter Ops
• You can now apply the operator to parameter groups such as kine.local.ori.euler

You can download the new version and beat the snot out of it if you wish. I can’t guarantee it’ll survive though, and if it doesn’t drop me a line and I can fix bugs.

I only have a windows version but if someone has a Linux box and would care to recompile I can host both versions.

Cheers,
Patrick

Initially, figuring out how this could work seems like a pretty tricky problem. The bone chain needs to stretch intuitively to give the impression that it’s stuck to the IK effector. But we also need it to flatten gradually to avoid the snap effect. Unfortunately, the only thing we have available to us to control these two effects, is the ability to change the length of the bones, since the end position of the chain must always be located at the effector.

So we need to scale the bones to allow them to stretch, but at a specific rate to make sure we avoid the snap. Like this:

The first thing to do is to look at how we solved the problem for the non-stretchy bone chain. When we look at what happens when we move the Soft Effector past the point where it starts to leave the original IK effector behind, all we want to do for the stretchy bone chain system is make sure that the bone chain gets longer so that the original IK effector is at the same position as the Soft Effector. But we need to do this while also keeping the bones angles the same, this avoids the snap effect since the bone angles would change at exactly the same rate as in the non-stretchy bone chain system.

A diagram should make things a bit clearer. Basically, given a non-stretchy chain, we want to match the position of the two effectors by scaling the chain length, but without changing the bone angles, like this:

Okay, so that doesn’t seem too hard. We just need to figure out how much to move the IK Effector to match the position of the soft effector.

The only way we can do this is by changing the length of the bone chain. Again, this initially looks like quite a tricky problem, until you realize that scaling the length of each bone by some value X, also scales the distance of the original IK effector from the IK root by a factor of X as well. A simple test proves it (and it can be easily shown with a little math too):

This diagram shows a bone chain with bones of different lengths, before and after I scaled each individual bone length by a factor of 2. It can be seen that the IK effector has traveled directly away from the IK root and has doubled it’s distance from it. Not only that, but the bone angles have remained unchanged. This is exactly what we need!

Now, all we need to do is to figure out the amount we need to scale each bone length to make the original IK effector match the position of the soft effector. That scale factor is just the ratio of the distances of the two effectors:

Scale Factor = EvalDistance(IK root, Soft Effector) / EvalDistance(IK root, original IK effector)

Where EvalDistance() is just a function that measure the distance between the two objects.

Finally, we need to position constrain the original IK effector to the soft effector, so that it follows it. Otherwise, all that would happen is that the length of the chain would change.

So that’s it, we’re there. All that remains is to package it all up into a plugin with a handy custom property to drive it all. This is what the new property page looks like:

The bone lengths can be adjusted (and even animated) from this PPG, as well as the Soft Distance parameter, which behaves in exactly the same way as in the non-stretchy Soft IK. The bone lengths are all driven from this PPG using expressions. Note, that for extra performance we only have one Scripted Operator here which operates on the BoneScale parameter of the PPG.

I’ve updated the original Soft Effector script to include a new menu item to apply the Stretchy Bone Soft IK system to any chain you select.

Soft Effector Download (3Kb)

Incidentally, if you ever want to remove the Soft IK (stretchy or otherwise) from a bone chain. All you have to do is delete the Soft Effector object. All the expressions and operators will be automatically deleted since they will have lost their connections. Nice and tidy!

P.S. Thanks to Daniele Niero for suggesting I cache the math objects inside the operator user data!

In XSI, we need a way of allowing the chain to gradually fall behind the position of the effector. This stops the effector from snapping the chain into the final position, and gives rise to a much more natural motion (as seen below).

(Note: If you are unable to view the GIF animation, you may need to change settings in your Firewall. For example, in Zone Alarm, you need to disable Privacy->Ad Blocking)

The snap effect visible in the top chain isn’t due to some inaccuracy in the IK calculation, but is just down to a simple geometrical effect. It can be shown mathematically (see diagram below) that as the effector pulls the chain into the final position, the velocity of the bones towards their final position tends towards infinity! This is obviously not desirable and can make the animation look jerky and artificial.


(click to enlarge)

To achieve the type of motion shown in the lower chain, we need to add an extra object which is used to drive the IK chain effector. A scripted operator links the two objects and we use a mathematical expression to give the desired result. The only bits of information the operator needs are the total chain length, and something I call the “Soft Distance”. This represents the distance from the chain’s full extension that the effect starts to work.

There is one downside with this approach, as it means you can’t fix the end point of the chain in position. However, this can be solved as long as you are willing to put a small amount of extra work to animate the Soft Distance parameter.

The soft effector object has a custom property to allow you to easily control the operator.

To make things easier for you, the Chain Length is calculated by an expression which sums the length of each bone in the chain. This allows you to change the length of a bone, and still have the Soft IK work.

In the diagram below, the Soft Distance is represented by the distance between the green and the red lines (actually, they’re circles, but too large to see). As soon as the effector reaches the green line, our scripted operator starts to lag the chain effector behind our soft effector. This is done while making sure that we have no jerkiness in the motion of the chain effector.

(Note: If you are unable to view the GIF animation, you may need to change settings in your Firewall. For example, in Zone Alarm, you need to disable Privacy->Ad Blocking)

The equation used to create this effect is shown below:

(click to enlarge)

For a chain length of 3 and a soft distance of 1, this equation looks something like this:

(click to enlarge)

You can use the following link to download the Soft Effector plugin:

Soft Effector Download (3 Kb)

(Note that this also features the Stretchy Bone Chain extension shown in Part 2 of this article)

To install the plugin, just download the zip file, unzip the script, and copy it to a “Plugins” folder. So for example, you can use your “C:\users\username\Softimage\XSI\Application\Plugins” folder, or one in your workgroup.

To use the Soft Effector, create a 2D or 3D Skeleton Chain, select any part of the chain, and go to “Model->Create->Skeleton->Apply Soft Effector”.

I created an object, and extracted a curve from the boundary edges. Of course, you can always go the other route and create the curve first, and then afterwards create the mesh off the curve.

What I now want to do is roll the object along this curve, like a footroll. Here’s a picture and a movie for further explanation:

The movie is encoded using Camtasia (Techsmith Codec) and is around 3 MB small / large… ;-) Click on this link to get it.

The code is fairly simple, and I tried to comment / document as much as possible.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132

// get the transform of the control object
var inXf = Inlocal1.value.transform;
 
// get the control object's rotation
var inRot = XSIMath.CreateRotation();
inXf.GetRotation(inRot);
 
// get the value of the operator's roll parameter
var roll = In_UpdateContext.parameters("roll").value;
 
// get the value of the operator's distance parameter
var distance = In_UpdateContext.parameters("distance").value;
 
// set a vector which is going to start out as the y axis
var yAxis = XSIMath.CreateVector3();
yAxis.Set(0,1,0);
 
// multiply y by the rotation, so it points along the
// control object's y axis
yAxis.MulByRotationInPlace(inRot);
 
// create the output transform of the curve
var xf = XSIMath.CreateTransform();
 
// if there is any rotation (if y doesn't point straight up)
if(yAxis.x!=0||yAxis.z!=0)
{
	// store the height of y temporarily
	var yAxisHeight = yAxis.y;
 
	// remove the y part of yAxis
	yAxis.y = 0;
 
	// normalize it (puts its length to 1)
	yAxis.NormalizeInPlace();
 
	// multiply the length of y by the distance param
	yAxis.ScaleInPlace(distance );
 
	// get the curve's geometry
	var geo = Incrvlist.value.Geometry;
 
	// get the closest curve position to the resulting
	// position of the yAxis (on the ground plane)
	var vbArgs = new VBArray(geo.GetClosestCurvePosition2(yAxis));
	var args = vbArgs.toArray();
 
	// change the length of y relatively to the distance
	// of the closest point on the curve
	// this removes the unwanted jittery behaviour
	yAxis.NormalizeInPlace();
	yAxis.ScaleInPlace(args[3].length());
 
	// now get the closest position again
	var vbArgs = new VBArray(geo.GetClosestCurvePosition2(yAxis));
	var args = vbArgs.toArray();
 
	// get the U value of the curve for the closest point
	var curveU = args[2];
 
	// now evaluate the curve for the given U
	// note: this is using the hard coded subcurve index
	// 0, so it never works for multi-curve-curvelists
	var vbArgs = new VBArray(geo.curves(0).EvaluatePosition(curveU));
	var args = vbArgs.toArray();
 
	// get the position and tangent
	var curvePos = args[0];
	var curveTan = args[1];
 
	// calculate the angle between the former Y axis and
	// the ground plane
	var angle = Math.acos(yAxisHeight)/2;
 
	// negate (turn around) the yAxis
	// as we want to move the object position
	// relatively to the position on the curve
	curvePos.NegateInPlace();
 
	// define a quaternion for the rotation
	var quat = XSIMath.CreateQuaternion();
 
	// the rotation for the tilt is a rotation around
	// the tangent of the curve on the given position
	// with an angle defined by the overall rotation
	// of the control object
	quat.Set(
		Math.cos(angle),
		Math.sin(angle) * curveTan.x,
		Math.sin(angle) * curveTan.y,
		Math.sin(angle) * curveTan.z
	);
 
	// create a rotation for the tilt
	var rotTilt = XSIMath.CreateRotation();
	rotTilt .SetFromQuaternion(quat);
 
	// now we rotation the position of the curveobject
	// around the tangent of the curve, by using
	// our rotTilt rotation
	var centerPos = XSIMath.CreateVector3();
	centerPos.MulByRotation(curvePos,rotTilt);
 
	// define an additional rotation for the roll	
	var rotRoll = XSIMath.CreateRotation();
	// define an eulerangles vector for the rotation
	var rotAngles = XSIMath.CreateVector3();
	// set the y rotation by the given roll param
	rotAngles.y = XSIMath.DegreesToRadians(roll);
	// set the rotations euler angles by the vector
	rotRoll .SetFromXYZAngles(rotAngles);
	// rotate the position of the curveobject again
	// by the rotRoll rotation
	centerPos.MulByRotationInPlace(rotRoll);
 
	// for the output transform, create a new rotation
	// and multiply both rotations together
	var rot = XSIMath.CreateRotation();
	rot.Mul(rotTilt,rotRoll);
 
	// now substract the curvePosition off the center
	// position, to put it back in its own space
	centerPos.SubInPlace(curvePos);
 
	// for the output transform, set the
	// translation and rotation
	xf.SetTranslation(centerPos);
	xf.SetRotation(rot);
}
 
// output the transform
Out.value.transform = xf;

If you want to checkout the scene, here it is (XSI 5.1).
You can find the scripted operator below the local kinematics of the curve. Enjoy!

1. Shift+Click and drag to create a region around the keys on the timeline
2. Shift+Double Click to select a single frame
3. Hold Middle mouse button and drag that region to move keys
4. Ctrl+Drag on that region to copy keys
5. Middle Click on the extremities of the region to scale keys
6. There are some more timeline options in the right mouse click menu
7. You can view an audio wave in the timeline if you have an audio clip in the scene.

Simply select the rows in the data grid control by left clicking on the row header on the left hand side, and shift clicking further down. Press Ctrl+C and the data will be copied to the clipboard. Open up Microsoft Excel, and you can paste this data into a worksheet!

Luckily, this copy and paste process also works in reverse (using Ctrl+V to paste back in to XSI), meaning that you can use all of Excel’’s handy sorting and filtering utilities to make sure that you don”t have duplicate bones in the list. When pasting back into the XSI template, you”ll need to make sure you”ve created the necessary number of rows in the template before you paste the data.

If you don”t have Excel, then this will also work in any standard text editor (although, you”ll need to make sure that the text editor copies the data as plain text, rather than including any font information). Each row will be pasted as a separate line with columns separated by a tab character.

The Dead Simple Keying Workflow

Every XSI user understands the basic workflow. “Mark”, or select, a parameter, and hit “K” to key it.
Actually, it’’s possible to use XSI without ever knowing anything about marking, because the basic SRT tools mark what’’s appropriate for you.
This is a good ”next gen” version of the ”Set Key” menu in SI3D, where you”d click the SaveKey button and pick in the Windows Start menu-style hierachy of eeeevvverything that’’s keyable, and then middle mouse button to repeatedly key the same things.

Behind the scene, parameters are not in fact directly “selected”. The Marking list is in fact just a list of strings, a relative path to parameters, and XSI commands will try to match these strings against the parameter names on the current object at the moment of executing a command.

Selective Keying

As you”re building your rigs, quickly you end up wanting to key translation, and only translation, on one object, and rotation on “Y” on another. Ideally you want to key everything all at once so that the rig is ”locked” at every key and there is no random behaviour caused by not having keyed everything you intented.

This is all doable with scripts and buttons that will key exactly the parameters of interests, and people do this, however XSI has services for that built-in.

Marking Sets and Setting Branch Keys (ALT+K)

Marking sets are cool and they”re also a “demo” of sorts of the architecture of XSI. Basically, it’’s a property set that’’s created which contains links to the parameters that you want to key.

In fact, there is nothing special behind Marking Sets, it’’s directly using the Custom Parameter Set commands, and creates a property set called “MarkingSet”. Then, the “Set Key Using Marking Set” commands simply looks for a custom property set with that exact name and set keys on everything.

There is a second command “Set Branch Key” [..."Using Marking Set", in XSI 4.2] that does the same thing, but searches the entier branch of selected objects for the Marking Sets. It’’s mapped to ATL+K.

These commands are all in the Animation menu that is that the bottom of the interface.

Here Comes …Keyable Parameters

In V5.0 we introduce Keyable Parameters, and this is where we”ll focus in the future. Of course, Maya users are familiar with Keyable Parameters, and the words mean the same thing as in Maya. At this point, although it can be confusing to have ”animatable parameters” and ”keyable parameters”, IMO it’’s probably best to not try to invent a new word for it, so we will use the same.

Keyable Parameters do the same job as marking sets : they allow you to select which parameters you want to key. By default, local kinestate – scale, rotate, translate – is keyable.

The Keying Panel is the panel that shows you which parameters are keyable on the current object. What you see is what is going to be keyed. The Keyable Parameter Editor allows you to choose which parameters are keyable or not.

IMHO, it’’s a little more clear in XSI what these things do, because the Keying Panel isn”t doing double-duty as a generic property panel, which it isn”t meant to be. It’’s really a Panel for Keying. In the future we”ll continue to invest into more animation workflow items in there.

We”ve also added a few script commands to get the keyable parameters.

Save Key Command Mapping

The most critical user preference setting in XSI animation is in the animation menu at the bottom of the screen, is it called Save Key Command Mapping. These are the three choices :

On Marked Parametes

This is the default XSI method, pre-keyable parameters. Hitting “K” will set keys on the parameters that are currently Marked on the currently selected objects. Hitting ALT+K with set a branch key on the Marking Sets it finds on the selected objects.

On All Keyable Parameters

This is a Maya-like behaviour and it’’s the default in QWERTY Interaction.
However, XSI users should use it as well, but obviously we couldn”t put that as the default in V5.0 without causing some confusion with current users.

Hitting “K” will set keys on all the keyable parameters on the selected objects. Hitting ALT+K will save a key on all keyable parameter in the selected branch, no need for Marking Sets.

On All Marked Keyable Parameters

This is a mix of both, it sets keys on the intersection of both keyable AND marked parameters.
For example, if you”re in the Translate tool, it will set a key on the position of the object, IF the parameters are keyable. This is a good mode as well for XSI users, because it means that the transform tool will key the parameters you expect, unless you removed their ”keyable” flag. (Note that you could also use Locking to protect parameters from being keyed)

Here is my take on Jitter+ for XSI: PB_Jitter! Just drop both files in the Application/Plugins folder and don’t forget to install the buzz.xsi library first.

Back in the days of Softimage|3D there was a plug-in called Jitter. It was used to create random movement on objects. My take on this idea, thanks to the XSI architecture, takes it one little step further. If it’s animateable, you can add a PB_Jitter to it. Which means you can add a PB_Jitter to almost anything. Once you add this Scripted Operator to a marked parameter (Animate -> Create -> Parameter -> Jitter Marked Parameter) you’ve basically piped a procedural animation generator into your parameter.

Here is the main tab of PB_Jitter:

This generator has two facets to it. A sine wave generator and a random noise generator.

The first part of the generator creates a sine wave. This regular motion can be toggled on or off, it’s amplitude can be animated and the base wave can be shifted left or right. A concept of octaves is also available. Each successive octave’s computed value is added to the total value for that calculation but each octave’s wavelength is half the preceding octave’s and each successive octave’s amplitude is diminished by a factor corresponding to the overall persistance. This means that the higher your persistance and the larger the number of octaves the wilder your motion, while the lower the persistance and the lower the number of octaves the smoother the motion.

The second part of the generator creates random noise. The wavelength of this noise is animateable as well as it’s amplitude. The octaves/persistance concepts show up again like in the sine wave generator. This time around the extra features you get are a choice of curve interpolation and a smoothing option. The horizontal shift this time can be used as such or it can be used as a random seed by inputing ridiculous numbers and shifting the curve left by, say, -2845037.

Another neat thing about the PB_Jitter is that if the parameter you want to jitter allready has an f-curve animation, this animation will be preserved in the offset parameter of the PB_Jitter and the computed sine or noise values will be added to the allready existing animation.

There are two bugs that I know about that are both related to the layout of a runtime scripted operator. The first is, if you plug a PB_Jitter in the offset value of a PB_Jitter (why not), the next time you want to inspect the outer PB_Jitter, the PPG layout and all it’s parameters will be gone. The second is if you save a scene that has a PB_Jitter inside it, next time you open the scene the PPG layout will be gone – though you still have access to the parameters, they’re just not nicely laid out in tabs.

Because of these little problems and the lack of documentation this is release 0.9 but hopefully I’ll be able to call it a full 1.0 in a short amount of time.

Hope you like! If you have questions, comment or verbal abuse, don’t be shy!