This article discusses some of the problems and solutions involved in exporting shaders from XSI, using custom tools and custom file formats.

THE INITIAL GOALS

At Big Bang I wanted the ability to write materials to XML files (and read them back, of course). This would serve two major purposes. First, it would offer a granular set of data, which could be re-used in different contexts. I could have used preset file instead of a XML file, but the second major purpose of the project prevented that: I wanted to be able to not only read the file, but also to make modifications in it if there was ever such a need.

To be honest I was quite disappointed that after I spent so much time and effort into this project, it turned out I needed to make such edits to material files only for about 20 of them (as of this writing, there are about 3,800 such material files in the pipeline). I could have also used the XML file at other things, like building templates for standard materials, using it for the approval of shading tasks, etc. But in the end that did not happen. However, the work I did for exporting materials could be used for other things.

THE CHALLENGES OF MATERIAL EXPORT

It didn’t take too long until I felt that exporting materials can be a rocky business. At first glance it seems very straightforward: select an object, run a script to export its material. In the material file, I wanted not only the entire material described, but I also wanted to list the image clips and the image sources used by this material. Plus, I wanted to know which scene objects are using this material, including geometry, clusters, groups and partitions. So that when I import the material, I could re-create it, as well as its image clips and sources, and apply it to objects using it if they are in the scene.

Next, I wanted the code to handle inputs that consisted of more than one object. For instance, given an input of objects (either geometry, partitions, clusters, or even materials), the code had to figure out what to export and do it right. I didn’t want the script to attempt to export the material of a chain, for example. Given that input management allowed many objects to be visited, and that some of these objects could use the same material, I had to keep track of which material had already been exported.

On top of the material, the image sources, the image clips and the objects using this material, I had to write the actual shaders to the file. Afterall the goal was to be able to re-create the material from an XML file, so I needed the list of shaders. One may think that to write shaders to a file, all that is needed is to recursively walk down the material’s Render Tree and, well, write the shaders as they are visited.

Not quite. Some shaders are connected into several other inputs. Unaware recursive traversal of the Render Tree meant that entire branches could end up being written several times. That was a waste of space and CPU cycles. Plus, if many branches are being repeated, how would the importing code know that it’s actually the same branch?

So I had to find a way to organise shaders with a unique “key”. At first I considered using their FullName. I was in for a big surprise.

In the Explorer, switch to Scene Root scope. If you iterate the Properties/LocalProperties of an object, find a material, and print its FullName, it will return the name of the material with the object’s FullName prefixed. Same if you select the material under the object. However, switch the Explorer’s scope to Materials. If you select the same material and print its FullName, then you get the name of the material prefixed with the library’s FullName, which is garanteed to be unique.

I call this last one the material’s “AbsoluteName”. There might be better terms for this, however I thought of it as browsing the file system, where there can be no two absolute file name for the same file (well, it’s possible, but let’s not get carried away).

It’s easy to work out the AbsoluteName of a material, even when you get the material from the objects owning it instead of the library. But to my knowledge, it’s impossible to use the same approach for shaders under this material. The FullName of a shader changes depending on the parameter it is being read from. For example, if you have an Image shader connected into an input of a Blinn shader and the input of an another shader, logging the Image’s FullName when traversing the Render Tree will return ..Blinn.Image and …Image.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

xsi = Application
 
# Get selected object
oSel = xsi.selection(0)
 
# Get selected object's material
oMat = oSel.material
 
def walk( oShader ):
	xsi.logmessage( oShader.fullname )
 
	# Iterate the shader parameters
	for oParameter in oShader.parameters:
 
		# Get the parameter source
		oSource = oParameter.source
 
		# Check if we got a source
		if oSource != None:
 
			# Get the classname of the source
			sClassName = xsi.classname( oSource )
 
			# Check if the classname is a shader
			if sClassName == 'Shader' or sClassName == 'Texture':
				walk( oSource )
 
# Loop over material parameters
walk( oMat )

Prints:

# INFO : sphere.Material
# INFO : sphere.Material.Blinn
# INFO : sphere.Material.Blinn.Image
# INFO : sphere.Material.Blinn.Image
# INFO : sphere.Material.Blinn
# INFO : sphere.Material.Blinn.Image
# INFO : sphere.Material.Blinn.Image
# INFO : sphere.Material.Cell
# INFO : sphere.Material.Cell.Image

So this means you can’t use the shader FullName as a unique key. If you can’t rely on the shader’s FullName, then how can you make sure that you have not visited this shader already? You could possibly use the shader’s Name. Afterall, in a material, all shaders have a unique name.

But instead of using the names as the key, I preferred to give them a unique ID number (unsigned integer). Image clips would also receive such an ID. When writing the shader parameters to the file, if they had a shader or an image clip connected, it would use the previously defined ID of the shader/image clip. This meant that before writing anything to the XML file, all shaders and image clips would need to be visited. Ultimately, a map of all shaders, based on their ID and their name, could be established.

This resulted in somewhat complex dictionaries and lists, which were hard to maintain and debug. Nonetheless, I managed to make the material export work reliably. It was slow (since it’s uncompiled code), but it was totally reliable.

I should add that handling parameters such as texture projection pointers posed a challenge worthy of talking about. Those parameters are not like other parameters, they require a different handling. Of course all that is needed is available in the SDK, still, I thought I’d mention it to those attempting at writing shaders to file.

BEYOND MATERIALS

All went smooth for many many months, until we started looking into lighting and rendering. Then it occured to me that I needed the ability to export passes, so they could be re-created later on. I thought that since I had code that could write shaders to file, then surely I could reuse this code for other things than materials: overrides and pass shader stacks. Well, that’s what I thought. But my code was built entirely around the material export, so I could hardly use it for overrides and pass stacks without making drastic modifications.

Pass shader stacks
There are two major problems with pass shader stacks.

The first problem is that from all practical purposes, passes have the appearance of materials. If you select a pass and open the Render Tree, you’ll see the pass object being displayed as if it was a material. In reality, that is not quite right.

For instance, you can iterate the parameters of material, and effectively traverse the Render Tree from there. That is not happening with passes. First, passes do not have a Parameters property, and second, they do not expose shaders in any way. So the Render Tree display is highly misleading, and I had to write code to handle passes differently than materials. In short, I did not find any quick and easy way to get the stack shaders. This is illustrated by this stripped down method that collects the stack “root” shaders (see next paragraph):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

def addpass( self, oPass ):
 
	aStacks = [
		'Environment Shader Stack',
		'Output Shader Stack',
		'Volume Shader Stack'
	]
 
	# Iterate pass stacks
	for sStack in aStacks:
 
		# Get the stack object
		oStack = oPass.nestedobjects( sStack )
 
		# Check if the stack has shaders
		if oStack.nestedobjects.count > 0:
 
			# Iterate root shaders of this stack
			for oStackShader in oStack.nestedobjects:
 
				# do something with shader

The second problem is that in a pass, there can be multiple shaders living at the top-level of a stack (which I shall call “root” shaders). For instance, if you apply two volume shaders to a pass and then open the Render Tree of that pass, you’ll only see the latest volume shader. That’s another misleading aspect of the pass’s Render Tree. So it can be assumed a pass shader stack can have multiple “root” shaders, and each of them is entirely valid and must be exported if you want to fully export the pass shader stacks. The upshot is that the pass object act as a name space, just as a material: among all the shader stacks of a pass, no two shaders can share the same name. This is great, but that doesn’t solve our problem of multiple root shaders for a same stack.

Also, when importing a pass shader stack, using the typical ways of creating shaders and plugging them afterward worked for all but the “root” shaders. To import pass shader stacks, each shader had to have a “stackroot” flag in the XML file. When meeting such a flag, the script would not simply create the shader and plug it later, it would have to run the command that creates root shaders in stacks. This command is the only way to not destroy other existing root shaders!

Overrides
Overrides pose an entirely different set of problems.

They are not materials, and they are not pass shader stacks, yet they can host complex networks of shaders. There also can be more than one override per partition! An override could not be handled as a material nor a pass. Materials, in my framework, were meant to be exported as single files, overrides needed to be exported in a pass file, along the shader stacks, partitions and render options. An override doesn’t have things like the material’s UsedBy property, it is a partition property.

Also, overrides on partitions are visible only in the current pass. I thought that this was case only in the GUI, but found out that this extends to the scripting interface. If you iterate the properties of a partition not of the current pass, you’ll never catch that partition’s overrides. The only solution was to use a low-level command such as FindObjects() to collect all overrides, which works, and then organise them by pass and by partition, to later write them to file along the partition that’s being written.

Using FindObjects() to collect overrides returned a lot more overrides that what I expected. All these overrides were named VisibilityOverrideXX (“XX” being a number). I basically found out that group objects (groups, layers, partitions) implement the parameter inheritance via an override. This override is created when a parameter of the group object is not set to “no effect on members”, but remain invisible at all times, except to commands like FindObjects(). This can be tricky, because you don’t want to export this override. However you never know when a user will create an override to disable a parameter like “viewvis”, and name this override “VisibilityOverride”. The only solution when such an override name is encountered is to check that the partition parameters all set to “no effect on members”.

But the biggest problem with overrides came with the import. This article is not about shader import (which may be covered in another article), but few things are worth mentioning.

You can’t add a parameter to an override if the parameter cannot be found in the partition members. The only way to add a parameter to an override is to use SIAddEntryToOverride(), which requires a parameter FullName string. My solution was not very elegant, and is not garanteed. Basically the code tries to find the parameter among the partition members. If find its, then good, the entry can be added. Otherwise, you’re out of luck. Materials are also tricky because the override parameter name doesn’t exactly reflect the real shader parameter FullName that has to be overridden. With elaborate string manipulation this can all be worked out, but this is slow and like I said, not very reliable.

Another annoying thing is that an entry added to an override cannot be accessed right away via the object model. At least that was the case in XSI 5.11 (I tried in 6.01 and it seems to work). For instance, the following code would raise an error:

1
2
3
4
5
6
7
8
9
10
11

xsi = Application
 
# Get the selected override
oOverride = xsi.selection(0)
 
sParameterName = 'viewvis'
sParameterFullName = 'cube.visibility.%s' % sParameterName
 
xsi.SIAddEntryToOverride( oOverride, sParameterFullName )
 
oOverride.parameters( sParameterName ).value = True

So to get back the added parameters I had to use strings.

Cameras and lights
I have not talked about cameras and lights so far, simply because as the time of this writing, the need to export their shaders never arose.

In our pipeline, cameras and lights are exported as full-fledge models. Their importance is rather low compared to other models (like characters, for instance), so that exporting their network of shaders was not felt as a necessity. Also, cameras and lights are very light-weight compared to other models, so managing them always as models posed no particular problem.

My belief is that lights can be handled not too differently than materials. The catch word is “too”. The light primitive is a bit like a material, it has inputs for connecting shaders. But a light is not a material, and I’m affraid that handling lights would require yet another dedicated wrapper to hide the real nature of the light.

In the case of camera shaders, my guess is that they behave in a similar fashion as pass shader stacks do. The camera shader stack lives right under the camera primitive, and appears to be accessible only through the NestedObjects property. Just as a pass shader stack, the camera shader stacks accepts multiple shaders at its top level, but when you open the Render Tree, it shows only the first shader of the stack.

THE SHADER NAME SPACE

Obviously, trying to adapt the existing material export code to also export pass shader stacks and overrides turned out to be very tedious. So I had to make drastic changes to the framework, and implement a new interface: the shader name space.

I consider the material, the override and the pass shader stack to be shader name spaces. That is, a space where a collection of shader names live. A material, a pass and an override as all shader name spaces. Lights and cameras are also shader name spaces. As of this writing, however, my framework does not include anything to export and import lights and cameras.

The object used as the shader name space is not necessarily the “root” shader, that is, the shader of the tree where all others ultimately plug in. Materials and overrides are both name space and root, but in the case of pass shader stacks, each “root” shader of a stack is a shader name space root. So a shader name space can have multiple roots. In the shader name space, the exact nature of the name space is not really important, as it is hidden by the shader name space interface.

Treating data that way made things far more easier to work with. Low-level functions would write to file without knowing the context of use, while wrappers would handle all the specificities of the name space.

PROPOSITION OF IMPROVEMENT FOR MATERIALS

All of this hassle with shader export and shader name space made me think about how it could have been a lot easier. I had to write code to handle all types of shader name space, which resulted in significant amounts of code and increased opportunities for bugs to make it through. The shader name space is in fact an attempt at implementing something that, I believe, should be in XSI: new kinds of materials.

Lights, passes and cameras should implement a material. I don’t see any reason why not.

- Imagine if light, pass, and camera materials could be managed from the Explorer, just like other materials. Light materials could use an orange color, while pass materials could be green and camera materials could use a gray color. The color is not important, but this mean they could a lot easier to manage, especially in large amounts. They could be organised in libraries, exported, and referenced.

- The mess of pass and camera shader stacks would not be visible anymore.

- Sharing materials among lights, passes and cameras would be a breeze. Increased reusability.

- Programming-wise, they would make the process very homogenous. No matter if you select a light, geometry, or a pass, you can always use its Material property to access the material. Simple, straightforward and consistent.

I don’t really see where overrides would fit in this design. Perhaps they could be left as they are, but at least it’s only one special case to handle, not every case.

I’m strongly against the idea of having a “super” material. For instance this material could be applied to anything, however it would expose only parameters suitable for the context it’s being used in. When applied to different kinds of objects, I hardly see how it would be easy to work with. It would make scripting very inconsistent and require the same kind of tedious handling I have been discussing in this article.

Get a material and a passes file here: Material and passes XML files

ABOUT DATA INPUT MANAGEMENT IN MATERIAL EXPORT

If you think that this section title sounds redundant, you are right. I have written an article on XSI-Blog about this very subject. I talk again about it because it really is during the development of this project that the importance of data input management struck me. I think that the data input management is one of the most fundamental aspects of any larger programming project.

I wanted the material export to be easy for the user. I wanted the interactive input to be simple and predictable. Users tend to favor the simplest ways of doing things. Given a model, they would expect to export everything related to this model. They don’t really care about the details, but they know that if all components of character lives under a unique model, then they have to be able to do things by simply handing the model to the script (via selection or picking, for example). However, exporting everything under a model can generate lots of data and take considerable time, so I wanted to give the users the ability to “focus” the export: given a selected object, it would export only the object’s material(s). Or the user could even select the material and export it. Painless, fast and predictable.

But giving the users various ways of exporting things meant that I had to write considerable code to manage the data input. The code that wrote to XML files was dumb, but lives below thick wrappers that would manage all the data input from the user. When the materials to export would reach the actual exporting functions, it would have to be guaranteed to be adequate.

Here is a little chart to show the framework design to cope with this task.

- call from the user
– user data input management (objects, materials…)
– final data input management (by type of data)
– export of shader name spaces
– fetch data from structures
– write to file

The difference between “user data input management” and “final data input management” is that they used a different interface. Both interface would handle data, but with different purposes in mind. Ultimately, they would end up populating data structures that, too, served specific puroses.

The first layer of input management handled the data provided by the user to make sure it was acceptable for the export and/or the import. For example, it ould make sure that only materials, overrides and passes could make it through for an export. The interface used to perform this task is an interface used throughout the entire framework to take care of all initial data inputs.

That interface, that I called the “Chapter” (as in a chapter in the biker world) would know everything about the context: is the user exporting or importing, is he exporting/importing in a shot or in an element, things like that. Through a system of promotion and demotion, the Chapter would store “suitable” primary data into structures that could be later accessed by everyone else.

The second layer of input management, “final data input management”, as well as “fetch data from structures”, used a more specialized interface, dedicated to shader export. Its job would be to hide all the complexities in the differences that exist between materials, overrides and shader stacks.

In “final data input management”, it would extract the final data from the “suitable” primary data coming from the Chapter, and store it in data structures. In “fetch data from structures”, it would read this final data on-demand as the script goes on. The interface would make the storing and reading of that data completely homogenous, no matter if dealing with a material, a pass, or an override. While resulting in very long code (the entire interface is implemented as nested classes), it made the overall data handling incredibly easy.

The first part of this article looks at the ideas and concepts behind XML data. The second half will show how easy it is to put it into practice in scripting and C++.

Making sure you have the latest vesion

Before we go on, you should make sure that you have the latest version of Microsoft’s XML support installed. Note that MSXML 4.0 sits on top of the previous versions of MSXML, and so it shouldn’t cause any conflicts with any other software.

To obtain the latest version, I’ve created a web page which should perform the installation quickly and easily through your web browser. Alternatively if you prefer, you can download it directly from Microsoft by visiting this web page here.

XML Basics

Without further delay, let’s look at a very simple example of XML before we examine the benefits of using it:

1
2
3
4
5

< ?xml version="1.0" encoding="utf-8" ?>
<object>
   <name>robot</name>
   <file>c:\project\models\robot.emdl</file>
</object>

The first line is the declaration of the XML file type. At this stage, it’s not worth examining any further, except to say that this is how the file identifies itself as XML.

The second line declares an opening tag with “object” as the identifier. The last line then specifies the corresponding closing tag. All tags must have an open and close tag, unless it is a special XML tag (indicated by a <?, see the first line) or if it contains no child data. If it contains no child data, then as a shortcut it can be written as ““.

The nice indented layout of the example is optional. It wouldn’t matter at all if there were no new lines or white space between the tags. You should note however, that this doesn’t mean that XML ignores the white space. Many XML parsers have by default the option to store the white space as extra child nodes of the parent node, so be warned!

The big advantage of using XML as a file format is scalability. This piece of jargon basically just means that it’s easy to expand on, without needing to have to plan for every eventuality at the outset. For example, if we later decide that we need to store information about the animation of the object, we can just add in an extra set of nodes like this:

1
2
3
4
5
6
7
8

< ?xml version="1.0" encoding="utf-8" ?>
<object>
   <name>robot</name>
   <file>c:\project\models\robot.emdl</file>
   <animation>
      <!-- ...other data... -->
   </animation>
</object>

Imagine that we had written a script with a function to parse the original node tree structure. If that old function had been written properly, it will now just ignore the new nodes. If we update the function, we can tell it to read the new nodes if they exist, or to use some predefined default values if they don’t.

We can also embed the node tree into an entirely different XML tree. For example, we might have an XML file that describes a scene:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

< ?xml version="1.0" encoding="utf-8" ?>
<scene>
   <numobjects>1</numobjects>
   <file>c:\project\scenes\robotfight.scn</file>
   <objectlist>
      <object>
         <name>robot</name>
         <file>c:\project\models\robot.emdl</file>
         <animation>
            <!-- ...other data... -->
         </animation>
      </object>
   </objectlist>
</scene>

The point to note here is that we can still use our original function to parse the node sub-tree, even though it’s located in an entirely different file format. This is why XML is so nice to use; it saves so much unnecessary work maintaining input/output (IO) routines.

Transforming XML with XSL Transforms

Since the format of XML is standardised, it is possible for any application to read the data. Of course, it doesn’t necessarily mean that the application will know what to do with it or what the information means.

For example, if you double click on any valid XML document in Windows, Internet Explorer will start up and display the document, allowing you to expand and collapse the tree nodes by clicking on them. Internet Explorer doesn’t know what the XML data means, but that doesn’t stop it from presenting it to you in a convenient manner.

The way that Internet Explorer does this is by using what is known as a XML style sheet. These are themselves XML documents, but they also use a convention known as XSL. An XSL document has the ability to transform an XML document into another text based format. This new format can be anything of your choosing, but you’ll generally want it to be another XML document.

When Internet Explorer loads your XML file, it is using a default XSL document to transform it into XHTML (an HTML format that is compliant with XML standards). By writing your own XSL documents, you can easily edit and present your XML data with minimum effort.

Here’s an example based on the previous section. The XML file below has an extra XML tag at line 2 which specifies a default XSL document to transform it. This is only a recommendation to the application loading it, and does not mean that it cannot be transformed by a different XSL document.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

< ?xml version="1.0" encoding="utf-8" ?>
< ?xml-stylesheet type="text/xsl" href="TabulateRobots.xslt"?>
<scene>
   <objectlist>
      <object>
         <name>robot</name>
         <file>c:\project\models\robot.emdl</file>
      </object>
      <object>
         <name>droid</name>
         <file>c:\project\models\droid.emdl</file>
      </object>
      <object>
         <name>mech</name>
         <file>c:\project\models\mech.emdl</file>
      </object>
   </objectlist>
</scene>

Below is an XSL transform document that will process this XML, and turn it into an XHTML document that displays the data as a table.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

< ?xml version="1.0" encoding="utf-8" ?>
<xsl :stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
</xsl><xsl :template match="/scene/objectlist">
   <html>
      <body>   
         <table border="1" width="500" cellpadding="5">      
            <tr bgcolor="#cccccc">
               <td width="200" align="center"><b>Model Name</b></td>
               <td align="center"><b>File Path</b></td>
            </tr>      
            <xsl :for-each select="object">
            <tr>
               <td><xsl :value-of select="name"/></td>
               <td><xsl :value-of select="file"/></td>
            </tr>
            </xsl>
         </table>
      </body>
   </html>
</xsl>

If you copy and paste the examples into Robot.xml and TabulateRobots.xslt, then you can double click on Robot.xml to see the result, which should look something like this:

This hopefully illustrates how easy it is to manipulate XML using XSL. In the above example, we chose to transform the data into XHTML. In fact, it’s possible to transform it into any text based format we choose by using “CDATA” tags. We could even create a script from an XML document like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

< ?xml version="1.0" encoding="utf-8" ?>
<xsl :stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl :output omit-xml-declaration = "yes"/>
   </xsl><xsl :template match="/scene/objectlist">
      < ![CDATA[Class Robot   
      Public name
      Public file
      End Class
      Sub PrintRobots
      ]]>
      < ![CDATA[Dim robotArray(]]><xsl :value-of select="count(object)"/>< ![CDATA[)
      ]]>
      </xsl><xsl :for-each select="object">
         < ![CDATA[Set curRobot =  new Robot
         curRobot.name = "]]><xsl :value-of select="name"/>< ![CDATA["
         curRobot.file = "]]><xsl :value-of select="file"/>< ![CDATA["
         Set robotArray(]]><xsl :value-of select="position()-1"/>< ![CDATA[) = curRobot
         ]]>   
      </xsl>
         < ![CDATA[For i = 0 To ]]><xsl :value-of select="count(object)-1"/>
         < ![CDATA[
         Msgbox robotArray(i).name +" : "+ robotArray(i).file
         Next
         End Sub]]>

The above XSL transform will generate the following VBScript code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

Class Robot
   Public name
   Public file
End Class 
 
Sub PrintRobots
   Dim robotArray(3)
   Set curRobot = new Robot
   curRobot.name = "robot"
   curRobot.file = "c:\project\models\robot.emdl"
   Set robotArray(0) = curRobot
   Set curRobot = new Robot
   curRobot.name = "droid"
   curRobot.file = "c:\project\models\droid.emdl"
   Set robotArray(1) = curRobot
   Set curRobot = new Robot
   curRobot.name = "mech"
   curRobot.file = "c:\project\models\mech.emdl"
   Set robotArray(2) = curRobot
   For i = 0 To 2
      Msgbox robotArray(i).name +" : "+ robotArray(i).file
   Next 
End Sub

A few things to note: Firstly, I’ve manually tidied up the formatting of the code (good luck trying to get it to look that neat straight from XSL). Generally it shouldn’t be a problem, unless of course you’re trying to generate Python code.

Secondly, it’s not hard to see that it doesn’t look like a particularly efficient way of doing things. For example, what happens if I have 12000 robots? That’d be one big VBScript!

Thirdly; great, so now we’ve got this script. How can we actually use it? Well, one easy way is to embed the script into HTML and let Internet Explorer run it. A better way though is to call it from your own scripts. That’s where the second half of this article begins.

Lastly, I should point out that generating a script from an XML document is not a particularly common thing to do. Usually, you will be generating a different XML structure, such as XHTML. This example is only to illustrate the flexability of using XML with XSL tranforms.

If you didn’t before, you should now have some sort of understanding of what XML is all about, and how it can be used to store and manipulate data. In the next section, I’ll be showing you how you can easily use it from VBScript and C++ without having to write your own XML parser.

XML and VBScript

You may be wondering why I’m using VBScript in this article. The main reason is that I experienced a few problems at work with JScript returning me the wrong interface from the COM object. I’ve since tested JScript successfully at home, and have come to the conclusion that it must be a problem with the machine at work. I still don’t know what the cause of this is, but until I do, I’m sticking with VBScript for any work with XML.

With the amount of people using XML in web applications, it shouldn’t come as any surprise that Microsoft support the use of XML in the form of COM objects (also known as ActiveX objects). This means that we can use these objects in scripts and let them do all the hard work.

Let’s go straight to an example. Save the VBScript XSL document example above as “VBSScriptRobots.xslt”, and then copy and paste the code below into a new text document and rename it “GenerateScript.vbs”.

1
2
3
4
5
6
7
8
9
10
11
12
13

set xmlDoc = CreateObject("Msxml2.DOMDocument.4.0")
xmlDoc.async=false    
xmlDoc.load "Robots.xml"
 
set xslDoc= CreateObject("Msxml2.DOMDocument.4.0")
xslDoc.async = false
xslDoc.load "VBSScriptRobots.xslt"
 
funcStr = xmlDoc.transformNode(xslDoc)
 
msgbox funcStr
execute funcStr
PrintRobots

If you now run the script, it will display the generated code in a message box, and then it will run the function and present you with three message boxes displaying the XML data records.

I’m sure you’ll still be concerned about the fact that we generated the script with the robot names and filenames hard-coded into the generated function (if not, you should be!). A much better way of going about this task, would be to use the XML document object model to iterate through the XML tree structure, like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

Set xmlDoc = CreateObject("Msxml2.DOMDocument.4.0")
xmlDoc.async = False
xmlDoc.load "Robots.xml"
 
Set sceneNode = xmlDoc.documentElement
Set objListNode = sceneNode.firstChild
Set objectListNode = objListNode.childNodes
 
For i=0 To objectListNode.length-1
   Set objectNode = objectListNode.item(i)
   set nameNode = objectNode.firstChild
   set fileNode = nameNode.nextSibling
   msgbox nameNode.text &" : "& fileNode.text
Next

This script doesn’t even need an XSL document. It simply traverses the XML node tree and iterates through each node, displaying the name and file as it goes.

When writing a function to parse an XML node, we should generally make sure it does the following:

The following code does exactly the same job as the code you’ve just seen, but it also implements the above points. While the code is much larger (and slightly slower), it is definitely more robust and is capable of withstanding most syntactical errors in the format. It also provides helpful error messages should anything go wrong by encapsulating the output data inside a resusable class that stores the error information.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99

Class SceneData
   Private Sub Class_Initialize
      Set objectsDict = CreateObject("Scripting.Dictionary")
   End Sub
   Public objectsDict
End Class
 
Class XMLDataWrapper
   Private Sub Class_Initialize
      Set dataObject = Nothing
      errNum=0
      errMsg=""
   End Sub
 
   public dataObject
 
   Public errNum
   Public errMsg
End Class
 
LoadSceneXMLDoc "Robots.xml"
 
Function LoadSceneXMLDoc(filename)
   Set xmlDoc = CreateObject("Msxml2.DOMDocument.4.0")
   xmlDoc.async = False
   xmlDoc.load filename
 
   Set errObj = xmlDoc.parseError
   If (errObj.errorCode <> 0) Then     
        Msgbox "XML Load Error:"& vbCrLf & errObj.reason
        LoadSceneXMLDoc=False
        Exit Function
   End If
 
   Set XMLDataObj = new XMLDataWrapper
   Set XMLDataObj.dataObject = new SceneData
 
   If LoadSceneNode(xmlDoc.documentElement, XMLDataObj)=True Then
      keysArr = XMLDataObj.dataObject.objectsDict.keys
      dataArr = XMLDataObj.dataObject.objectsDict.items
      num = XMLDataObj.dataObject.objectsDict.count
      For i = 0 to num-1
         Msgbox keysArr(i) &" : "& dataArr(i)
      Next
      LoadSceneXMLDoc=True
   Else
      Msgbox "XML Load Error ("&XMLDataObj.errNum&"):"& vbCrLf & XMLDataObj.errMsg
      LoadSceneXMLDoc=False
   End If
End Function
 
Function LoadSceneNode(sceneNode, XMLDataObj)
   Set objListNode = sceneNode.selectSingleNode("objectlist")   
   If Not objListNode Is Nothing Then
      LoadSceneNode = LoadObjectListNode(objListNode, XMLDataObj)
   Else
      XMLDataObj.errNum=1
      XMLDataObj.errMsg="Can't find <objectlist> node"
      LoadSceneNode = False
   End If
End Function
 
Function LoadObjectListNode(objListNode, XMLDataObj)
   Set objList = objListNode.selectNodes("object")   
   If Not objList Is Nothing Then   
      For i=0 To objList.length-1
         If LoadObjectNode(objList.item(i), XMLDataObj) Then
            LoadObjectListNode = True
         End If
      Next
      If LoadObjectListNode=False Then
         XMLDataObj.errNum=2
         XMLDataObj.errMsg="Could not load any <object> nodes correctly"   
      End If
   Else
      XMLDataObj.errNum=1
      XMLDataObj.errMsg="Can't find any </object><object> nodes"
      LoadObjectListNode = False
   End If
End Function
 
Function LoadObjectNode(objNode, XMLDataObj)
   Set nameNode = objNode.selectSingleNode("name")   
   If Not nameNode Is Nothing Then
      Set fileNode = objNode.selectSingleNode("file")   
      If Not fileNode Is Nothing Then
         XMLDataObj.dataObject.objectsDict.Add nameNode.text, fileNode.text
         LoadObjectNode=True
      Else
         XMLDataObj.errNum=1
         XMLDataObj.errMsg="Can't find <file> node for "&nameNode.text
         LoadObjectNode = False
      End If
   Else
      XMLDataObj.errNum=1
      XMLDataObj.errMsg="Can't find <name> node"
      LoadObjectNode = False
   End If
End Function</name></file></object></objectlist>

If you look at the node loading functions, you should see that they are all quite similar. This means that writing them isn’t that much extra work, since once you’ve written one, you can duplicate alot of the functionality in the others. It also gives rise to an interesting thought; why not try using XSL to automatically generate the code for your XML loader? (That’s left as an exercise to the reader!)

Following these guidelines will generally mean that XML IO routines are well behaved, expandable, and should mean minimal effort when updating their functionality.

To this end, a good idea is to use XML attributes to specify the version of a node. For example, we could rewrite the object node as follows:

1

<object version="1.0"></object>

This would mean that we could easily tell which version of the scene structure we are loading, and know in advance which child nodes are to be expected.

Finding the documentation

You may have been wondering where you can find the documentation on the various objects that we’re using. For example, how do we find out what properties and methods our XML document object has? The first step is to enter the following code:

1
2

Set xmlDoc = CreateObject("Msxml2.DOMDocument.4.0")
MsgBox(typename(xmlDoc))

You should get a message box with the word “DOMDocument” in it. If you now go to msdn.microsoft.com and do a search in the MSDN library for DOMDocument, click on the first result and you should see a list of properties, methods, and events.

The encompassing area that you need to be looking under is MSXML which can be found here. Of course, you can also find this information in the MSDN documentation that comes with any version of Visual Studio .NET.

Moving on to C++

Although it helps, you don’t really have to know that much about COM to be able to use MSXML objects in your C++ code.

The first thing to do is to use the #import statement. This pre-processor directive automatically creates and includes two header files which contain the necessary information for using the MSXML COM interfaces. We use this statement to include the msxml4.dll file which contains the necessary type information to generate the headers. This statement should be placed in a suitable header file to be included by any source files that need to use any of the MSXML interfaces.

1

#import "msxml4.dll"

Once you’ve done that, you can go ahead and create your DOMDocument object. To create the object, you do something like this:

1
2
3
4
5
6
7
8
9
10
11

MSXML2::IXMLDOMDocument2 * xmlDoc=0;
 
HRESULT hr = CoCreateInstance(__uuidof(MSXML2::DOMDocument40),
                      0,
                      CLSCTX_INPROC_SERVER,
                      __uuidof(MSXML2::IXMLDOMDocument2),
                      (void**)&xmlDoc);
if(SUCCEEDED(hr))
{
   //etc.
}

Now that we have a pointer to the IXMLDOMDocument2 interface of the DOMDocument40 object, we can then proceed to use its methods and properties.

To save yourself a lot of grief when using COM interfaces, make sure you check out the Compiler COM Support classes:

That should hopefully be enough to get you started with using XML in C++. I’ll save a more detailed XML C++tutorial for a future article.

XML (Extensible Markup Language) was originally developed for web documents. Designers can create their own tags to store custom data in an organized way to transfer information between applications using XML. Softimage started to support the standard for the ScnToc files, which store a summary of the scene elements in a XML file. With these files, production facilities e. g. are able to switch the reference models used in a scene without opening the scenefile, but by changing ASCII text instead, which is much faster, and even more importantly batchable throughout a whole sequence of scenes.

Furthermore Softimage’s XSI now supports XML as the standard for all layout/view related files. Layouts, views and the new relational views are all based of the ASCII standard. Also this feature enables large production to deal much more flexibly with their interface and generate new relational content on the fly.

dotXSI, Softimage’s own interactive media format, which supports export as ASCII or Binary compressed files, already stores a huge amount of scene-data, including geometry data, animation, camera information, lights etc….constraints, XSI specific operators (ala cluster shape combine), and Animation Mixer relevant data are not supported yet. Softimage took an amazing step by implementing the dotXSI standard, giving the users the ability to store model data in ASCII and enabling them to modify the information externaly of XSI. More than that: There are plugins for the major 3D packages to import/export dotXSI files, Softimage provides an external 3d viewer for dotXSI files, all major conversion programs support dotXSI by now!

Sounding like the ultimate solution for 3D data storage, dotXSI became more and more popular. From a non-linear pipeline point of view though, dotXSI, as well as the binary EMDL and SCN file solution are only partially applicable.

Working with static data such as EMDLs of dotXSI files, the only way to transport information from one model to another one is copying. Practically that means: Open a scene, import two models, copy the UVs from one model to the next, as an example. Considering the size of nowadays productions, the amount of rigs/models they are dealing with and the fact that schedules become tighter and even tighter, the process of copying is simply too slow and inefficient. dotXSI is a great feature of the non-linear package XSI, though the standard itself is totally linear.

That raises the question for alternatives… Here’s the idea (which is already in use here at Omation): Using XML for custom storage of scene/model information, which enables the users to reuse the data without importing the whole dataset.

Description of a cube in XML 1.0 (IE 5.0 or higher)
Description of a camera in XML 1.0 (IE 5.0 or higher)
Description of a light in XML 1.0 (IE 5.0 or higher)

Example:

XML description of a “texture-asset” including Geometry, Clusters, UVs, Rendertrees, Expressions. There could be custom tools which read the whole dataset and generate a model with geometry, applied UVs, materials etc. There can also be a custom tool which only reads the UV information, one which only reads cluster information and applies this information to an existing model in the scene, or even better, a XML description of a model, so you never have to open an EMDL file to actually apply these changes. Furthermore, all these processes would be batchable, so e.g. to get the current UVs on a bunch of models is not a problem, and can be executed by a CRON job on a server on a schedule-basis. The advantage over dotXSI/ScnToc is the non-linearity in a pipeline point of view. Models/Assets/Scenes can be broken up in discreet units for different departments/people to work on, and can be merged together on a per-unit level basis. So e.g. as soon as the texture part of the model is ready all related models get the most current UVs/materials.

This way of dealing with data changes the process of “copying” to a process of “applying” changes in an incremental way. And this is possible on a per model/per sequence/per show basis.