About exporting shaders

July 8th, 2007 by Bernard Lebel - Viewed 15908 times -




ABSTRACT

This article discusses some of the problems and solutions involved in exporting shaders from XSI, using custom tools and custom file formats.

THE INITIAL GOALS

At Big Bang I wanted the ability to write materials to XML files (and read them back, of course). This would serve two major purposes. First, it would offer a granular set of data, which could be re-used in different contexts. I could have used preset file instead of a XML file, but the second major purpose of the project prevented that: I wanted to be able to not only read the file, but also to make modifications in it if there was ever such a need.

To be honest I was quite disappointed that after I spent so much time and effort into this project, it turned out I needed to make such edits to material files only for about 20 of them (as of this writing, there are about 3,800 such material files in the pipeline). I could have also used the XML file at other things, like building templates for standard materials, using it for the approval of shading tasks, etc. But in the end that did not happen. However, the work I did for exporting materials could be used for other things.

THE CHALLENGES OF MATERIAL EXPORT

It didn’t take too long until I felt that exporting materials can be a rocky business. At first glance it seems very straightforward: select an object, run a script to export its material. In the material file, I wanted not only the entire material described, but I also wanted to list the image clips and the image sources used by this material. Plus, I wanted to know which scene objects are using this material, including geometry, clusters, groups and partitions. So that when I import the material, I could re-create it, as well as its image clips and sources, and apply it to objects using it if they are in the scene.

Next, I wanted the code to handle inputs that consisted of more than one object. For instance, given an input of objects (either geometry, partitions, clusters, or even materials), the code had to figure out what to export and do it right. I didn’t want the script to attempt to export the material of a chain, for example. Given that input management allowed many objects to be visited, and that some of these objects could use the same material, I had to keep track of which material had already been exported.

On top of the material, the image sources, the image clips and the objects using this material, I had to write the actual shaders to the file. Afterall the goal was to be able to re-create the material from an XML file, so I needed the list of shaders. One may think that to write shaders to a file, all that is needed is to recursively walk down the material’s Render Tree and, well, write the shaders as they are visited.

Not quite. Some shaders are connected into several other inputs. Unaware recursive traversal of the Render Tree meant that entire branches could end up being written several times. That was a waste of space and CPU cycles. Plus, if many branches are being repeated, how would the importing code know that it’s actually the same branch?

So I had to find a way to organise shaders with a unique “key”. At first I considered using their FullName. I was in for a big surprise.

In the Explorer, switch to Scene Root scope. If you iterate the Properties/LocalProperties of an object, find a material, and print its FullName, it will return the name of the material with the object’s FullName prefixed. Same if you select the material under the object. However, switch the Explorer’s scope to Materials. If you select the same material and print its FullName, then you get the name of the material prefixed with the library’s FullName, which is garanteed to be unique.

I call this last one the material’s “AbsoluteName”. There might be better terms for this, however I thought of it as browsing the file system, where there can be no two absolute file name for the same file (well, it’s possible, but let’s not get carried away).

It’s easy to work out the AbsoluteName of a material, even when you get the material from the objects owning it instead of the library. But to my knowledge, it’s impossible to use the same approach for shaders under this material. The FullName of a shader changes depending on the parameter it is being read from. For example, if you have an Image shader connected into an input of a Blinn shader and the input of an another shader, logging the Image’s FullName when traversing the Render Tree will return ..Blinn.Image and …Image.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
xsi = Application
 
# Get selected object
oSel = xsi.selection(0)
 
# Get selected object's material
oMat = oSel.material
 
def walk( oShader ):
	xsi.logmessage( oShader.fullname )
 
	# Iterate the shader parameters
	for oParameter in oShader.parameters:
 
		# Get the parameter source
		oSource = oParameter.source
 
		# Check if we got a source
		if oSource != None:
 
			# Get the classname of the source
			sClassName = xsi.classname( oSource )
 
			# Check if the classname is a shader
			if sClassName == 'Shader' or sClassName == 'Texture':
				walk( oSource )
 
# Loop over material parameters
walk( oMat )

Prints:

# INFO : sphere.Material
# INFO : sphere.Material.Blinn
# INFO : sphere.Material.Blinn.Image
# INFO : sphere.Material.Blinn.Image
# INFO : sphere.Material.Blinn
# INFO : sphere.Material.Blinn.Image
# INFO : sphere.Material.Blinn.Image
# INFO : sphere.Material.Cell
# INFO : sphere.Material.Cell.Image

So this means you can’t use the shader FullName as a unique key. If you can’t rely on the shader’s FullName, then how can you make sure that you have not visited this shader already? You could possibly use the shader’s Name. Afterall, in a material, all shaders have a unique name.

But instead of using the names as the key, I preferred to give them a unique ID number (unsigned integer). Image clips would also receive such an ID. When writing the shader parameters to the file, if they had a shader or an image clip connected, it would use the previously defined ID of the shader/image clip. This meant that before writing anything to the XML file, all shaders and image clips would need to be visited. Ultimately, a map of all shaders, based on their ID and their name, could be established.

This resulted in somewhat complex dictionaries and lists, which were hard to maintain and debug. Nonetheless, I managed to make the material export work reliably. It was slow (since it’s uncompiled code), but it was totally reliable.

I should add that handling parameters such as texture projection pointers posed a challenge worthy of talking about. Those parameters are not like other parameters, they require a different handling. Of course all that is needed is available in the SDK, still, I thought I’d mention it to those attempting at writing shaders to file.

BEYOND MATERIALS

All went smooth for many many months, until we started looking into lighting and rendering. Then it occured to me that I needed the ability to export passes, so they could be re-created later on. I thought that since I had code that could write shaders to file, then surely I could reuse this code for other things than materials: overrides and pass shader stacks. Well, that’s what I thought. But my code was built entirely around the material export, so I could hardly use it for overrides and pass stacks without making drastic modifications.

Pass shader stacks
There are two major problems with pass shader stacks.

The first problem is that from all practical purposes, passes have the appearance of materials. If you select a pass and open the Render Tree, you’ll see the pass object being displayed as if it was a material. In reality, that is not quite right.

For instance, you can iterate the parameters of material, and effectively traverse the Render Tree from there. That is not happening with passes. First, passes do not have a Parameters property, and second, they do not expose shaders in any way. So the Render Tree display is highly misleading, and I had to write code to handle passes differently than materials. In short, I did not find any quick and easy way to get the stack shaders. This is illustrated by this stripped down method that collects the stack “root” shaders (see next paragraph):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
def addpass( self, oPass ):
 
	aStacks = [
		'Environment Shader Stack',
		'Output Shader Stack',
		'Volume Shader Stack'
	]
 
	# Iterate pass stacks
	for sStack in aStacks:
 
		# Get the stack object
		oStack = oPass.nestedobjects( sStack )
 
		# Check if the stack has shaders
		if oStack.nestedobjects.count > 0:
 
			# Iterate root shaders of this stack
			for oStackShader in oStack.nestedobjects:
 
				# do something with shader

The second problem is that in a pass, there can be multiple shaders living at the top-level of a stack (which I shall call “root” shaders). For instance, if you apply two volume shaders to a pass and then open the Render Tree of that pass, you’ll only see the latest volume shader. That’s another misleading aspect of the pass’s Render Tree. So it can be assumed a pass shader stack can have multiple “root” shaders, and each of them is entirely valid and must be exported if you want to fully export the pass shader stacks. The upshot is that the pass object act as a name space, just as a material: among all the shader stacks of a pass, no two shaders can share the same name. This is great, but that doesn’t solve our problem of multiple root shaders for a same stack.

Also, when importing a pass shader stack, using the typical ways of creating shaders and plugging them afterward worked for all but the “root” shaders. To import pass shader stacks, each shader had to have a “stackroot” flag in the XML file. When meeting such a flag, the script would not simply create the shader and plug it later, it would have to run the command that creates root shaders in stacks. This command is the only way to not destroy other existing root shaders!

Overrides
Overrides pose an entirely different set of problems.

They are not materials, and they are not pass shader stacks, yet they can host complex networks of shaders. There also can be more than one override per partition! An override could not be handled as a material nor a pass. Materials, in my framework, were meant to be exported as single files, overrides needed to be exported in a pass file, along the shader stacks, partitions and render options. An override doesn’t have things like the material’s UsedBy property, it is a partition property.

Also, overrides on partitions are visible only in the current pass. I thought that this was case only in the GUI, but found out that this extends to the scripting interface. If you iterate the properties of a partition not of the current pass, you’ll never catch that partition’s overrides. The only solution was to use a low-level command such as FindObjects() to collect all overrides, which works, and then organise them by pass and by partition, to later write them to file along the partition that’s being written.

Using FindObjects() to collect overrides returned a lot more overrides that what I expected. All these overrides were named VisibilityOverrideXX (“XX” being a number). I basically found out that group objects (groups, layers, partitions) implement the parameter inheritance via an override. This override is created when a parameter of the group object is not set to “no effect on members”, but remain invisible at all times, except to commands like FindObjects(). This can be tricky, because you don’t want to export this override. However you never know when a user will create an override to disable a parameter like “viewvis”, and name this override “VisibilityOverride”. The only solution when such an override name is encountered is to check that the partition parameters all set to “no effect on members”.

But the biggest problem with overrides came with the import. This article is not about shader import (which may be covered in another article), but few things are worth mentioning.

You can’t add a parameter to an override if the parameter cannot be found in the partition members. The only way to add a parameter to an override is to use SIAddEntryToOverride(), which requires a parameter FullName string. My solution was not very elegant, and is not garanteed. Basically the code tries to find the parameter among the partition members. If find its, then good, the entry can be added. Otherwise, you’re out of luck. Materials are also tricky because the override parameter name doesn’t exactly reflect the real shader parameter FullName that has to be overridden. With elaborate string manipulation this can all be worked out, but this is slow and like I said, not very reliable.

Another annoying thing is that an entry added to an override cannot be accessed right away via the object model. At least that was the case in XSI 5.11 (I tried in 6.01 and it seems to work). For instance, the following code would raise an error:

1
2
3
4
5
6
7
8
9
10
11
xsi = Application
 
# Get the selected override
oOverride = xsi.selection(0)
 
sParameterName = 'viewvis'
sParameterFullName = 'cube.visibility.%s' % sParameterName
 
xsi.SIAddEntryToOverride( oOverride, sParameterFullName )
 
oOverride.parameters( sParameterName ).value = True

So to get back the added parameters I had to use strings.

Cameras and lights
I have not talked about cameras and lights so far, simply because as the time of this writing, the need to export their shaders never arose.

In our pipeline, cameras and lights are exported as full-fledge models. Their importance is rather low compared to other models (like characters, for instance), so that exporting their network of shaders was not felt as a necessity. Also, cameras and lights are very light-weight compared to other models, so managing them always as models posed no particular problem.

My belief is that lights can be handled not too differently than materials. The catch word is “too”. The light primitive is a bit like a material, it has inputs for connecting shaders. But a light is not a material, and I’m affraid that handling lights would require yet another dedicated wrapper to hide the real nature of the light.

In the case of camera shaders, my guess is that they behave in a similar fashion as pass shader stacks do. The camera shader stack lives right under the camera primitive, and appears to be accessible only through the NestedObjects property. Just as a pass shader stack, the camera shader stacks accepts multiple shaders at its top level, but when you open the Render Tree, it shows only the first shader of the stack.

THE SHADER NAME SPACE

Obviously, trying to adapt the existing material export code to also export pass shader stacks and overrides turned out to be very tedious. So I had to make drastic changes to the framework, and implement a new interface: the shader name space.

I consider the material, the override and the pass shader stack to be shader name spaces. That is, a space where a collection of shader names live. A material, a pass and an override as all shader name spaces. Lights and cameras are also shader name spaces. As of this writing, however, my framework does not include anything to export and import lights and cameras.

The object used as the shader name space is not necessarily the “root” shader, that is, the shader of the tree where all others ultimately plug in. Materials and overrides are both name space and root, but in the case of pass shader stacks, each “root” shader of a stack is a shader name space root. So a shader name space can have multiple roots. In the shader name space, the exact nature of the name space is not really important, as it is hidden by the shader name space interface.

Treating data that way made things far more easier to work with. Low-level functions would write to file without knowing the context of use, while wrappers would handle all the specificities of the name space.

PROPOSITION OF IMPROVEMENT FOR MATERIALS

All of this hassle with shader export and shader name space made me think about how it could have been a lot easier. I had to write code to handle all types of shader name space, which resulted in significant amounts of code and increased opportunities for bugs to make it through. The shader name space is in fact an attempt at implementing something that, I believe, should be in XSI: new kinds of materials.

Lights, passes and cameras should implement a material. I don’t see any reason why not.

- Imagine if light, pass, and camera materials could be managed from the Explorer, just like other materials. Light materials could use an orange color, while pass materials could be green and camera materials could use a gray color. The color is not important, but this mean they could a lot easier to manage, especially in large amounts. They could be organised in libraries, exported, and referenced.

- The mess of pass and camera shader stacks would not be visible anymore.

- Sharing materials among lights, passes and cameras would be a breeze. Increased reusability.

- Programming-wise, they would make the process very homogenous. No matter if you select a light, geometry, or a pass, you can always use its Material property to access the material. Simple, straightforward and consistent.

I don’t really see where overrides would fit in this design. Perhaps they could be left as they are, but at least it’s only one special case to handle, not every case.

I’m strongly against the idea of having a “super” material. For instance this material could be applied to anything, however it would expose only parameters suitable for the context it’s being used in. When applied to different kinds of objects, I hardly see how it would be easy to work with. It would make scripting very inconsistent and require the same kind of tedious handling I have been discussing in this article.

Get a material and a passes file here: Material and passes XML files

ABOUT DATA INPUT MANAGEMENT IN MATERIAL EXPORT

If you think that this section title sounds redundant, you are right. I have written an article on XSI-Blog about this very subject. I talk again about it because it really is during the development of this project that the importance of data input management struck me. I think that the data input management is one of the most fundamental aspects of any larger programming project.

I wanted the material export to be easy for the user. I wanted the interactive input to be simple and predictable. Users tend to favor the simplest ways of doing things. Given a model, they would expect to export everything related to this model. They don’t really care about the details, but they know that if all components of character lives under a unique model, then they have to be able to do things by simply handing the model to the script (via selection or picking, for example). However, exporting everything under a model can generate lots of data and take considerable time, so I wanted to give the users the ability to “focus” the export: given a selected object, it would export only the object’s material(s). Or the user could even select the material and export it. Painless, fast and predictable.

But giving the users various ways of exporting things meant that I had to write considerable code to manage the data input. The code that wrote to XML files was dumb, but lives below thick wrappers that would manage all the data input from the user. When the materials to export would reach the actual exporting functions, it would have to be guaranteed to be adequate.

Here is a little chart to show the framework design to cope with this task.

- call from the user
– user data input management (objects, materials…)
– final data input management (by type of data)
– export of shader name spaces
– fetch data from structures
– write to file

The difference between “user data input management” and “final data input management” is that they used a different interface. Both interface would handle data, but with different purposes in mind. Ultimately, they would end up populating data structures that, too, served specific puroses.

The first layer of input management handled the data provided by the user to make sure it was acceptable for the export and/or the import. For example, it ould make sure that only materials, overrides and passes could make it through for an export. The interface used to perform this task is an interface used throughout the entire framework to take care of all initial data inputs.

That interface, that I called the “Chapter” (as in a chapter in the biker world) would know everything about the context: is the user exporting or importing, is he exporting/importing in a shot or in an element, things like that. Through a system of promotion and demotion, the Chapter would store “suitable” primary data into structures that could be later accessed by everyone else.

The second layer of input management, “final data input management”, as well as “fetch data from structures”, used a more specialized interface, dedicated to shader export. Its job would be to hide all the complexities in the differences that exist between materials, overrides and shader stacks.

In “final data input management”, it would extract the final data from the “suitable” primary data coming from the Chapter, and store it in data structures. In “fetch data from structures”, it would read this final data on-demand as the script goes on. The interface would make the storing and reading of that data completely homogenous, no matter if dealing with a material, a pass, or an override. While resulting in very long code (the entire interface is implemented as nested classes), it made the overall data handling incredibly easy.

5 Responses to “About exporting shaders”

  1. Jason Dexter says:

    Very thorough article Bernard. If you’re at Siggraph, hit me up for a beer, this has saved me quite a bit of effort.

  2. Hey Jason, glad you find this article useful. I won’t be at Siggraph this year. I’m considering the idea of writing a quick start scripting tutorial on the blog about dealing with shaders, maybe that could help a few people out there :-)

  3. robert says:

    let me add – great article Bernard. Very thorough and clearly the result of experience. Good job!

  4. thiago says:

    thanks for share your experience with this.
    I found that share passes is really hard…. I wish to have a tool that let us able to manage passes inside the whole project or at least a couple of scenes.

  5. Hi thiago, sorry for the long delay in replying, just saw your comment.

    Well for this project I designed a “pass” granule. The pass granule is a file that is not too far from the scntoc. It is a XML file.

    It lists all passes with render options, but also all partitions with their properties (including things like applied materials, overrides, volumic, etc). It also contains the pass shader stacks.

    Each pass granule is tied to a lightset, as passes are created at the same as lightsets. Each lightset is tied to a specific stage (aka environment, aka set). The pass granule has the same name as the lightset, so it’s clear what goes with what.

    Now the thing is that our pipeline is database-driven, so it’s easy to export and import pass granules, and it’s also easy to establish relations between pass granules, overrides (which are exported separately), lightsets, and stages. All the user has to do is to specify which stage he is working in, and he gets a choice of related assets to import. Works well for the most part.

    Cheers
    Bernard