About Input Data Management

April 27th, 2006 by Bernard Lebel - Viewed 8518 times -




ABSTRACT

This articles is two-fold: first, it presents a scripting tool that I wrote nearly a year ago, called xsiHandler. Second, the tool is used to discuss various aspects of data inputs into scripts.

INTRODUCTION

Recently I was having a discussion with a fellow technical director, and he said something along the lines of “In my scripts, 75% of the code is dedicated to error-checking and making sure the user doesn’t break the script. This drives me nuts!” That statement reminded me why I wrote xsiHandler about a year ago or so. xsiHandler is a Python library that deals specifically with this aspect of programming. Here I will use xsiHandler to discuss various aspects of error-checking and data input management in the context of providing scene elements to scripts.

WHY DATA INPUT MANAGEMENT?

I did not write xsiHandler out of creative inspiration. I started looking into a consolidated data input management solution because I was having the same frustrations as my TD friend, that is, most of my scripting time was spent on making sure my code would not break at the slightest unexpected input. Not only this was making my code longer, but I was always looking into my other scripts to find out the proper function for managing data inputs, as different scripts needed different data inputs.

But managing data input in a consistent and reliable fashion in my script could easily involve convoluted functions. For instance, let say you write a script that would subdivide a mesh. There would be several layers of error-checking in order to get a mesh in the script:

  • having something selected,
  • having not more than a single object selected (unless you put the code in a loop),
  • among these selected object, at least one needs to be a mesh,
  • if there is a loop, each object type individually needs to be tested.

That can make up for a significant amount of code and time spent writing it. If you need more than one object of different types, it becomes even more convoluted. You can take shortcuts by using pick sessions (although I believe pick sessions work well only when involving less than 5 clicks), but you get the idea.

So at one point I decided to consolidate all these functions into a single library that would do all the work for me. Or, at least, most of the work.

PRESENTATION OF xsiHandler

Before we go, I will briefly introduce the basic usage of xsiHandler. xsiHandler is a Python package. It is a data input manager that handles most inputs happening through user interaction with scene objects.

The only thing you have to do to use xsiHandler is to import the top program file (xsiHandler.py), call its main function using the right argument values. When the function returns, you either get the right data, or you get None.

So for example, if you want is the selected meshes, you could run this:

1
2
3
4
5
6
7
8
9
10
11
12
13
from xsiHandler import xsiHandler as xh
xsi = Application
 
oMeshes = xh.process( 10, None, '01_04_00', (1,0), 23, False )
 
# Or, using constants
oMeshes = xh.process( xh.scopeSelection, None, xh.typePolygonMesh,
	xh.amountAtLeastOne, xh.outpoutXSICollection, False )
 
if oMeshes == None:
	xsi.logmessage( 'No meshes selected, script aborted.' )
else:
	xsi.logmessage( 'Thank you.' )

The library does everything else, in this order: testing the arguments, checking if something is selected, checking if meshes are in the selection, checking if the required amount of meshes is selected, and wrapping everything into a XSICollection. If, at any point, a requirement can’t be met, then None is returned to the caller.

The first argument is the scope, then the custom data (which will be discussed last), then the data type, followed by the amount, then the output format, and finally the verbosity flag, used for debugging.

The only ‘real’ disadvantage (from my perspective) of xsiHandler therein lies in the import mechanism. When you use the tool for the first time in a live XSI session, it takes a few second before something happens. No biggie, but I would be very happy if there was a way to avoid that.

If you want to try out xsiHandler, you can find it here.

Note that non-Python users can also take advantage of xsiHandler. Simply write a command that wraps it: the command imports the module, call the process() function, and return the process() output as is. Since the output of xsiHandler is ActiveX-compliant, it should never be a problem.

Alright, let’s move on to the actual article.

WHAT IS “INPUT DATA”?

A programmer is writing a script that deals with scene data. To carry out the series of instructions designed by the programmer, the script needs to access the right data. In a significant majority of the time, the data is input right at the beginning (or very early in the process).

With a data input manager like xsiHandler, the “right” data is what comes out of several steps of processing, in order to meet the exact needs of the script implementing it. The “input data” refers to what xsiHandler outputs, which is then used as input data in the main script.

THE SCOPE (How is raw data initially acquired?)

The first step is to gather raw data. In some cases, the user has to explicitly provide the raw data (via selection, pick sessions, modal property pages, etc.), sometimes it has to be searched in the scene (like objects with a specific name, objects of a given type, etc.). There are many other means of getting data, but these are out of the scope of this article.

The xsiHandler “scope” defines by what mean is the raw data gathered, or within through what context is the raw data acquired. There are a few models.

Some are interactive, they will require the user to select something, or pick things. These are useful for scripts that operate on X3DObjects and scene elements living under the Scene_Root. Some scopes require that a collection or other sequential data type is provided, like lists or tuples.

On the other hand, the Application scope is not interactive at all, in fact, it will completely ignore the user and instead will look for data through a search function. This is most useful for scripts that operate on all instances of a given data type (like all the lights in the scene).

When I designed the scope argument, I also wanted to give the user a “second chance” when using the tool in interactive contexts. For instance, what happens if nothing is selected, or if not the right thing is selected, or if not enough items are selected? Several scopes can be combined with a pick session, so if the user initially supply incorrect data, a pick-session starts to give him a second chance. Pick sessions also have the advantage of providing more control over the input data, as you can specify a picking filter.

THE DATA TYPE (What actual data type am I looking for?)

At this point we can start processing the raw data to find out if we have data is of the right type.

“Type” is not to be confused with the XSI scripting API’s Type property. xsiHandler implements its own types. The two main reasons are:

  1. The XSI scripting API does not provide an easy way to test the type of any conceivable object (just think of models, reference models and instances, for example),
  2. I needed my own custom types. Such types include several rigs and a handful of custom properties used in very specific contexts.

No matter if I’m testing against a standard XSI data type (like a polygon mesh) or I’m testing against a custom data type (like the LightTools’ Advanced Light), xsiHandler uses all the tricks available to determine if the raw data is of the right type: the XSI Type property, the XSI ClassName value, the return value of commands (like GetMaster() and GetInstances(), for models and instances), the XSI ClassID, custom functions, and some more that I might be forgetting.

Custom functions are most useful with custom data types. An example: I make extensive use of customized lights. One of these setups consists of some lights being “instances” of other lights. That is, they inherit all their parameter values from other lights through expressions. To run some of the scripts of the LightTools package, I need to find out if the light is a “master” or an “instance”. Therefore, I need some custom type testing functions to perform a reliable evaluation.

There is something worthy of attention: the Application scope. When using the Application scope, there is much less type testing work needed, because a lot of data is firstly acquired through search functions that use the FindObjects command (which is fast and reliable), and then further filtered if needed. If the search functions return successfully, then the gathered raw data is necessarily of the right type. In other words, when using the Application scope, the type testing happens at the same time it is gathered. Some search functions need the specialized type testing functions though, a few of them will actually import the type testers and use them as if the data was tested through other scopes.

Lastly, there is a sad limitation regarding data types: the current implementation does not allow for multiple data types to be specified. The reason is solely because of the picking filter, which can’t be combined. So if I want to the pick filters to do a proper filtering job, I can’t specify multiple types. In order to specify multiple types, I would have to use a generic filter, which allows to pick too many different types. I have yet to decide which implementation is more advantageous.

THE QUANTITY (How many units of data do I want?)

So I have gathered data, and I have data of the right type. But how much of it do I want? Do I want one object? Do I want three objects? Do I care at all about quantity? First off, you definitely care about quantity, because otherwise you’d need not a tool like xsiHandler. That said, my experience has led me to believe that there are in fact two traditional values: it’s either one instance of the data type, or it’s at least one instance of the said data type. I have never used any other quantity notation with xsiHandler than these two.

In fact, I think the ‘at least one’ value is the one I use 99% of the time. I try to implement my scripts in loops as much as possible so they operate successfully on multiple units of data. If I enable the possibility to select or pick multiple objects and the user chooses to get these multiple instances of data, then the script should work through it reliably. Specifying a quantity is an elegant way to control how much data is provided to the script.

THE FORMATTING (In what format do I want the data passed to the script?)

Beyond the data type, the coder might want these units of data to be wrapped into something else, like an XSICollection or a list, for example. The output argument allows to specify such a format.

The output format is tied with the quantity. If you want one object only, you can output it as an object, or wrap it in a collection. But if you want multiple objects (or “at least one” object), then you have no choice but to ship those as a sequence.

The output format is another layer of processing, where the granules of data go through a final step of packaging.

AN EXAMPLE

Here is an example all these aspects can work together to make my coding time a pleasant experience.

I recently devised a script that creates a particle cloud consisting of geometry instances and uses a reference model as the PType instances. However, to determine how many instances the cloud consists of and to populate the RotationArray and PositionArray, I needed to supply multiple objects that already had the proper rotation and translation. I could extract the rotation and translation values from these objects, and transfer them in the particle arrays. The script works in the following manner:

  1. The user selects all the SRT-reference objects, no picking session allowed. It could be any X3DObject (which have kinematics).
  2. The user runs the script and is prompted to pick a reference model. This ref model has to be picked so as not to be confused with the original selected objects, and also to help the user understand what he is doing. Only one reference model can be picked, the pick session ending immediately after the first pick.

All of this is handled through two calls to the xsiHandler.process() function! The script continues:

  1. Create a group named according to the reference model
  2. Make the reference model member of that group
  3. Create a PType named according to the reference model
  4. Use the group as the instance group of this PType
  5. Create the cloud (also named according to the ref model) where the number of particles is based on the number of selected objects
  6. Connect the cloud to the PType
  7. Populate the rotation and position arrays using the selected object SRTs.

While it was certainly possible to test the type of SRT-reference objects and the reference model, the data input manager made things incredibly elegant and simple. If I had to deal with a more specialized type of object, then the data input manager would have been closer to a necessity.

CUSTOM DATA (What happens if I need more?)

I think that in most programs, there should be room for custom data. You never know when you’ll need it.

The xsiHandler main function allows you to pass a custom data argument. This argument is the only one not tested anywhere in the library other than where it is used. In other words, the argument handling functions do not test the custom data. But if you use a search function that requires custom data (searchAShader() needs a string, for instance), then the custom data value is tested within that search function. The custom data’s nature is left entirely to the developer.

SOME NOTES ABOUT THE IMPLEMENTATION

I have mentioned type testers and search functions. These form the majority of the code of xsiHandler. If you look at the code closely, you’ll notice that some type-testing involves barely more than a single line of comparison against a XSI Type or ClassName, and could probably have been replaced with lighter code.

The reason each xsiHandler type has its own type tester and search function is that it leaves the library wide open for development and is consistent throughout. You always know that for a given type, you have a type-tester and a search function. It’s up to the developer to implement these as he sees fit.

For my part I’m rather happy with the performance (besides the little gotcha mentioned earlier), only in larger scenes involving significant a amount of calls is performance an issue. For example, I used xsiHandler to collect all the instances in a scene where there were 1700 instances. Because of the command calls required to determine if a model is an instance, it took 10-12 seconds. In computer time that’s an eternity, but for a single line of code that prints a reliable result, I was not to complain. The beauty of the thing though is that there was ever a way to make this faster, it would need to be done only in one place, and all the scripts implementing xsiHandler would suddenly run faster. That’s a big advantage.

One last thing. When I designed xsiHandler, I wanted to leave it open for development. It is a piece of cake to implement new scopes, types, type-testers, search functions, and output formats. The documentation even contains development notes for each of these aspects :-)

CONCLUSION

I must say that having a proper data input manger made my life much simpler. The vast majority of my scripts implement it. When I need a new type, I simply add it. The length of my code has decreased significantly since then, and I can spend more time on writing the actual script.

One Response to “About Input Data Management”

  1. Hi Bernard,

    That”s a great solution to a problem that I”m always coming across. It”d be really nice if something like this could be integrated into the next version of XSI”s SDK. E.g. to have a new plugin category that defines types. This type of data/object abstraction is something that I think is desperately needed.

    What”d be really nice is if XSI could hide implementations inside scenes in a similar way to how Houdini does it”s “Digital Assets”. The TD would have much more control over what was presented to the artist. I think a lot of the tests that we have to do would be made much simpler if we knew that an artist couldn”t just dive into an Explorer view and start renaming and deleting stuff.

    Thanks for sharing your work.