Automating Max, Maya and possibly others from Python "interactively"

Coding
#1

Hey everyone,

I am recently looking into automating things quite a bit differently than i used to. And i am looking for input on how to best approach this.
As an example i’ll use a geometry conversion pipeline. Imagine incoming data in format .in and it needs to be converted to format .out.
While doing the conversion there might be an arbitrary amount of intermediate steps to be performed, ranging from small cleanups and
sanity checks to more complex tasks. So all in all we’ll end up with something like this:

.in files -> in-task-1 -> in-task-2 -> in-task-3 -> convert to .out file -> out-task-1 -> out-task-2 -> final .out file

Now currently this is implemented in a multi-step process. Let’s assume .in being Maya and .out being Max (in the end, it should not matter at all ideally).

So there currently would be a Maya toolset that performs in-tasks 1-3, then exports to an intermediate format. Then there would be a Max
toolset that imports the intermediate file, performs out-tasks 1 and 2 and saves as the final .out file.
Yes…this is cumbersome. Now to ease things in a quick and hacky way the scripts have been adopted to allow being run via cmdline instances
so there is no more user interaction. This has proven to be incredibly cumbersome in many many ways. Especially handling output (both errors or information)
is really bad and it is quite easy to have it run for an hour, offer no progress report whatsoever and not even produce a single output file because
an error is not catched properly.
In addition this setup is VERY specific to the involved apps. Doing the same thing for the other direction is pretty much a full re-write.

Now what i would like to do is have a standalone tool to manage the conversion. And depending on the in and out files it would launch the needed tool to the
background and remote control it. Ideally it would get life feedback from the app to be easily able to stop if an error occurred or the user wants to.

Also, I would like to save myself from having to implement all the needed functionality on the application side and then calling the methods via sockets, or
whatever. Ideally i’d have a kind of interactive session that i can send commands to (which would be application native commands) and get the proper
returns.

Now has anyone done something alike and has any recommendations? Using Pyro to wrap things? Using Sockets and blindly execute strings as commands?
Or using 0mq to do the very same?

Am eager for any input, even if it crushes my findings so far, keep em coming :wink:

Kind Regards,
Thorsten

Communication from the outside with Maya, Softimage, Nuke etc
#2

We do this extensively (in fact we just replaced our LoD creation with XSI and had a server version of it running in a few hours). It’s pretty simple, and at this point, has been abstracted away into a few base classes that do everything.

You have a ‘client’ class that uses a zmq REQ socket. It sends [funcname, args, kwargs] to the server.
You have a ‘server’ class that uses a zmq REP socket. It has Eval and Exec methods. So the ‘funcname’ from the client should be ‘Eval’ or ‘Exec’. The server gets the command and calls the eval or exec statement (and for eval, returns the result).

Incredibly simple, incredibly powerful. That’s all there is to it, but the more complex portion becomes error handling (what happens when you encounter an exception in an eval’ed/exec’ed function?); what we call the ‘bootstrap/handshake’, where a client starts up its own server instance, since we need multiple instances of our apps running (ie, we can’t hard-code what ports things run on); the very different concept of a ‘server’ (ie, usually servers are not stateful, all state is in a DB. In our case, DCC apps are stateful, so we end up with tightly coupled server-client pairs); and other things like that.

What you end up with is actually a very robust system and the calling code makes sense (unlike DCC code) and stays in pure python (which also makes it easier to test). I’m hoping to find some time to present something more in-depth, since I keep hinting at its power but haven’t discussed it enough.

#3

Thanks a lot Rob. It’s actually a great help already to get a direction hinted. Even more so that it works out in a robust way for you! So i’ll look into it in more depth.

So in order to get things into “ready for work” state the standalone tool would launch max/maya/whatever into cmdline mode with a script to start the REP socket?

Thanks Rob,
Thorsten

#4

Correct. Though it doesn’t have to be commandline mode. In fact having a GUI can be very useful for debugging. For example today, we have a part of our LoD pipeline that is Maya->FBX->XSI->FBX->Maya->GR2. I wondered what the last Maya looked like. I just ran our server as a maya.exe instead of mayabatch.exe, and I was able to watch and inspect stuff as it (more like after it) happened.

Our startup script for Maya looks like ‘import mayaservice.server; mayaservice.server.ServiceImpl(xxxx, xxxx).Run()’, which 1) imports, 2) creates a new service implementation (which will bind to the port, handshake, etc), and then runs it in an infinite loop.

#5

I see. Thanks again for the input. I guess i’ll be doing a POC implementation to see where that leads to. Sounds to be feasible and rather quick to get basically going.

Thorsten

#6

Bringing this back up as now that i started with a simple POC implementation some questions arose:
[ul]
[li]Do you simply wrap [funcname, args, kwargs] as a string and reparse it to actual commands? Do you automatically derive these or construct a string manually?[/li][li]The exec method is non-verbose, otherwise the same as eval? In that case i would probably ignore exec.[/li][li]Error handling seems tricky indeed. Especially with complex task chains. Or what happens if the server crashes? Standard timeout procedure or something more elaborate?[/li][li]Do you also do this with Max? Or only XSI, Maya? In Max i will either have to try and get an mxs wrapper for 0mq working via dotNet, or go the python route. With our way of using python in max (through blur’s Py3dsMax) i am seeing issues with getting return values and catching errors. Any hints maybe?[/li][/ul]

Regards,
Thorsten

#7

if you want to run a python based automation server you should also take a look at RPYC, which makes it much less painful

DO pay attention to the security issues once you go from test code to something that will be released into the wild! RPYC makes it pretty easy to create limited services but it also allows you to create completely open ‘remote control’ access which is fine for testing but a Bad Idea™ for anything that will get used unsupervised.

#8

Without having looked into it in Detail this seems similar to Pyro? How does it compare? Or am i off track there?

Thorsten

#9

yep, in the same ballpark. I like RPYC for ease-of-use, network discovery, and the ‘classic mode’ that gives you complete interactive access to the target machine is very useful (as long as you never, ever, ever let it out into the hands of users :slight_smile: ! )

#10

Very cool! I’ll take a look asap! I still need to find a way to make Py3dsmax work more reliable in terms of return values :confused:

#11

>Do you simply wrap [funcname, args, kwargs] as a string and reparse it to actual commands? Do you automatically derive these or construct a string manually?


# Client
zmqsock.send(cPickle.dumps([funcname, args, kwargs])

# On the server:
func, args, kwargs = cPickle.loads(zmqsock.recv())
getattr(self, func)(*args, **kwargs)

The server class (self) has Eval and Exec methods on it.

The exec method is non-verbose, otherwise the same as eval? In that case i would probably ignore exec.
Eval is limited because it can only evaluate expressions. So you need exec.

Error handling seems tricky indeed. Especially with complex task chains. Or what happens if the server crashes? Standard timeout procedure or something more elaborate?
We break it down into 2 errors:
The first is error in user code- so we’d do something like:


# Server
f = getattr(self, func)
try:
    result = func(*args, **kwargs)
except Exception:
    result = IpcError(sys.exc_info())
zmqsock.send(cPickle.dumps(result))

# On the client
result = cPickle.loads(zmqsock.recv())
if isinstance(result, IpcError):
    raise result

In the case of other errors, we run the server in a function that will email us if we crash. The server shouldn’t crash!

>Do you also do this with Max? Or only XSI, Maya? In Max i will either have to try and get an mxs wrapper for 0mq working via dotNet, or go the python route. With our way of using python in max (through blur’s Py3dsMax) i am seeing issues with getting return values and catching errors. Any hints maybe?
I think ZMQ already has a .NET wrapper? You should be able to get it working with any program/language. The only problem would be handling errors (and obviously, use json instead of cPickle)- you’d need a different protocol since you can’t pickle python types.

#12

Thanks again Rob! Makes a lot of sense. Will have to dive into that again once the current “distractions” are out of the way heh.

#13

Hi Rob,

I’ve used python to send commands to Maya and adobe fireworks but am looking for a solution that will let me send commands to various other software packages (Nuke, Photoshop, 3dsMax) and keep it all together in one tidy package. The idea is to have one python tool (let’s say an “open file” tool) that can be given a filepath and then send the open command to any of the above applications.

You mentioned zeromq - can you recommend any sites that’ll give some code examples for communicating with zeromq to any of these applications? I’ve read up on the zeromq site but am having some difficulty wrapping my head around everything or at least understanding how I’m going to implement it with my own code.

I’d greatly appreciate any insight!