Speeding up MayaPy processing of simple objects en masse?

I’ve got a large number (10,000’s) of files I’m running through MayaPy. As each file gets brought in to process I’m setting/getting attrs, snapping with cmds.xform, creating constraints, setting keyframes, etc. Somewhat vanilla stuff.

I don’t have a lot of experience with substantial file processing. What’s the best way to speed things up? I’m not maxing out RAM, so is the CPU the only thing that would really bump up performance here? Any tips on what the “best” hardware (and methods) at this type of job would be appreciated.

Code optimization is a win, but I feel like I’m extracting blood from a stone there now. It’s just a really heavy file load.

PS I’ve looked at multithreading (Maya only uses 12% of CPU!) but that seems like a big headache to get working.

Any advice or recommendations here are appreciated!

Well one option is to run multiple instances in parallel.
In theory you should be able to process N number of objects where N is the number of cores on your CPU. As long as everything fits into memory when batched like this.

Another thing to keep in mind is that spinning up a mayapy instance can be kind of costly time wise, so if you can set things up so that each process doesn’t have to shut down after finishing a file that’s always a great savings. But you do need to make sure that nothing leaks from one object to the next.

Python has various tools that you can leverage for this, the multiprocessing module for instance will let you leverage a bunch of processes, but its tricky to use with maya because of spinning up a standalone environment for each subprocess. Also it works better on *nix than windows.

1 Like

I think there was an old python module called ManyMaya that we used to use sometimes.
It handled all the multi process stuff for you.

One thing you could do is split your list of files into multiple lists and manually start multiple of your mayapy batch giving each one a different list.

Nowadays we have our renderfarm setup to do arbitrary tasks so I would just set it up to run on there and have a dozen computers plow through it.

1 Like

Yeah, if you’ve got a way to distribute the tasks to even more machines, it can be a super win.

1 Like

For the simplest of tasks, like getting and setting attributes - and maybe creating and connecting things too? - there is also this.

It’d enable you to open and edit files without Maya.

It’s probably using 100% of a single core, the multi-processing route should help you maximise that as well albeit a little more slowly and costly (and complexly) than if you manage to operate solely on the file itself.

1 Like

It’s probably already on your radar – but for batch jobs like this it is worth pointing out that starting and shutting down a mayapy instance takes time, so it’s a lot better not to treat each job as

for job in jobs:
    start standalone 
         do work 

instead you want

start standalone
for job in jobs:
     do work

in other words – what @bob.w said.

1 Like

ManyMaya looks really interesting. Haven’t been able to get it working with 2020 though. Just opens up a bunch of Output windows and hangs. I don’t have a good handle on its multiprocessing, still digging around though.

Are you aware of something similar to ManyMaya for 2020 that also does instance management based on an arbitrary for loop instead (e.g. “for thing in my_list: run_script_with_new_instance()”? Trying to get away from using individual files, but I really like the queue management that ManyMaya does.

FWIW I ended up reworking some bits in ManyMaya and it does the trick pretty darn well. It requires a filePath argument (to either load that file or save to that file). So that took some wrangling of my own code. But the results are pretty great, minus all the outputs randomly intertwining text. Just wish I had a way to farm it to some other machines in the house! Someday…

Cheers!

A Python-3 friendly version of MultiMaya would be a great contribution to the TAO Github repository… just sayin…