tl;dr The best I’ve got so far is to manually spawn a new subprocess, and pass data and commands to that subprocess, and have the subprocess handle any multiprocessing tasks.
And as always: If anybody reads this and has a better solution, please let me know!
Ah, welcome to my little bit of insanity!
Threading certainly isn’t great in Python. There’s a guy working on removing the GIL from Python, and it’s a neat idea, but it’s gonna be quite a while before we see it, and even longer before Autodesk integrates it into Maya.
So the way that people have figured out how to do something close to threading in Python is to use the
multiprocessing module. You start multiple python separate processes, and send pickled data between them via sockets. Best part?
multiprocessing is almost a drop-in replacement for
One thing you’ve gotta be careful of is which process gets created.
multiprocessing defaults to using whatever is in
sys.executable … And in Maya, that’s the current maya executable. So you have to explicitly pass the path to python instead, otherwise you’ll start spawning new mayas left and right. So you’d think you could pass the path to mayapy … but alas, you’ll get command prompts flashing up all over the place… So maybe you pass a path to the system python windowless executable. (for me it’s
r'C:\Python27\pythonw.exe', note the “w”) but that doesn’t work either.
If you look REALLY fast, you can see the flashes of mayapy prompts erroring with
ImportError: No module named maya … which I guess makes sense since maya is part of the main namespace … (edit: Maybe doing it in a different namespace would work? Like instead of doing it in the script editor, do it in a file on disk that gets imported? Ooh, something new to try!) However, we don’t really have any control over how the processes are spawned and what modules are imported or when, so … this is where I called it a dead end
The best I’ve got so far is to manually spawn a new subprocess (using the system python so I can use modules incompatible with maya!), and pass data and commands to that subprocess via pipes or pickled objects, and have the subprocess do any multiprocessing. Luckily work has a bunch of libraries to make that easy for me (we use this exact process for spawning other programs for automation). One major hint to get it working: you’ve gotta clean up the
env for the subprocess you spawn, otherwise you start calling maya dll’s and pyd’s from system python, and you get errors everywhere.