Maya / Python threading

Coding
, ,
#1

Hey!

I’m trying to do some background processing while the main Maya UI is still responsive, so, I’m trying some thread stuff from within Python. I’m currently testing on Maya2019, so I’m limited to Python2.7.

I’ve tried creating a thread subclassing python’s threading.Thread and also using the built-in maya.utils.executeDeferred().

The result is the same - the thread gets executed, but Maya’s UI is blocked during its execution.

I have even tried a super simple test where the thread just calls time.sleep(1) 20 times while print that out, and yes, I see the prints go by in the script editor output window, but Maya is still blocked.

Any clues on what’s going on?

#2

so you will want to start your new thread with daemon set to true, so it runs along side the main thread. But also keep in mind concurrent programming is hard, and from your thread you can not access most of maya’s api. so if its just some data crunching going on with it sure it could work but if its doing stuff with the cmds or the maya api then you will have problems.

#3

Huh, daemon flag? I’ll try that, thanks. Yeah, concurrency is hard but fun! I’m only doing my own processing, nothing touching maya. Well, some read-only USD stuff, but nothing that’s not explicitly thread safe. At the end of processing I trigger a refresh in the main thread, sp that should be fine.

#4

yeah have not had to do this in maya, but have done it in other similar cases, but was able to just make my own class extend from Thread, set daemon to true when i call bases init and override run.

then just use queue.Queue as a thread safe way to push data that can be picked up by the main thread later

#5

Setting daemon=True didn’t change a thing, unfortunately! Doing heavy processing in that background thread will just bog down Maya so it’s absolutely unresponsive. The only thing I have managed to get to work was to put in a 10ms sleep in there every now and again, which is far from ideal.

#6

chances are you are running into something in python called global interpreter lock. more or less true multi-threading is really hard in python because of it, if you were doing something that was more IO bound and not CPU bound you might have been getting results more similar to what i have.

#7

tl;dr The best I’ve got so far is to manually spawn a new subprocess, and pass data and commands to that subprocess, and have the subprocess handle any multiprocessing tasks.

And as always: If anybody reads this and has a better solution, please let me know!

Ah, welcome to my little bit of insanity!

Threading certainly isn’t great in Python. There’s a guy working on removing the GIL from Python, and it’s a neat idea, but it’s gonna be quite a while before we see it, and even longer before Autodesk integrates it into Maya.

So the way that people have figured out how to do something close to threading in Python is to use the multiprocessing module. You start multiple python separate processes, and send pickled data between them via sockets. Best part? multiprocessing is almost a drop-in replacement for threading.

One thing you’ve gotta be careful of is which process gets created. multiprocessing defaults to using whatever is in sys.executable … And in Maya, that’s the current maya executable. So you have to explicitly pass the path to python instead, otherwise you’ll start spawning new mayas left and right. So you’d think you could pass the path to mayapy … but alas, you’ll get command prompts flashing up all over the place… So maybe you pass a path to the system python windowless executable. (for me it’s r'C:\Python27\pythonw.exe', note the “w”) but that doesn’t work either.

If you look REALLY fast, you can see the flashes of mayapy prompts erroring with
ImportError: No module named maya … which I guess makes sense since maya is part of the main namespace … (edit: Maybe doing it in a different namespace would work? Like instead of doing it in the script editor, do it in a file on disk that gets imported? Ooh, something new to try!) However, we don’t really have any control over how the processes are spawned and what modules are imported or when, so … this is where I called it a dead end :frowning:

The best I’ve got so far is to manually spawn a new subprocess (using the system python so I can use modules incompatible with maya!), and pass data and commands to that subprocess via pipes or pickled objects, and have the subprocess do any multiprocessing. Luckily work has a bunch of libraries to make that easy for me (we use this exact process for spawning other programs for automation). One major hint to get it working: you’ve gotta clean up the cwd and env for the subprocess you spawn, otherwise you start calling maya dll’s and pyd’s from system python, and you get errors everywhere.

#8

The thing about threading is that – in Maya land, anyway – it’s often just not worth the hassle.

The ideal scenario is a long but self contained operation – you set it up, fire it off, and then integrate the results into the main Maya world when it’s all done. There are variations – for example, you might want to catch status updates or maybe the big job is really a queue of many smaller ones. But the key rule is the same: what happens in a thread, stays in a thread until its done. Otherwise you have to expend so much work on thread safety that the gains are often minimized.

Dealing with external processes is one good use case for threads. Another is dealing with the internet. In both of these cases the default behavior would be a long freeze.

The generic setup is always three parts:

  1. Collecting information from the Maya session up front – anything about the state of the scene or the UI – and setting up the job. The key here is to collect the information up front – you don’t want to reach out from the thread and try to run a call from cmds or something like that.
  2. Processing that information – whatever that involves . The results are popped into some kind of thread-safe container, must likey a Queue.
  3. Collecting the results and returning them to the main thread. 9 times out of 10 this is going to involve polling the container from step 2 and reacting when something new shows up. A good pattern is to create an idle-time script job that checks the container and handles the messages from the queue.

One thing that’s worth considering is whether you really need threads or whether you just need concurrency . Threads are just one way to achieve concurrency – and because getting your thread setup wrong is pretty crashy, it’s not always the right one. Simple stuff like time delays or interleaved functions can be done with Python generators – those allow you to set up jobs that work one step at a time and to ‘tick’ them in ways that work for you. This can be nice if you have a lot of processes which have to happen in your main Maya thread but which don’t need to follow a rigid sequence .

If you dont’ want to spend a bunch of time setting up the various pieces of a thread-polling or job-dispatching system, you can steal ideas from this project (which is older, so 2.7x)

It’s an unthreaded concurrency system but it also includes a way to build threaded tasks with a built in “do the work, toss the results in a Queue, then react when it’s done” primitive. Another off-the-shelf system you can leverage is the QThread that comes with QT.

1 Like
#9

As with any tool, software or otherwise, it has its use and potential for abuse!

In my particular case, the processing I needed to do is lengthy, but unimportant for immediate UI, so it’s a great candidate to thread out. The scene gets loaded quickly, the user can start using Maya straight away, while things are getting processed in the background. Basically, spawn off a thread, let it do its magic, and when it’s done, it triggers a Qt update to refresh a tree.

In the end, doing this processing in Python was just way too slow, because of Python. So I moved it to C++ and it runs fast enough that it can be done in blocking mode without the user worrying too much.