I am working on some python 27 code for a publish pipeline.
In maya, I import my module foo.main which import foo.publish
If I need to change foo.publish, I can reload foo.main in Maya, but changes in foo.publish won’t take effect. The only way I found to relaunch the process is to close and launch Maya again.
How can I speed up the process? I tried deleting .pyc files in /foo but this does not seem to work.
Thanks
In the init file of your main module you can add:
try:
# First time the module imports, loaded is undefined and will trigger the exception.
# loaded is then defined in the exception block
# when reloaded, loaded is defined as true, so reload your submodules:
if loaded:
reload(subModule)
except:
loaded = True
I have done something along these lines for a few modules and it works well.
Gonna try this out, thanks!
I’ve posted this before, but here is my favorite way of handling this by far!
Python stores all imports in a dictionary sys.modules. So if you remove an entry from there, it’s like it was never imported in the first place!
Now, of course, this can be dangerous (you could technically unload built-in stuff). But with proper filtering and edge case handling (which we’ve handled already!) we almost never have to restart maya to get code updates out to our artists.
One final note: If you’re doing anything with Qt (or any other non-maya-builtin ui), then you need to close the window you’re working on before you remove the entries from sys.modules. Our implementation emits a Qt signal that all of our UI’s connect to so it auto-closes the windows, but that’s the one thing you’ll have to handle yourself if you decide to use this.
import os, sys
def clearPathSymbols(paths, keepers=None):
"""
Removes path symbols from the environment.
This means I can unload my tools from the current process and re-import them
rather than dealing with the always finicky reload()
I use directory paths rather than module names because it gives me more control
over what is unloaded
*Make sure to close any UI's you're clearing before using this function*
Parameters
----------
paths : list
List of directory paths that will have their modules removed
keepers : list, optional
List of module names that will not be removed
"""
## TODO ## Possibly emit a signal to close my custom UI's
keepers = keepers or []
paths = [os.path.normcase(os.path.normpath(p)) for p in paths]
for key, value in sys.modules.items():
protected = False
# Used by multiprocessing library, don't remove this.
if key == '__parents_main__':
protected = True
# Protect submodules of protected packages
if key in keepers:
protected = True
ckey = key
while not protected and '.' in ckey:
ckey = ckey.rsplit('.', 1)[0]
if ckey in keepers:
protected = True
if protected:
continue
try:
packPath = value.__file__
except AttributeError:
continue
packPath = os.path.normcase(os.path.normpath(packPath))
isEnvPackage = any(packPath.startswith(p) for p in paths)
if isEnvPackage:
sys.modules.pop(key)
With that function defined, you could do something like this:
clearPathSymbols([r'drive:\path\to\myModule', r'drive:\path\to\some\other\submodule'])
import myModule
That’s pretty clever, thanks a lot! I’m not sure how you implement it in your code: do you use the function with every import statement in every file?
This function removes any modules imported from below some given folder. So just pass it the top folder of the tool you’re developing.
Here, let me give you some usage examples.
First, I drop a file with clearPathSymbols into my scripts folder, so it’s available to Maya’s python.
Then, when I’m developing anything, I just make a shelf button for “Reload this tool”.
For example, my shelf would contain something like:
from clearPathSymbols import clearPathSymbols
clearPathSymbols(['/path/to/moduleInDdevelopment'])
from moduleInDevelopment import someFunction
someFunction()
That way, without changing anything about the tool I’m writing, I can force a reload whenever I want without caring how the imports are structured.
The other thing that really comes in handy is having a shelf button that reloads the entire user scripts directory so any changes I make anywhere can be easily reloaded. I just have to make sure that clearPathSymbols is in my keepers list because I’ve got it defined in my user scripts folder.
from clearPathSymbols import clearPathSymbols
clearPathSymbols(['/path/to/scripts'], keepers=['clearPathSymbols'])
We have something similar to this at work. Everybody in the company just has a shelf button that reloads the entire scripts directory, so any time a hotfix comes out, we just tell the users to hit a button, and relaunch the fixed tool. No restarts required.
Really nice, I’ll try it out for our Shelf! hanks a lot
I want to throw out my usual warning about reload being a tricky beast.
One thing to keep in mind is you can only reload pure python modules, so if you’ve got any modules that are compiled extensions, be super careful.
Another is that reload doesn’t go fix up references to objects from before the reload triggered, so depending on how you’re interacting with the code you can end up with multiple instances of the same class definition in memory, this can really annoy parts of the runtime. This would usually bite me when trying to super a method on an instance that pre-dated a module reload where it was still referencing the older version of the class instance.
Some libraries are very reload resistant, and will even fight you when doing the “clear sys.modules and re-import trick”. Pymel is a perfect example of a library that will cause a fairly huge memory leak if you do this. (It purposefully creates reference cycles on a bunch of internal modules, and the garbage collector never seems to be able to break them).
Reload is a great tool for rapidly iterating on a tool during development, but really shouldn’t be relied on by users in the wild.
Yeah, wild reloads cause all sorts of problems.
Make sure you are very precise in your use of reload outside of development, as @bob.w says.
The instances where I have used my example have been to reload sub-modules that I have broken up for the sake of organisation, but I still want the top level module to function as if it were flat and had no sub-modules.
Other than that, I do regular reload checks across our in house code base to make sure no reloads have been left uncommented as they can cause major headaches.
That’s why I like @Munkybutt first answer: modules are automatically reloaded at import, that seem pretty safe.
Thanks all of you for your help
Is that Qt functionality dependent on all your UIs living in the same package then? Or do you have some os-level or maya-level way of registering Qt UIs at creation, regardless of which tool or module created them?
Hmm, I guess I’m not following the question very well. Or, more precisely, I can’t figure out the pattern that would lead you to ask that question. So, I’ll just give you a more detailed rundown of how our system works in hopes that answers the question.
The vast majority of our custom tools are under one big folder (the tools repository) so we just clear that path. That folder is added to sys.path via a “magic” import at runtime, so we can easily track where we’re pointing, and switch between development branches on the fly.
There are some other folders that we don’t clear (reloading these does require a maya restart). One of these contains subclasses of QDialog and QWindow that automatically connect to our aboutToClearPaths signal. We enforce that all UI’s written here derive from one of those subclasses, and are saved in the tools repository.
There are, of course, edge cases or third party tools we can’t change. In some cases, we still have access to the main UI, and can manually connect aboutToClearPaths to the window close… but since we can’t develop those tools, reloading them doesn’t matter.
Thanks, and sorry I could have phrased it better, but that answered the question