Degrading import speed w/ consecutive FBX files

Coding
, ,
#1

I’ve got a few thousand FBX files I need to process, and they all contain just curves and transforms. They all get imported into one scene file. The amount of curves overall in each FBX on the high end is around 800 shape nodes (accompanied by another 800 xform nodes)

When I import a file from this set manually, it takes about 15 seconds. But while batching, the same process around position 600 in the list takes close to 2 minutes! Something is happening that is gradually degrading the import speed of all FBXs, and I’m at a loss to what it could be. It’s super fast to start, then degrades file by file.

The longest wait period in the import (or so it appears) is the FBX importer stating “Initialization” is working. That gets stuck for a minute or more at the 79% stage. Again, this issue doesn’t really exist when importing manually one by one.

I’m at a loss here. I’m already using FBX binary, bypassing importing lights, anims, etc. Does FBX do any garbage collection? Is there something I’m missing here, or does it just become horribly slow after consecutive imports via Python? I’ve looked at other formats for CAD like DWG, but those aren’t very well supported in Max or Maya it seems e.g. they’re broken as heck.

#2

It sounds like a memory issue - they should be garbage collected (although it depends on what you’re doing in honesty).
It might be faster to do this in the the fbx sdk - although it means more manual management of the merge operations.

Merging Two Scenes

#3

Hi John,

we had a similar experience with batching thousands of FBX files, Maya would just start to crawl and memory would continually climb. We ended up wrapping the fbx functions into our own class and making sure we cleared everything at the end of each loop. The del(fbx) here is just deleting our class object to make sure we release the memory, everything else is just trying to force Maya to release the memory. Now we never got to the bottom of exactly what it was that was doing this, but I know we fixed it with the addition of this clear block. Of course it might not be anything to do with these, but give it a go.

            cmds.flushUndo()
            cmds.clearCache(all=True)
            cmds.DeleteHistory()
            del(fbx)

We also started to wrap our handling of the fbx load to only actually use the import if we really needed too, we use Maya referencing to temp load the fbx wherever we can and handle the copying of key data in our code, then unload the data when we’re done.

I never fully trust fbx, it always seems like you’re stepping on eggshells. Take referencing, it’s actually fundamentally broken unless you code around it, the cmds.file(r=True) code doesn’t set the take ID for fbx so you end up never actually loading animation if you’re not careful, you actually have to set the FBXImporrtSetTake takeindex flag prior to the file call yourself.

cheers Mark

#4

@ldunham1 it turns out memory was part of the issue. Once it got really later, things were crawling. I need to find a different way to store this data as it eats up 32GB+ of RAM towards the end. Thank you.

@Mark-J This seems to have gotten me over the hump. Thank you! Would have never thought to wrap it it its own class. In the end I used your method but to convert all the FBXs over to MBs. Believe it or not, that was pretty quick via mayapy (way faster than importing them all together as FBXs into one scene). I’m not sure why it’s super crazy slow when combining them into one file as FBXs. Good ol FBX. Cheers!