Decoupling source and game directory trees

a question for game TAs

For every game project I’ve been on , I have seen a similar pattern of asset directories:
mirrored directory trees between DCC source files (Maya, 3DS Max) and exported game assets (materials, skinmeshes and animations)

The tool chains usually do something like


def convert_to_export_path(source_path):
	export_path= source_path.replace(SOURCE_ROOT_DIR, EXPORT_ROOT_DIR)
	return export_path

so C:/project/source_assets/path/to/file
becomes C:/project/game_assets/path/to/file

It’s nice because the export path is easy to determine from the source path. (of course it’s oversimplified here, but that’s the basic approach )

It’s been sufficient for the smaller teams and projects I have been on. However, as I have moved to larger teams and projects I have seen this tight coupling between directory structures introduce some friction and cost. Differences in what departments want create confusing structures for the other teams, and moving assets breaks things.

Have you achieved a decoupling of the source and exported asset directories?
If so, how to you make the export paths deterministic and tool friendly?
How do you trace exports back to source files for iteration & debugging?

This is something a lot of people struggle with - folder structure as a means of organisation can only offer so much of a solution as there will be many different and opposing factors wanting to manipulate it in their own direction. The folder structure should remain as simple as possible and then a layer of abstraction should sit over the top providing information to the users in customisable ways.
The only way you could do this is to move your source of truth to a database that would use a set of rules to determine paths and names and the mapping between source and content and any other meta data between the two. As games get bigger, managing them gets more complicated and thus needs something that can handle more complication than a folder structure.
In my experience, it seems the games industry is stuck in a bit of a “well we did it this way for 20 years, so why change it” rut in regards to this sort of thing.

You can take a look at Autodesk Shotgrid’s way of dealing with that:
A template field define some string “token” variables (such as asset_name, step, task_name, version_number, etc) and paths with those token (eg [project]/[step]/[asset_name]/[task_name]/[version])

You can define any token or path needed for your specific pipeline needs, which allows you to create file paths on the fly.

It also deals with decoupling source / export directories by listing “dependencies” (eg to build version X of your game asset you used version Y of the 3dsmax asset).

Yeah - it’s just a shame shotgrid has been completely poo-poo’d by the AD migration nonsense…

There are a few ways I can think of moving away from the mirrored folder method.

  1. As already mentioned database as source of truth. Requires all your tools (and version control?) to interact with the database.

  2. Control the file format. Wrap your assets in some kind of container format that allows you to include metadata (zip?). This way you can encode whatever properties you want in the asset itself.

  3. Side-car file. Similar to the above, but instead of wrapping everyone gets a buddy file that hold the metadata. Probably a bit easier if you don’t control all the things file related, but becomes more of a data management issue.

Thanks for the input. Storing path data is not really the part of the problem I’m pondering.
Our exports already contain references to the sourcefiles.

It’s the determination of the export file path that has me wondering.

Mirrored directories explicitly imply the correct sub path for an exported animation or rig.

But if you want to decouple the dir structures, then how do you enforce conventions or expectations in the exported file paths? Leaving it up to the users you get all kinds of inconsistent ad hoc decisions and frequent violations of the documented naming procedure (which everyone forgets about)

I’m playing this mind game:
What if all our exported anims had to live in a single flat folder and we had to name them uniquely, procedurally?

I imagine something like a UI with item lists which “build” a file path based on user selections, detected rigs, and source file name. Or something like the tag system in this forum. Perhaps saving that data to a scene node on export, so users don’t have to repeat the selections.

Or perhaps you have a tool that directly connects the exported data to the proper “container” resource in the game engine…

I just wonder how others have approached it.

So an important question here is “Do the file names matter?”
If not, give them all a random GUID, and leverage your resource management tools to map that GUID back to some display name.

Otherwise, yes I agree with you, don’t leave it purely up to the users, that way lies madness.
Also keep in mind that not all tools are created equal when it comes to dropping everything in to a single folder, we’ve bumped into some possible O(N^2) issues when passing the 120k+ files in a folder with some tools.

As for how we’ve approached it, we went with what I feel is the worst of all worlds.
The filenames matter, but the file-path doesn’t, (outside of a high level package folder).
Also the source and export files can be anywhere as long as they are under the proper package.
The names can also be anything, as long as there is no other file in any package with the same name.
Oh, and we lack a proper system for backreferencing so you can never safely rename something once its in.
So my core advice, is don’t do any of the above, its messy, error-prone, and creates a significant mental burden for your team, and its a very hard ditch to dig yourself out of.

@bob.w Would you go so far as to say Mirrored directories is a superior approach? maybe coupling source and export dirs is a actually a darn good idea?

@BenWall @Munkybutt Shotgrid is actually the genesis of this question. We’re having trouble getting the toolkit schema to shoehorn into our existing source asset structure, and someone asked “why are our source directories tied to our export directories anyway?”

my response is “how else can tools know where to export?” Which I suppose is the real question I’m trying to answer.

A mirrored folder structure is a solid foundation - it won’t provide any further functionality, but people still try desperately to make it so, resulting in it not working for any one thing.
Keep it simple and it is a great starting point.

Putting things in a single flat folder is not scalable, like Bob said. And when it comes to file names, if you have regular reused tokens that group files together, then drop those files in a folder as that is what folders are there for.

The two main things I have learned that need to happen to get a pipe up and running:

  • names and structure should be consistent
    – with this you can automate changes a lot more easily.
  • tooling should automate everything possible
    – Humans suck at following conventions, computers are great at it.

The two go hand in hand, but you can find yourself in a chicken or egg scenario: do we try to create a consistent structure that is geared towards abstraction and tooling and then tool or do we tool it first and tweak the structure as we go.
You would think the latter makes the most sense but you need to know your process to automate it and most production environments don’t provide the room to do either before you are drowning in assets.

The conclusion I have come to over the years is:

  • Mirrored folder structures are a good idea
    AND
  • Make the importance of getting pipeline sorted, known and truly understood amongst the higher ups so they can hire more people than just me to deal with it…

my response is “how else can tools know where to export?” Which I suppose is the real question I’m trying to answer.

I kept it simple on my CGI pipeline:

  • exported items are only tied to the scene they are exported from. Each “context” (task + scene version) has its own workspace, so in caches and images are saved in the task’s output directory (except for textures which only depend on tasks and not on scene versions to avoid redondancy)
    This way everyone always know exactly what has been exported from which scene and it’s pretty easy to backtrack on any retake made

  • the whole “what goes where” is dealt with by the scene build pipeline:

  • i have a json file which maps what should be looked for by each task for building the scene (eg in a shot task render scene I can specifically load assets from assets task shading and anim caches from shot task anim, in an asset rig i’ll load the asset task model and uv/maps from assets task shading, etc)

  • the build process always get the latest exported stuff from the targeted tasks, but users can override the exported version if needed.

This way no need to complicate things with specific export directories : a team export a set of predefined stuff in it’s context, and another team just load stuffs.
If I need to pack the work from the second team, then everything will be copied / exported to this team context directories.

If you need to keep track if what is loaded where or which version of team 1 work has been used to create team 2 work, you can save dependencies trees on Shotgun.

I think any deterministic structure is a "Good Idea :tm: ", mirroring the structure is generally the easiest way to achieve this. As said above, don’t rely on people, people are terrible at remembering complex rules (I am the worst at following these things, because I lie to myself about how I’ll never check anything in).

So generally, the biggest problem with mirroring directories is that you are using the file system to provide metadata, and it’s not intrinsically designed for that, beyond its limited tree-like structure (which it is super good at).

In my mind, if not mirroring, some deterministic mapping between source, intermediate and final assets is 100% essential, these rules can be saved in a database, a configuration file, can be determined based on a series of complex rules, etc…

You’ll probably also want a system for capturing and applying metadata to your assets, if this is going to get buried in the folders and/or filenames, then it needs to be 100% tool driven, because again, people will derp up. Otherwise, you’re probably looking at a sidecar file, database, or container file like I mentioned above.

The other thing to consider, if you are working with source control, is that the file structure ends up becoming a defacto means of access control: it’s easier for an artist to check out a whole folder at once than to hand-pick assets for checkout and checkin.

This is a useful property but it cuts against the other thing that people want out of the file system: semantics. Every art team wants to be able to use the file system to find things they need – and every art team can’t agree about how to organize the folders. Is picnic_table_01 under “furniture” or “outdoors”?

Personally, I tend to lean on the folder structure for access control and use something else, ideally an asset browser with tags instead of a binary tree.

(Just as a preface) : Worked on multiple AAA titles at multiple studios (SquareEnix\EA\ATVI\2K). Nearly all use a mirrored structure. Most are pseud-mirrored:

C:\myStudio\myProject\source\goes\here
C:\myStudio\myProject\someBsDirNeededBecauseReasons\game\goes\here

We setup the toolchains so that as long as the users store the source in the right direction, they never have to choose a filepath\directory again downstream. Tools handle all that for them.

" have seen this tight coupling between directory structures introduce some friction and cost. Differences in what departments want create confusing structures for the other teams, and moving assets breaks things."

In my experience this stems from the “the only thing that’s important is my dept” syndrome. It’s an ego-driven way of running a team, and I steer clear of it / personally don’t allow for it based on my position, where I can.

“Have you achieved a decoupling of the source and exported asset directories?” : Yes, consistently

“If so, how to you make the export paths deterministic and tool friendly?” : Single system-level Python module that handles it all, importable by all DCC tools.

“How do you trace exports back to source files for iteration & debugging?” : since they’re always consistently mirrored, this is easy. but we also embed metadata in our DCC exports (to FBX) that can be read by our engine (Uneral), and have tools there to handle it.

my entire issue with using the filesystem like this is that it tightly couples TWO tree structures instead of just one. And users know how to use explorer/finder and it is way easier to duplicate/move files in there. So even with tools to handle it, that one innocent action can wreck your day. Sure, you can train users to only use tools/engine to move or duplicate, but it will happen, so you’ll need to be prepared or have a contingency plan.

One thing we’ve used on a subset of assets was to make the source part ephemeral. It will still mirror the export structure, but only while you’re actually working on the thing. once you check in, your source files can be deleted as they are no longer trusted. The actual source files are stored in a CAS and pulled down on demand.

While this does have the same issues i highlighted above, it cuts down the exposure time to them.