My Humble Pipeline [MHP] project

Hi everyone!

Some of you may have noticed that I askedsome thinhs on slack, apparently disconnected from each other…
Of course they are not! :sunglasses:
For years our company didn’t used any proper pipeline: nothing were automated or used to help artists doing the right stuff: each of us did his job, sometimes with different tools, and all that kind of things that a pipeline helps to keep clean were managed by the person himself. We always used version control, but we ever had for example some tool to check for example assets naming or folder (so we often had in Unreal “Graphic” and “Graphics” folder… or an fbx and its textures called in slightly different ways… things like that).
Sometimes this is not a big deal, sometimes generate chaos, especially when (months later) we have to retrieve something or migrate to another project or simply edit and update stuff.

Well… you surely all know the problem! :slight_smile:

So now I’m “forcing” everyone to begin to adehere to a standardized process, starting from the basisc (names and folders).
I’d like to build with time, piece after piece, “My Humble Pipeline”: we are a small company yet so we don’t need huge tools… just something to make life easier and most important our projects cleaner than how they are now.
So if you think it is appropriate for the forum, I’d like to share here my thoughts and reasonings and problems… lots of problems :rofl:

The highest priority problem that my pipeline should solve I think is assets export from different tools: we use both 3dsmax and blender… well, I personally don’t use blender but some artist here and let’s talk frankly… if I can let them use a free software for modeling, I’ll surely let them do so :slight_smile:
3dsmax is mostly used for (besides modeling and unwrap) rig/skin/animation, some other complex stuff and as intermediate tool between cad files and Unreal (we are Unreal Studio Beta and workin quite heavily with its datasmith).Some also use Substance (Designer and Painter) in use and the evergreen Photoshop/Illustrator.
We have UE4 as main destination game engine.
I’m currently writing down some specifics for our assets export and I’d like to automate the process in each of these DCC tools in order to write some metadata inside files. It could be great to use some python code for common parts (for example to get all info about system, user,save timestamp…) and use proprietary script (aka maxscript) only for software specific stuff.

In conclusion… I have a lot of ideas in my mind and I know the task is huge: it will be good however to break it into smaller chunks and accomplish them one by one… keeping an eye to the possibility to link them all together one day.

Can I count on you for a little help and support for this quest?

Bye
Daniele

3 Likes

Here we go!
STEP 1: project structure

We almost always work Unreal Engine 4… I could say 95% of our 3D projects use this engine.
So my first idea is to design the folders structure with Unreal in mind.
We use Perforce and sometimes SVN for version control and for now we include in repos all project files.

The root folder will be simply called ProjectName and will contain everything. It coincides with the folder created by UE4 when creating a new project, so directly under it we find the Unreal Content folder.
The Content has a folder inside called (again) ProjectName: this is due to UE4… helps in case of assets migration and it’s a recomendation by Epic to use stuff from the marketplace.

At the same level of Content I’d like to store all source files for assets and here I have the first decision to make.

Option 1 - use a WorkFiles folder holding all source files (gfx, sounds,…) divided into folders as appropriate but keeping DCC files and “to-import” files together. So .max at the same level of .fbx, for example.

Pros:
. Less folders to take care of and folders trees to mirror
. Sources files alway close to the exported files, so easy to find and edit stuff
. Everything is backuped, versioned and everyone can find files as needed, even months later

Cons:
. From a single source file one could export more intermediate files, maybe with slightly different names and having all together could be chaotic
. Demanding in terms of storage on version control repos
. Non-artists download a lot of art files

Option 2 - use an AssetsSources folder branching into WorkFiles (containing all DCC sources) and ImportFiles (where to export intermediate files). The folder tree is then mirrored between these 2 folders.

Pros:
. Cleaner folders structure
. Uses less storage on server (and on non-artists local drives)

Cons:
. We must take care of 3 different but mirrored folder trees
. Slightly longer path for intermediate files (UE4 has some limitations)
. Must export asset from engine to retrieve the intermediate file
. Some confusion could arise between source and intermediate files (i.e. What is a recorded .wav file? An intermediate without its source?)

In both cases, the folders tree is mirrored even in Content folder.

We talked a little about this on the Slack and I honestly have to say that the idea of keeping sources and intermediate files in separated folders (already in my mind) were reinforced after out talk and the advice (sorry, I don’t remember who of you told me) to NOT version controls the intermediate files because it’s always possible to retrieve them from Unreal with an export: only artist will have the WorkFiles folder and each artist with only his/her personally exported files.

I think that this is a matter to be solved befor planning other stuff :slight_smile:

I’ll say that I don’t actually like many of the ideas in this link:

But it might make useful reading to think how you’d tackle the same thing. My main issue is the overuse of cryptic naming conventions, which I mistrust because I know my artists are among the world’s worst typists. However he does have some good points about top-level folder structure.

Also: I definitely feel like not storing intermediates in source control is a good idea. Unreal stores the same data internally, so it’s basically redundant (and you don’t want to have to write tools to unlock the intermediates everytime somebody re-exports).

In engines where the game keeps only the final packed data saving intermediates is great because it allows the game to be-built without requiring re-exports. Unreal however handles that for you for free.

Hi Theodox!
I’m already following some parts of the Allar’s style guide, even if some ideas don’t convince me.

I’m fine with the top-level folder inside Content, even if having basically 2 folders with the same name bugs me a little (project root and content top-level).

I personally don’t like prefixes for assets names, I prefer a suffix (so at a first glance I can see what an asset IS and not what kind… There are icons and colors for that) but at the studio we agreed to use prefixes because Epic Games itself use them, so in a way we try stay consistent with their assets and marketplace stuff.

What I’m absolutely against is to put textures,materials,meshes and so on in the same folder and rely only on Unreal filters and prefixes: I prefer to have the 3 folders: less stuff in the same directory, easier life to me :slight_smile:
So here everyone agreed with me to keep my dream structure.

He doesn’t give hints about that matter of intermediate files, but your point good! And probably you were the guy who planted the idea in my mind on slak.
I just have some revervations about files like audio or images from 3rd parties or external freelances, when we don’t have a proper “source” to store.

If you’re paying them for the assets, you should make getting the source part of the contract.
Otherwise you’re in the annoying situation of having to return to them every time you want a small change.

1 Like

You’re right… I usually get, in the end, the source.
What I mean is that we often work in partnership with other people or company for some B2B projects (not full games): they provide for example 2D art like UI elements and send us final and already cut images that we import “as is” in engine. I don’t have control on the working process in this case.
Same for voiceover: the partner studio records with the speaker and directly send us cleaned wav files: I honestly don’t know what source I could ask in this case: I’m not very into sound production but I suppose they simply record one or more files and then simply cuts,cleans and saves with proper settings.
From this my concerns about the real nature of a bunch of wav files :slight_smile:

Audio is a dark an mysterious art in my mind.
But there probably exists some kind of master project file, plus all the original recordings.

However in the cases where you don’t have as much control over what assets you get from other teams, get what you can, and version it. You want at the absolute minimum all files needed to recreate the game in case of spontaneous catastrophic asset explosion.

The other thing about audio is that the source files are huge. Almost everybody in our team unmaps them in perforce because they are so big.

A guy here (who is into music and digital audio a little more than me) told me the same thing: the master, multitrack with effects file is usually quite big.

You may have a look at Unreal’s Collections. They take a lot of work from actual folder structure alowing you to have a lot of different asset representations without messing with actual folders. With them you can heavily simplify folder structure as artists can work only with collections and folders remain for the tech tasks. And with that having meshes and textures etc. in one folder is becoming not a such a bad idea because it allows you to see references in assets quickly.

A good way to think about folders is “what would I like to checkin/checkout together?” Folders are a pretty bad tool for discovering or organizing things but they are a good way to let people assert ownership of related files in perforce

If understand your point, the idea is to consider folders like containers of files that are related each others as asset, so looking more at dependencies than logical/types categories objects like (trivial examples) "environments, props, rocks, furnitures…
If so, the idea to store textures and meshes in the same directory seems appropriate… But given a good system for artists to filter assets in a nice way.
Unreal collections as said by @Alexey_Fedorov could help, but only inside the editor: outside it and without an external asset browser it could be more complex to handle files.

Which is what leads many studios to develop their own external asset / resource browser.

This could be a big step for a small studio like us: adding a tool could be perceived like a sort of extra layer of complexity… An unecessary complication instead of an improvement, considering that my “soft revolution” will probably imply already scripted stuff.
Btw, do you know some really simple 2d/3d assets browser? I think that just a viewer with some tags system could be enough…
Don’t know… Maybe developing one (even the most extreme simple form) could be too much effort right now: I think my priority at thw moment could be project structure and assets import/export automation.
( but this doesn’t mean that I discard the idea… It could even be fun to do! :slight_smile: )

Yeah, wasn’t trying to imply that you should jump right on that, in fact you might not actually need one. But this kind of kind of thing is often what leads a studio to build one.

I can always add this later:
Test this new pipeline with a folder structure, collect feedbacks, change for the next time :wink:

Sooo…
I have to think a little more about having or not multiple folders to store related assets like textures and meshes, but this won’t stop me to start working on the first (and I suppose primary) task, that is “export” from DCC.

Speaking about meshes, I’ve listed these requirements:

  • 3D artists here work both with 3dsmax and blender, so the exporter should do the same stuff on both software

  • let’s handle static objects only (so no character or skinned stuff at the moment)

  • I’m not sure how to manage choosing the save folder for the DCC file: should I write a sort of project dependent “wizard” executed when artist create a brand new file?
    Something like “what are you about to create? What’s the object name” + a set of options (indoor prop, environment element, tree, weapon…) and according to this choice automatically save the scene with a proper name in the correct folder.

  • It must check if unit settings of the scene are ok for game engine (UE4: 1 unit = 1 cm)

  • I’d like to export FBXs in a folder derived from the DCC file’s (as said some posts ago, I think to mirror the 3 trees) so the user can’t make mistake once the source is properly saved in the correct place

  • It should store some metadata inside the FBX: I’ve listed CreatorName (first artist who create the scene), CreationDate (year-month-day-hour-minutes), Machine (name of pc used), LastEditBy (artist name who edited the file for the last time), LastEditDate (same format for CreationDate) and SourceFilePath (this one is something I’ve tested, to use in Unreal for automatically open the DCC file to make edit since I’ll probably version only sources)
    I don’t know if there are some informations I’m missing that could be useful in future or if something were redundant (for example the distinction between creator and editor name)

  • It should check name of stuff and warns the user OR automatically correct it. For example according to UE4 naming conventions static meshes must begin with SM_ : should I let call the dcc file without prefix and automatically add it to the FBX at export time? Or do this at first and add to the DCC file too? Do it by script or just tell the user “Ehi! You have to check the name!”

  • It should also sheck the mesh before exporting it. Here I have confused ideas: I think about polycount, materials count, UV channels, scale… Here I need you: what is important to check?

  • [wishlist] Some sort of integration with Perforce (mostly) and/or SVN could be great! Chec in/out sources and fbx without using P4V app should be amazing! (and also useful to avoid the evergreen “Hit save…Oh crap!! File is write only…let’s go on p4v, checkout it, refresh, try to save!” Or forget to mark files for adding.

Besides mere algorithms that is not a problem, my biggest practical problem is how to do this for different softwares and how to handle project related infos (like types of assets in the wizard or specific mesh requirements) if I’ll have to use them.

A lot of stuff here, I know… Don’t kill me!! :smile:

One piece at time, I’m trying to learn something more about pipelines and assets management…
At the moment I’m working on very small scripts, trying to understand what I can achieve and what not, what we need for start and what could be left behind.
Me and another artist (who basically is our principal modeler) spoke a few times and we thought a lot about different options.
I admit I changed a little my opinion on a couple of stuff and I’ll give a try to something else…
Since our primary game engine in Unreal the pipeline is still heavily influenced by it, but I’m trying to keep every door open…

So for now my plan is this:

  • Automated Unreal project creation using a Python script. It will create all needed folder and stuff given a ProjectName. I would like to add here the option to create a P4V depot (or link to an existing one) and a workspace.

  • Folder structure: assets will be stored in 2 mirrored folder structures:a root named AssetsFiles will branch into WorkFiles (containing all DCC sources) and ImportFiles (containing stuff to be imported in engine: no version control for this).
    ImportFiles folder structure will be mirrored under Content/ProjectName/ in UE4

  • Precise naming convention. From the ImportFiles down all stuff will be engine-specific, so for UE4 we’ll follow a custom version of epic one and allar’s. We have short prefixes and suffixes insome cases: stuff created by UE4 itself will probably keep their name (for example given SK_Character we’ll keep SK_Character_Skeleton instead of renaming it with _SKEL or something similar)

  • Against my initial “hostility” we’ll try to keep textures,materials and meshes in the same folder. As suggested by @Theodox we’ll leave the idea of “what stuff must always travel together” idea rule on folders… Just try :slight_smile:

  • Write a custom exporter for Blender in order to control the fbx production.
    It will have some simple features:
    1- Option to export all or selected meshes
    2- Option to export one fbx file with all meshes inside or multiple ones (one for each mesh in blend scene)
    3- It will automatically add prefix to the FBXs, so SM_ for Unreal (I don’t want to force the artist to name stuff for a specific engine: source working file should remain engine-agnostic)
    4- When exporting a single fbx it will have the same name of the blend file: when exporting multiple files (like for a “set” of props) the name will be SM_blendFilename_meshName
    5- It will save as metadata ArtistName , LastEditDate, SourceFilePath (path to the source .blend file)
    This is half done but it completely lacks of any UI: it’s just a script and a bunch of functions

  • Write a custom importer for Unreal: this is half done too. It imports stuff in the right folder (creating it if not existing), identify file types and performs simple operations (like changing texture compression based of name suffix

I know these parts are way far from be finished… this is just my starting point! :wink:

One thing I’d consider is how to remove options from the exporter. It’s usually better to take any human decision making out of it where possible; otherwise the game data exported by the same file can vary wildly based on user decisions (or errors!) and you’ll have no way to recreate the exact state of the scene at export time.