Advice on Stand Alone Software creation for Pipeline Tool

Pipelines
#1

Hello Tartists,

I recently finished a 2 year game-programming course.
I have a background in 3D modeling and animation, so my interests naturally leaned towards techArt gubbins.
For my final project I built a rudimentary organizational and auto-bake tool for Maya.

While I Job Hunt in Berlin, I’d like to improve my final project.

I aim to make a central interface via-which a team can origanise, open, declare-state of, 3D (and anim) assets within a pipeline.

Does anyone have advice on what language would be most suitable for the front end of this thing?
Likewise the database which will be needed?
Any likely pitfalls I may encounter?

Any input greatly appreciated.

#2

Python and QT are the go-to language and windowing toolkit for most DCCs, and therefore TA projects. The database should probably be abstracted away to an interface layer, so that different studios can adopt your tool and use it with whatever database system they may already have in place (and know how to administer).

4 Likes
#3

As for pitfalls : the actual pipeline/database part of the system will be usually be fairly simple. Expect UX design issues to take up a lot of time – particularly if you need to learn a new framework in which to implement it.

As an alternative to a QT/Python app – which you’d have to be able to package and reliably deliver to team members – you might consider doing this as a web application instead. A Django or Flask application is a pretty natural conduit between a database and users, and will allow you bypass distribution problems and support remote teams.

3 Likes
#4

This is a cool idea. Are there any examples I should know about of this solution in a pipeline environment? Just curious.

#5

I use a really solid pipeline at work for standalone tools around python 3.6, PySide2 and PyInstaller, I can share a common code base for use in standalone tools and dcc apps. Its easy to compile the tool into a single exe using PyInstaller and convenient for users to have something that’s easy to launch and does not require a local install of python or local management of pip packages.

2 Likes
#6

Thanks @Jeff_Hanna, @Theodox & @leocov , for your input.

Regarding QT: I think I’ll definitely be using the OpenSource version, now I’ve seen the monthly price :smiley:

Regarding web app: I think I’ll see if I can get this thing working as a windows app first, before attempting it as a web-app.

Regarding the code base: That would be great! Share! Share!

#7

HOLY COW!

QT has syntax highlighting and auto-completion for GLSL Shaders!

drool maybe this is a workable environment in which to learn to code shaders properly <3

#8

PySide/Python for front-end.
DB: you pretty much have the choice between relational (MySQL, Postgres, MS SQL Server) and document based (MongoDB, CouchDB, etc.)

Database pretty much depends on the following:

  • will your schema grow - i.e. will different records have different amounts and types of information? If yes, choose a document based db (mongo, couch, etc.), otherwise SQL
  • will you need transactions (i.e. all or nothing operations) while you update multiple tables / documents? This can be difficult to figure out, especially if you’re concerned about consistency and concurrency. i.e. How important is consistency - does it always have to be 100%? How likely are people getting into each other’s way? e.g. 10000 people using the system concurrently will more likely create colliding requests than 10. A banking system may need to be more consistent all the time than a small pipeline for a vfx/games studio where (in the worst case) conflicts can be resolved by talking to the guy next to you. For this problem, you will have to define the risk you want to live with and see how it translates into engineering effort.
  • what sort of queries will be run? i.e. what questions does your database need to answer? Some queries can be very easy to write in one system and very difficult to write in another.

To answer those questions, you should define features for the end-user - what do they want to accomplish? This will tell you what data entities you will have to handle (e.g. information about shots, users, scenes, revisions, statuses, etc.) and what information you will have to get about them (i.e. your queries).

When choosing a database, I wouldn’t worry much about speed. Not at the low amount of number of users and concurrency that a typical single-studio pipeline experiences when it comes to database access. Think: number of users times pipeline interactions per second (you won’t hit 10000 or more :wink: )

Personally, I found MongoDB and MariaDB both very good (and free choices) for databases.
Get some basic Docker knowledge and you can quickly launch different database instances by using the official MariaDB or MongoDB Docker images, to create and re-create database instances as required for development.

Advanced tip: Clients should not interact directly with the database server. This “close coupling” prevents you from efficiently fixing and extending the database, as every database change will require you to update and re-distribute your client. A better approach would be a light-weight webserver (e.g. Python / WSGi) to talk to clients via HTTP, and then translate the HTTP requests into database interactions.

edit: spelling/grammar

2 Likes
#9

FWIW WebGL is essentially open GL and supports GLSL too; you can get completions from a lot of text editors, like Sublime.

So you can do 3d or whatnot in an HTML5 front end too.

1 Like
#10

My only experience scripting shaders is for Unity, in VS. I never found such magic for that. Thought I didn’t look super hard, as I never had much time to fiddle with them.

#11

This is all very interesting and I shall consider what you’ve said seriously.

Interestingly, the basic thing I built for my final project, when I actually put it to use, revealed many ways in which the functionality needed to improve.

You recommend Python variants for the front end. Though others have mentioned QT, which I am currently working through some tutorials of. Do you have thoughts on this?

#12

PyQt or PySide! Best of both worlds :smiley: Python bindings of the Qt library, some are compatible with QtDesigner, most are available for python 3.x. For front end gui stuff it’s the first thing I go to, even for 3d preview stuff - I haven’t had to venture further yet, although admittedly I’ve never attempted something as complex as a full pipeline setup.

1 Like
#13

Basically I think most us would agree on writing as much of this kind of thing in Python as possible, since it’s flexible and extensible.

The front end (however it’s developed) is more of a matter of choice both because people have different preferences and tastes in GUI toolkits and also different levels of stress about deployment. We didn’t go into it but if you were working in, say, a linux based studio where everybody’s machine already has TKinter installed it would make a fine option as well – but if you’re in an environment where you have to package and distribute binaries QT is better.

Web UI minimizes distribution problems – everybody is always on the same version – but it imposes architectural choices not every body can tolerate.

GUI always ends up taking more work than the “functional” part of the code, no matter what tools you use. Frustrating sometimes but a fact of life.

As long as you do a good job of separating the GUI from the backend – which you should always be able to create and run as scriptable command line tools – the frontend tech really does not matter too much. I’d try to focus on that part first and create the GUI once you understand the problem set. Cart > after > horse, not before.

3 Likes