Small studio pipeline question (noob / basic) looking for guidance

pipeline

#1

Hi Everyone.
History:
i am self taught developer (5 years + ). mostly doing simple automations in 3dsmax / foundy nuke, maya etc. i worked at a studio (film / animation) as Technical Artist. My weaker side is (server / client). i have done basic sockets tutorials etc to send simple msgs / chat. done enough Search on topic and almost ready to start.
PROBLEM:
i need to create DAM(Data Asset Management) system for my studio for smoother production. as first step, all artist tools have basic capability built (ready). some ppl call it PAS (Production Asset management) or pipeline as well. If you (reader) less aware of such systems then few links below for solutions available and my comments
SHOT GUN (industry standard) expensive for us
SouthPaw TACTIC (open source) too big for us, they charge XXXX U.S $ for 1 week consultancy setup etc,
Blender BAM (open source) currently Alpha 3 only dedicated with blender, but can be additionally integrated with other apps
STALKER (opensource) base python framework, 90% suitable for us >> (built by another small studio) very similar requirements. having issues in setting it up.
ANIMA(open source) its interface on top of stalker, awesome, built by same studio, (not documented) almost impossible to setup.

REQUIREMENTS:
20 ppl team(artists) on NETWORK. (clients).
they need to connect to server which shows projects (from database) , each project contains sequences / shots
server shows them tasks as well, which they can work on. each task contain details of shots / assets etc, ideally client should take relevant files from server and copy on local pc (artist) which he / she can commit later on (via client)… i have details for data base planned out about what needs to be stored in each table and data mapping etc.

GUIDANCE PART:
all the search and research in last few weeks, i have also found TWISTED and TORNADO etc…(earlier is more relevant).
1, i have invested enough time in tactic at some point and was almost comfortable. my only problem is, understanding repository concept or storage. for small companies like us, each project is few weeks of max few months long, client typically gives us massive footage (few hundred GB’s) we typically put in a shared folder (mount etc) so artist can pick assigned shot from there and work on etc. in my findings, i have to upload / copy all data to its repository(server) ideally i think, its unfair. there may be some solution, but i am unable to find it.
2- i prefer not to write from scratch, so STALKER is too good, almost perfect. except it confuses my few definitions about running server and clients connecting to it (with web or without web). plus how to configure db and config for team etc. orignal team has done that part in anima which is not very well documented. etc
3- if i have to do all that from scratch, i have done lot of search, to me, it seems “TWISTED” is best solution for what i am looking base framework. is it.?? is it really?? can i create server and client in it, which can connect on LAN (TCP) without web and all data (shown in custom UI , pyqt) which i am really comfortable with. this is how they have done in anima as well. plus 1 additional reluctancy is, clients have to copy lot of data (i.e 2 GB per shot, 200 files) in his computer, should i setup FTP protocol for that and normal TCP for basic query / information exchange.

in all that dissappointment, only hope i have is, i have preparred my requirements well, i know exactly where to put data in DB or file structure (mounts) etc…

if anyone in this magical community can find sometime in reading and answering based on his / her knowledgebase, i’ll be extremely greatful for guidance.
thanks,haseeb


#2

DAMAS and ftrack come to mind, but they too cost money.

When writing your own system, the very simplest solution for sharing inside the studio is Windows file sharing - as long as you don’t need to change permissions all the time, and as long as your users are disciplined enough to not manually mess around with files on the shares (but instead use a client program), it should be easier than programming your own client / server system.

At my company we have a similar problem - stock DAMs are too expensive and don’t offer the flexibility we need, and P4 all by itself just isn’t a “pipeline”. We ended up using HTTP protocol. Advantage is that it’s easy to implement a stateless system and that you can use a stock server e.g. Apache on the server side. Requests is a very mature HTTP library on Python which you can use on the client side. On the server side you can use e.g. Django. Other advantage of HTTP, even for small systems, is that you can cache responses - e.g. server replies with data that doesn’t change all the time - which really speeds up communication.

For designing your own system you should really pick up the books “Pipeline Fundamentals” (Dunlop et al.) and “The Game Asset Pipeline” (Carter). They can be very useful for requirements and implementation.


#3

Sir, just wanted to mention huge thanks for your time and valueable thoughts and mostly for sharing openly.

a- i kinda feel same about network share and life normal, team gradually getting better at this. eventually less errors etc.
b- using http apache/django/requests, sounds interesting.
have a good day.


#4

You can a lot with open web frameworks like Flask or Django and a database. I’ve done this 3 times now and it’s totally achievable for TA-level coders.

I would respectfully disagree about using file shares though: it’s too easy for one sloppy contractor to screw everybody else by doing something wrong. The central repository should be owned by a senior person, protected by login credentials, and should include version history and the ability to roll back on individual files or the whole project. Source control is not optional – it’s mission critical for your whole project. You can’t risk a bad mouse click (or a disgruntled contractor) wiping out your whole project! It’s worth the money. Peforce is free for 20 users. http://www.perforce.com/purchase/pricing-licensing

Unfortunately nothing will make working with lots of multi-gigabyte files easy. I’d definitely investigate a system for passing around proxy versions at lower resolutions except when people really need the high res. There’s a reason all those hollywood studios spend millions on custom server farms and private file transfer networks.


#5

[QUOTE=Theodox;28563]
I would respectfully disagree about using file shares though: it’s too easy for one sloppy contractor to screw everybody else by doing something wrong.[/QUOTE]

You need to host it on a Windows Server machine though. Then you can use Windows’ own revision system and backup systems to get some additional security. The biggest problem I see is that it requires admin rights to revert to old file versions - i.e. it’s not something a user can do herself.

I don’t really recommend it (well, it really depends on your use cases and your anticipated user base), but I’ve seen it work in a film studio where users never touched the share itself. Access happened exclusively through scripts and Python code and the users were disciplined enough to not circumvent this rule. I’ve also seen this solution spectacularly fail in a game studio where artists loved to dig in the share via Windows explorer, where people took the approach of “If I can directly access it, why shouldn’t I?”.

If you have users whom you cannot trust, then P4 (or any other RCS) is indeed a better option. Sure you can fuck up the production in P4 too, but I think it’s easier to roll back than with Windows.


Looking for advice on Asset Management for Mid-Sized studios
#6

I was looking at this very closely about two years ago. We tried Tactic and worked with Southpaw on a custom implementation of it. We also tried Shotgun at the same time and I found that even though Shotgun is more expensive and mostly closed source the usability is much, much, much, much, much, much better. You can create database schemas, custom views etc VERY quickly. The support is also excellent. I highly recommend taking a close look at Shotgun. For tooling that is this vital to everyday production, it’s wise to invest in a solution that will scale and comes with great support.

[QUOTE=lalamax3d;28538]Hi Everyone.
History:
i am self taught developer (5 years + ). mostly doing simple automations in 3dsmax / foundy nuke, maya etc. i worked at a studio (film / animation) as Technical Artist. My weaker side is (server / client). i have done basic sockets tutorials etc to send simple msgs / chat. done enough Search on topic and almost ready to start.
PROBLEM:
i need to create DAM(Data Asset Management) system for my studio for smoother production. as first step, all artist tools have basic capability built (ready). some ppl call it PAS (Production Asset management) or pipeline as well. If you (reader) less aware of such systems then few links below for solutions available and my comments
SHOT GUN (industry standard) expensive for us
SouthPaw TACTIC (open source) too big for us, they charge XXXX U.S $ for 1 week consultancy setup etc,
Blender BAM (open source) currently Alpha 3 only dedicated with blender, but can be additionally integrated with other apps
STALKER (opensource) base python framework, 90% suitable for us >> (built by another small studio) very similar requirements. having issues in setting it up.
ANIMA(open source) its interface on top of stalker, awesome, built by same studio, (not documented) almost impossible to setup.

REQUIREMENTS:
20 ppl team(artists) on NETWORK. (clients).
they need to connect to server which shows projects (from database) , each project contains sequences / shots
server shows them tasks as well, which they can work on. each task contain details of shots / assets etc, ideally client should take relevant files from server and copy on local pc (artist) which he / she can commit later on (via client)… i have details for data base planned out about what needs to be stored in each table and data mapping etc.

GUIDANCE PART:
all the search and research in last few weeks, i have also found TWISTED and TORNADO etc…(earlier is more relevant).
1, i have invested enough time in tactic at some point and was almost comfortable. my only problem is, understanding repository concept or storage. for small companies like us, each project is few weeks of max few months long, client typically gives us massive footage (few hundred GB’s) we typically put in a shared folder (mount etc) so artist can pick assigned shot from there and work on etc. in my findings, i have to upload / copy all data to its repository(server) ideally i think, its unfair. there may be some solution, but i am unable to find it.
2- i prefer not to write from scratch, so STALKER is too good, almost perfect. except it confuses my few definitions about running server and clients connecting to it (with web or without web). plus how to configure db and config for team etc. orignal team has done that part in anima which is not very well documented. etc
3- if i have to do all that from scratch, i have done lot of search, to me, it seems “TWISTED” is best solution for what i am looking base framework. is it.?? is it really?? can i create server and client in it, which can connect on LAN (TCP) without web and all data (shown in custom UI , pyqt) which i am really comfortable with. this is how they have done in anima as well. plus 1 additional reluctancy is, clients have to copy lot of data (i.e 2 GB per shot, 200 files) in his computer, should i setup FTP protocol for that and normal TCP for basic query / information exchange.

in all that dissappointment, only hope i have is, i have preparred my requirements well, i know exactly where to put data in DB or file structure (mounts) etc…

if anyone in this magical community can find sometime in reading and answering based on his / her knowledgebase, i’ll be extremely greatful for guidance.
thanks,haseeb[/QUOTE]