Creating a Pipeline?

Pipelines
#1

Hi everyone! I hope I’m posting this in the right place.

This semester as part of an independent study at college I am going to be developing a pipeline for use within the department and was wondering if those who have experience had any tips or tricks for getting started since I have never created something like this before.

After some thought I was able to list the key things I’d like to accomplish with the pipeline. These are, at the moment, creating/managing:

[ul]
[li]Texture and asset libraries
[/li][li] Tool Creation and maintenance
[/li][li] UDK installs (We have one that is for a 2D platformer and a clean one)
[/li][li] Tutorials for UDK and Maya (as needed)
[/li][li] Perforce
[/li][/ul]

Right now my plan is to start with laying out a foundation/skeletal structure for everything within a week or two and going from there.

The tool distribution and UDK installs will be done through Perforce once I have it set up and will be making tutorials available on a school website.

The one thing I feel like I’m in the dark about right now are the texture and asset libraries. Is there any recommended software for creating and managing a texture/asset library? And in general, is there anything crucial I might be overlooking?

Any and all advice would be greatly appreciated.

I plan to keep this thread updated as I progress throughout the semester, seeing as I’m sure to run into problems along the way.

Thanks in advance!

#2

There are some foundational tasks, like finding some extra libraries for binding to perforce and the like. But really you need to figure out clear specs- what things does the pipeline need to accomplish?

Good luck and keep us updated.

#3

Well that was a long delay in posting, last week was a bit hectic. I started working on getting things up and running and was able to get a few things flushed out with only a few road-bumps here and there.

To start things off, I feel like I was able to get a firmer grasp on what this pipeline needs to accomplish in the end and that is to organize alot of the data we already have (Projects,scripts/tools,tutorials,archives),into a nicer, more user friendly structure. With this it would also include the ability to distribute the content easily through Perforce.

This is most likely to change slightly as I find out something needs to be included or fine tuned within the system, but to get something up off the ground I feel like its a good start.

Along with getting a clearer goal in sights, the past week was also spent setting up Perforce within Amazon which was easy enough with their documentation. Aside from a few bumps here and there with users and permissions I was able to set up a few test subjects, mess around with depot/file creation and get acquainted with basic commands.

All of this gearing up to the first goal I had in mind which was to set up a reliable and organized file structure that we could use as a basis. Sparing a few of the other iterations I tried, I feel the one below was best suited to our needs.

639×558 39.7 KB

First thing to note is that since its going to be used within a scholastic environment it needed a way to structure files in a way they’d it’d be very easy to find for instructors and students alike as well as allowing me to administer permissions in a clean manner with groups.

Starting from the Main Depot:
[ul]
[li]SuperClass - contains only the main class types or courses, merely a contain with organizational purpose. (ex. 3D Design,2D Design,Drawing)
[/li][li]Academic Year - Within the class types it would have the Academic years that are offered for them. (ex. 2010,2011,2012)
[/li][li]Course - These would be the courses offered within the chosen year, the same as the SuperClass. (ex. 3D Design,2D Design,Drawing)
[/li][li]Section - This would list all the sections available in the selected course. (ex. section_01,section_02)
[/li][li]Project - Finally within the projects folder it would list all projects for the selected class, this would contain a further set of folders specific to the projects within the class and contain information or files relevant to it. (ex. Create_A_Scene_Project)
[/li][/ul]
With this I would be able to administer permission easily to groups. The Admin having access to all, Teachers would have access to anything below classes and students would only have access to files within the projects folder. In my mind a very straight forward way of differentiating the levels of permissions within this structure.

For those who have set up these types of structures before or have worked within them, does this seem to be heading in a good direction as far as file structure goes (ease of navigation and access of information)?

Thanks for those who take the time to read and any and all feed back is welcome!

This upcoming week I’m going to be actually putting this structure and system into effect in Perforce and seeing how it all works out along with more general exploration of Perforce in setting up user accounts, groups and depots. Problems are sure to ensue haha

#4

Is the p4 depot going to be used for submitting work for classes, or just as content management for students/teachers’ work?

The structure looks great for a way of handling the first, but otherwise there’s a lot of stuff that you might want to be persistent for a student across classes and years. You can do that with forking and such, but you may want to look at creating something that is more user-centered. I’ve gotta knock out some stuff right now, but I’ll diagram what i’m talking about later.

#5

A little bit of both actually, a way of submitting work for classes as well as allowing the teacher to manage the content within the individual projects. I hope to move this over to a more user-centered layout at some point but for now just something that could manage the information and data I currently have.

Seeing a diagram for what you had in mind would be great though when you have the time!

#6

Another week and more items I was able to cross off the list of setting things up. This time around I was focusing on the actual setup of a test depot and file structure described in my week 1 post. Apart from just learning general commands in Perforce, the setup went fairly smoothly and I was able to get the file structure working along with the correct permissions set for each folder (Student and Teacher access).

The only problem that I really encountered was something about my p4 client settings were off for the Root which prevented me from submitting any files. This just took me a bit of messing around before I figured out the exact problem and got it working. To test it I was also logging on from 2 different computers to make sure it was sticking.

After it was all set up and running in its most basic form we decided that we should have a simple interface to allow Teachers to create specific Projects within the Projects folder easily. Below is the current progress of the tool, using wxpython and trying subprocesses instead of p4Python this time around, I have about a week to create, test and have it working to some degree.

391×521 11.5 KB

So far my thoughts behind the interface is that a user will “Login” when the script is executed through a small prompt. I say “Login” because they are not actually logging into the Perforce server at that moment but it is storing their information for when the subprocesses are executed. From there it will get a list of the folders within the Academic year to populate the Select Course drop down. Then from that it will list the Sections in the Course that was chosen. This will create the file path used when making the new Project directory. The user will then enter a new name for the Project Name and select the users that can be allowed to interact with the new Project Folder from the list box. (At the moment if currently makes a list of all the users on the server, I’m hoping to make a way to make this part less vague). Finally the Create Project button will call the p4 commands through subprocess to create the directory, submit it to the depot and assign the user permissions…or at least I hope it will.

Along with this simple project creation tool I’ve also been giving thought to the structure of the asset and texture libraries that will be built soon. This will most likely be managed by a separate depot with a similar file structure. A superclass containing the types of libraries (Texture, asset, archives) and from there branching off into more specific file types or structures to organize it as best as possible.

If I had any question coming from this week it would be, which is better or preferred when creating code for Perforce. p4Python or calling commands through subprocess? Is it just personal preference or does any have advantage over the other? I’ve heard p4Python can be a bit messy at times.
Also is there a way through the p4 commands that I can get a list of contents within a folder in a depot? I’ve been looking but haven’t stumbled upon anything just yet.

Hope I didn’t ramble on too much this time, Thanks again to all who view and read.

#7

Use the P4Python API instead of calling out to p4.exe as a subprocess. It’s faster and easier. I don’t know what “messy” could mean, honestly.

Here’s how you get a list of all files under a certain folder:

 
>>> import P4
>>> p4_api = P4.P4( )
>>> connect_result = p4_api.connect( )
>>> results = p4_api.run_files( '//tools/some_tool/...' )
>>> results[ 0 ]
{'rev': '7', 'time': '1191869901', 'action': 'edit', 'type': 'text', 'depotFile': '//tools/some_tool/some_file.txt', 'change': '258762'}

“results” is a list of dictionaries, one per file it found. Each dict has information on the revision you have synced, as shown above.

That’s the general model for all commands using the P4Python API. Call “run_command” and get back a list of dicts with the results, where “command” is any command you’d use with p4.exe.

#8

I guess it was just my own badly formed notions, but thank you for that it’s exactly what I was looking for! Ill definitely be switching up and using that instead now.

#9

so no experience with perforce yet(on my todolist) but about your login system:
Just from a UX perspective, which I guess is also a large part of the pipeline:
having a user login and showing it accepted and then using that login data later is a bit confusing if the user enters the wrong information.
Because it’ll seem like a success at first but will fail later. It’s probably best to do a test login first if you can just to make sure it’s accepted.

Cheers. and good luck witth the pipeline.

#10

Thanks dgovil for pointing that out, I was working on it some more earlier and realized this and was able to rework it. Now after the user tries to login with a name or password it will connect to the server and then try to login using the user and password they entered. If it fails then it disconnects and returns to them the reason why, then they can try again. If their username and password are right then it will log them in and open the projectCreationWindow.

This is the function I wrote for it where self.userNameEntry and self.userPWEntry are just wx.TextCtrl boxes.

def _on_login_btn(self,event):
    self.userName = self.userNameEntry.GetValue()
    self.userPW = self.userPWEntry.GetValue()

    p4.user = str(self.userName)
    p4.password = str(self.userPW)

    connect = p4.connect()

    try:
        login = p4.run_login()
        
    except P4Exception:
        for e in p4.errors:
            errordlg = wx.MessageDialog(None,e,'Error',wx.OK|wx.ICON_INFORMATION)
            answer = errordlg.ShowModal()
            errordlg.Destroy()
        disconnect = p4.disconnect()

    finally:
        if p4.connected() == True:
            pcw = projectCreationWindow()
            pcw.Show()
            self.Close()
        if p4.connected() == False:
            pass
#11

that looks like a tidy implementation.
Yeah, after you get all the pipeline setup, I’d go back through and make sure any UI bits make for a good user experience. Not that what you have is a bad UX by any means, but I’m sure as you add front end functionality, new ways of streamlining it will come up.

Our school uses moodle as their online class/asset system and just navigating it gives most people a headache, to the point that some teachers just implemented other alternatives of their own.

#12

Thanks dgovil, so far its seem to have worked out nicely. Just trying to get the rest of the code that way haha.

A third week down and Im back again, this time with an updated depot structure and more refinement on the Project Creation tool. A majority of the time creating the tool has been spend trying to figure out ways to get it to work properly and I think I was finally able to do that, the only thing left really for it to do is actually create the file structure, submit it and assign permissions to the newly created folder(s). The only part that I think might cause trouble or a bit of a road block is determining how I’d like it to assign permissions. I might do this through creating a new group upon the project creation and just adds the selected users to it for the duration (or at least this will be the first way I try).

The latest of the interface is posted below, upon logging in it will get all the years and semesters of the specified depot, then depending on which course is selected it will populate the sections accordingly. The only thing I do need to change is that instead of having a checkbox list to choose users from I’ll have 2 lists, one with all the users and one list that the user can drag the names to. I believe this will be more intuitive and more visually appealing to the users of this tool.

389×589 12.4 KB

Along with this I started to implement the second library depot and got the initial skeletal structure up and running. The layout below is what was decided on and what we will try first. These will be housed in a separate depot I am calling Assets because it will also contain script libraries.

Branching from the main Assets folder in the library we will have to subfolders: Tools and Libraries. The tools library will contain scripts and software that are used within the department. Within the Scripts folder it will be broken down into smaller folders so that it becomes more manageable (which is always lovely). The sub-catagories being: Dillo Tools, Modeling, Animation, and Lighting. Each of these containing tool sets specific to the task where as Dillo tools is a tool set that is used already so it will be manage separately from the categories stated. The thought for this is so that when a new project is created, instructors can assign tools that would be useful to it or to aid in workflow.

Going back up to the Libraries, it will be broken up into: Textures and Models. The textures library will be broken down further into different types of textures (cloth,wood,metal,ect), and even further from there into hand panted or photographic. So a structure might look something like: Libraries > Textures > Wood > Photographic > AMightyFineOakFinish.tga. This upcoming week I will be gathering a few files to test the structures of both of these and hopefully it’ll work out nicely.

Finally a few new things were added to the todo list. The big thing this week finishing up the project creation tool, creating the new Assets and Library depot, and creating a cleanup or revoke script that can be run at the end of the semester to clean up permissions. This would simply just remove the section groups and place all students back at a point where they can only view files and no long edit anything within the depot. This being used to just prevent tampering between semesters while its archived.

A general note just about the process thus far:

Though it has been tough at some points so far, it has definitely been enjoyable and a good learning experience for pipeline and tool creation. Along with what I have been posting here, I have also been documenting my work and thought process through all of this such as my notes, layouts, and brainstorming sessions if anyone would like to see those. I was also talking to my instructor at last weeks meeting about a question that seems to come up from time to time which is, “How do you teach a technical artist?” We both agreed that through this process we are experimenting a bit with the education of a technical artist and if nothing else we might have something to share on the subject at the end of it all.

#13

Got a lot done in the pipeline this week with structures and tools, or at least I feel like I did. The Project Creation tool is finally at a finishing point; it creates the new project in the correct directory within the users workspace and then submits it to the depot. The only thing left to do is figure out how to add users to a p4 group when it is created without having the user manually enter the names in the tmp file.

(A picture to show the final layout)

niDct.png410×631

Along with the creation of this tool I also created a clean-up/revoke script to be used when the projects are no longer active. It just removes the groups and protections created by the Project Creation tool so that users no longer have access to alter or remove files. Right now it relies on a naming scheme generated by the Project Creation tool but hoping I might be able to direct it in some other fashion on the removal of groups.

I was also able to update the structure of our depots and added in the assets and libraries framework and even expanded it to include 2 new libraries of brushes and fonts that can be used for Photoshop. The way I have these structures set up are similar to the way I described in last week’s post. The one I am focusing on getting up and running first is the textures library and model library. An example breakdown of how these are structured:

//Assets/Libraries
//Assets/Libraries/Textures
//Assets/Libraries/Textures/Wood/…
//Assets/Libraries/Textures/Metal/…

I haven’t decided on how I would like the breakdown of the Model library yet though so that is still a work in progress. Thought it might go something along the lines of:

//Assets/Libraries
//Assets/Libraries/Models
//Assets/Libraries/Models/FBX/…
//Assets/Libraries/Models/OBJ/…

Where within the FBX or OBJ directories they’d be broken down and categorized into what type of model they were exactly (vehicle, building, small prop).

I’ve also started the layout of the scripts library, its current organization below (just a block in). This is mostly for checking out scripts, the general though being organization of what type they are so they can be found easily.

//Assets/Tools
//Assets/Tools/Scripts
//Assets/Tools/Scripts/DilloTool/…
//Assets/Tools/Scripts/Modeling/…
//Assets/Tools/Scripts/Animation/…
//Assets/Tools/Scripts/Lighting/…

The next task to accompany these new libraries is to create an interface for users to get textures from the library without having to scroll through the depot and to do this I was thinking of creating an interface similar to the one cgtexture uses. The user can preview textures in a small window and from there download (copy) them from the library depot to a location in their workspace. Below is just a quick brainstorming session I had about how I’d like this to act and work.

The textures will be copied to a client’s workspace since these would be for reference and use and not as much for collaboration, though I will probably need to make some simple way for users to submit textures to the libraries, new or revised. This might be done by just simply redesigning the above to give it that added function as well. The model library will also probably be structured in the same way but use existing model viewer software so that I won’t have to write it myself.

This next week will be mostly brainstorming the implementation of the tool libraries within Maya (how the user would load them, what it looks like ect), cleaning up the current depots a bit and starting to add some test content along with prototyping the texture browser and of course figuring out how to add users automatically to a p4 group so I can finally get the first version out officially to test. If anyone knows how to do this it’d be greatly appreciated. Ill be trying a few more methods I thought of while writing this. That or just general feed back on the current progress / structures would also be great!

#14

I don’t know if many people are reading everything you write, but it is really awesome to see this evolve and you’re going a great job both doing your task and keeping us (and posterity) updated.

#15

agree. You basically, have a really solid beginning-to-end document, ready to be copy/pasted for submission, a good way to get an “A.”

That being said, getting an A, and doing a good job are 2 different things. I would be careful with the restrictions you put on the “tree”. You don’t want to create a “Mac” environment where people don’t feel free enough to make decisions, unless you get it perfect. I would also say think more about your asset tree. Although it’s close, make sure you add things like animation, models, shaders, etc. Think about how these reference each other. Also, you should talk to some artists to see if they would rather setup their own environment. Some like to have a file called, “Boy_1” with all the info below that, whereas others like having all textures for all characters in the same folder. I personally like the former, but I think it’s an arguable point.

Also, how are you storing your info for projects/semesters/teachers/etc? How do you deal with kicking someone out of the project? How long does it take to get rid of them? I think you’ve hit on a lot of points, but know there are more.

keep it up dude!

#16

Thank Rob and UncCheezy I really appreciate the encouragement and I’m definitely in this to not just get an “A” but to create something well structured and useful to others :D:

I will keep the things you said in mind UncCheezy and will think more about the Asset tree. I hadn’t thought of some of those things yet but glad you mentioned them! Right now the user will be able to setup their own environment under each project. Some testing will definitely need to be done though to see how a setup like the ones you mentioned would play out and to figure out what would be optimal for our purposes.

Could you also elaborate a bit on what you mean by storing my information, as in archival purposes or just in general?

Thanks again!

#17

ok, here is something to ponder.

you see what I did here? each project has it’s own assets, unless it’s more of a thing where all projects use the same assets, or you can have both. If it gets bigger, it can become a problem with jumble and shit. (aka artists just saving files, and throwing them in there)

also, maybe having an admin fork, that has the information the tool needs to run. And idk, a lot of this is personal preference, but I like to think of things as “source” and “export” basically a .ma and a .fbx. the source is what the artist works on, and the export is what the game takes in.

ALSO:
http://tech-artists.org/forum/showthread.php?t=1683&highlight=asset+library
http://tech-artists.org/forum/showthread.php?t=57&highlight=asset+library

take some time to read up the forums, I know your busy with school, but there are a shitload of posts that basically og through the same thing your doing. (which is why I don’t think a lot of people have commented.)

#18

Wow thanks for the diagram and explanation. I understand what you mean now and take a look at my structure again. I have also read through both of those threads and they have given me some ideas on where to head with this.

#19

While this week wasn’t that production heavy in content creation I was able to get some more of the conceptual side nailed down in terms of general structure and where this is headed. So sort of a re-group, breather week.

To just start from the top, I reorganized the the structure of both my RTest_Depot and R_Assets Depot. UncCheezy had a good point in the Admin folder being at the top of my main depot that contains the scripts for admins to run. So it now looks like:

//RTest_Depot/SuperClass
//RTest_Depot/Admin
//RTest_Depot/Admin/Scripts
//RTest_Depot/Admin/Projects
//RTest_Depot/Admin/Users

This way all utility scripts for the admin are contained in the Scripts folder. The Projects folder will contain a list of current active Projects in the tree and the Users folder will contain a list of the Users assigned to the tree. This is to just give a comprehensible list of everything to the Admin easily.

The R_Assets Depot was also reworked in a similar way where I removed the Admin directory from there and added a few more Libraries, those being for Shaders and Animation. I also ended up splitting the Models into two sub directories of FBX and OBJ format.

The biggest change in this Depot though was to the Texture Library since that is what I am trying to get up and running next. These directories now contain a subfolder that contains thumbnails of all the images in the specific folder. My thoughts for this being so that when the Image Library browser tool is brought up, instead of generating thumbnails on the fly for the visual navigation, it will just draw from the specific folder. example:

//R_Assets/Libraries/Textures/Wood
//R_Assets/Libraries/Textures/Wood/Thumbnails
//R_Assets/Libraries/Textures/Metal
//R_Assets/Libraries/Textures/Metal/Thumbnails

Along with all these slight changes I’ve also been writing a few simple utility scripts for the admin and have started the layout for the Texture browser. These being:

groupRevoke - This script removes all groups from the projects at the end of a semester, mostly for cleanup since the groups would no longer be needed.
p4pc_point.txt - A simple text document to auto populate some fields in the p4projectcreation tool. Automatically selects the current year and semester.
userProfiles - Updates the users list in the admin folder, Ill eventually add the Projects list into this tool
thumbnailer - Creates thumbnails of all textures in the texture library and saves them to the Thumbnail folder of the respective type. Im hoping to eventually make this a job on the server so that when new textures are uploaded they automatically have a thumbnail created to be used.

More functionality has also been added to the p4projectcreation tool, it now creates subfolders within the new project of all the users that are given permission, less setup the User and Admin would have to do overall.

This upcoming week is when things start to get fun again. As I mentioned earlier in the post I’ll be writing and hopefully have a working copy of the texture library browser up and running in a bare bones form. In conjunction with this ill be looking into some XML and finding a nice open source model viewer that we can use for the models library. This doesn’t seem too hard, just finding one that fits our needs.

I have also just received some of the scripts we’d like to start adding to the depot to be checked out, so I’ll be sorting those out into the respective sub-catagories (Texturing,Modeling,Lighting,Animating) and testing that system out! Overall just working on the scripts to refine them. In addition to this I will also be looking into a way to implement P4 in the Maya environment and coming up with a way for that. Jeremy Ernst had a nice setup of this so I might be modeling it after his.

The next step after this which I had some time to contemplate about is the creation of our social sites and documentation (Tutorials, script documenting) we will also be using but I’ll save that for next time when I have something more concrete laid out.

Sorry for the lack of pretty pictures this time haha :laugh: and thank you again to those that read!

#20

Another week passed and I was able to get to get a fair amount done work-wise as compared to last week. The biggest thing to note is that I was able to finally get a working prototype of a visual texture browser up and running for my texture depot! Currently it draws from the texture library to create a drop down list of all the types of textures and then depending on the type chose it will create a list of all textures within that type. When the Copy to Workspace button is clicked it will save a copy of the selected image to the users workspace, the path of which is defined in the File menu.

The only functionality problem right now though is with how the textures are displayed, it’s currently using a wx.StaticBitmap but as I have learned this will leave artifacts behind if not all bitmaps are the same sizes. I plan on changing this to a paint function and have done some tests with it and this seems to work out well. The only problem with this though is that I’m having a hard time binding the draw canvas to an event, with a bit more messing around though I’m sure it’ll be working. Below is the current interface which will be polished up a bit more this week (Adding more information displayed, fixing some display issues.)

YIUU7.png546×532

Along with making the texture browser this week, I was also able to start setting up the documentation sites and creating a standard layout for how tools and tutorials should be documented and commented for our specific depot and wikis.

Currently for tools documentation, within the actual code I plan on having a header that contains some basic information such as author, date, version, file name, description, and version history. The wiki page would then have the same information but a more in depth description of how the tool would work and some of the actual code (for those who maintain the tools). A sample of what a header might look like is below, I was also considering creating a simple interface tool to auto generate headers for scripts that didn’t have them just to keep things uniform throughout. This tool would just simply be a form where they’d be able to tag the script as a certain kind and just input the information.

“”"
Author : Name (e-mail)
Date : 2/24/12
Version : 0.1.0 beta
Filename : Nameoffile.py

Description : A description of the tool and how to use it.

What is new

Version 0.1.0
+ Changes made
“”"

A lot of this past week and upcoming weeks before GDC will be focused on creating the process for documenting tools on the wiki, setting those pages up, connecting and linking everything, and touching up on the current scripts I have written for the depot. This will also include hopefully writing a few scripts for Maya to check items in and out from the Perforce depot. That way I will have a small system set up that I can start checking to see if everything is working nicely together.

I might have more to add for this week later but currently it’s escaping me. If so I’ll just add it in as an edit later.