PDG thoughts (from slack)

A somewhat manual cut and paste of a slack chat about using PDG beyond just houdini

Pierre Augeard Today at 3:29 AM

Hello !
Does someone has some experience with PDG as a stand alone tool for pipeline ? Are there some postmortem from professional available out there ? Thanks

38 replies

Mattias Van Camp 14 hours ago

+1, i’ve been curious but cautious about approaching this thing

Dhruv Govil 10 hours ago

I’ve been curious too but haven’t been able to find any answers

Dhruv Govil 10 hours ago

Ironically, because I’m building a node based tool, everyone keeps saying “why aren’t you using PDG?”

And yet the same people can’t tell me how it’s been working, or how it embeds in to other DCCs etc…

Luiz Kruel 10 hours ago

it works great

Luiz Kruel 10 hours ago

Luiz Kruel 10 hours ago

Let me ask if we have any customer stories

Luiz Kruel 10 hours ago

The jist of it is that PDG is simply a command builder, so it just calls other DCCs commandline, for things like Maya we even have the ability of making a loop where you can open Maya and issue a bunch of commands as a single little burst

Dhruv Govil 10 hours ago

does it integrate inside yet? Like if a user wanted to use it inside an open Maya session to automate tasks

Dhruv Govil 10 hours ago

I have full faith in it as an external scheduler. Tops is already great

Ben Laidlaw 9 hours ago

I’ve only used it standalone to run jobs. Never inside another DCC. I guess I’ve used Houdini Engine for that stuff. Albeit I think we just extended it as a farm manager more than anything.

Luiz Kruel 9 hours ago

I think the Unity Engine is the only one that supports it right now

Luiz Kruel 9 hours ago

VimeoVimeo | SideFX Houdini

PDG as a Pipeline Tool for Small Teams | Pavel Smirnov | SIGGRAPH 2019

This talk is about how a small team could leverage PDG with mixed Maya-Houdini pipeline. We will examine a ROP-based pipeline Griot Groove has been using to get data in and out of Maya, and how the same goals could be easily achieved with PDG.Pavel Smirnov is a Houdini artist and educator based in Tokyo with twelve years of experience. He started his career in Moscow, Russia as a Renderman lighting TD working on commercials and feature films. After moving to Japan in 2011, he created and leads a team of Houdini artists, while developing pipeline tools at Griot Groove, a Tokyo-based VFX studio. In 2014-2017 he taught a Houdini course at Digital Hollywood Tokyo School.MORE HOUDINI HIVE SIGGRAP… Show more

Luiz Kruel 9 hours ago

This video is pretty good overview

Luiz Kruel 9 hours ago

VimeoVimeo | SideFX Houdini

Open Firehawk: Hybrid Open Cloud Infrastructure & PDG | Andrew Graham | SIGGRAPH Asia 2019

Open Firehawk is an initiative by Andrew Graham to automate direct access to cloud infrastructure at lower cost and with more control than prior solutions were able to offer. It also seeks to resolve issues of vendor lock in – it is open source, community focused and seeks to provide as much render power to artists as efficiently possible. Particularly for the Houdini user, it approaches heavy VFX scenarios by leveraging SideFX PDG to handle dependencies across multiple locations and avoids transport of heavy data unless necessary.Open Firehawk automates creation of VFX architecture with open source infrastructure as code. The design is fully automatable, and uses tools like Hashicorp Terraf… Show more

Luiz Kruel 9 hours ago

This one too touches on the cloud bits

Dhruv Govil 9 hours ago

Thanks!

Pierre Augeard 8 hours ago

thanks @lkruel

Luiz Kruel 8 hours ago

Pierre Augeard 8 hours ago

Do you have some knowledge of workflow management platform such as Apache Airflow ?

Pierre Augeard 8 hours ago

Because the scheduler and cloud backend is integrated

Pierre Augeard 8 hours ago

I am curious about the scope intersection between the two frameworks

Dhruv Govil 8 hours ago

Apache airflow doesn’t really work inside a DCC or locally per user right? Similar problems but different domain areas

Pierre Augeard 8 hours ago

no, not inside a DCC, that’s not my use case actually

Dhruv Govil 8 hours ago

Oh could you go into what your use case is? Pipelines an overloaded term

Steve Theodore 8 hours ago

hey folks, when this discussion wraps up it would be a great idea to use /discourse post to copy it up to the website for archiving !

Pierre Augeard 7 hours ago

@dhruv the generic use case is simple. Have a platform to define tasks, and theirs dependencies (as a graph), and to execute them in the cloud

Pierre Augeard 7 hours ago

When the task is “render an image” we call it renderfarm, the idea here is to have this specific use case but any other, such as 3D assets conversion (HD -> LD, FBX -> glTF/USDZ), declination, scene conformation, assembling, quality check, …

Pierre Augeard 7 hours ago

So I was aware of solutions such as OpenCue which was designed as renderfarm at first then can be tweaked to accept any kind of task, or Apache Airflow which was I think designed to handle a graph of task dependencies for data science

Dhruv Govil 7 hours ago

So it’s sort of apples and oranges IMHO.

  • PDG could work great for both local and farm
  • Airflow and OpenCue only work on the farm
  • Airflow and PDG provide UI’s for managing the flow of tasks
  • Only OpenCue afaik allows you to have full resource management etc whereas airflow and pdg would delegate it to the execution systems

Dhruv Govil 7 hours ago

I’m most familiar with OpenCue, since I worked at Sony and we used the in house version (Cue3) for 5+ years.It’s pretty easy to use it as a generic task manager. Just doesn’t have a dependency graph view to it.
But renders and tasks are the same thing for it. It’s just an outline script that gets run on a node

Luiz Kruel 6 hours ago

Yeah, so PDG still needs a scheduler, we ship HQueue and a Local Scheduler, but PDG essentially just builds graphs of commandline tasks

Luiz Kruel 6 hours ago

Seems like Airflow is both the task graph and the farm manager?

Dhruv Govil 5 hours ago

AFAIK airflow scheduler isn’t really a farm manager

Dhruv Govil 5 hours ago

I think you can just give it machines to work with but it’s not at the same level as OpenCue etc for example

Luiz Kruel 5 hours ago

yeah I think it’s more of a basic task scheduler

Luiz Kruel 5 hours ago

does seem like it has some scheduling capabilities?

Luiz Kruel 5 hours ago

but that was one of the pluses we saw on PDG, the earliest versions had us typing up graphs in Python, but the UI is the game changing part of it

2 Likes