Two Guys and a Toolkit - Week 1: Introduction, Planning, & Toolkit Setup


New to Shotgun or Toolkit? Check out Our Story, learn about Our Product, or get a Quick Toolkit Overview. If you have specific questions, please visit our Support Page!

Introduction 

Hi everyone! We are Jeff and Josh, the new guys on the Toolkit team, and we’d like to welcome you to part one of a ten part, weekly series where we discuss our experiences building a simple pipeline using Toolkit. Each week we’re going to walk you through our thought process and how we approach some of the common aspects of building a pipeline using the framework. We will tell you what bits of Toolkit worked well for us, what roadblocks we hit, and how we think things might be improved.

One thing we recognize is that many of you have been down this road before. You’ve built a pipeline with Toolkit, and you’ve come up with some really cool solutions at your studio. That’s why we don’t want these posts to just be a lot of us telling you what we’re doing. We want to know how and why you built your pipeline the way you did. Any input you (the Toolkit veterans) can provide will be tremendous insight for us. We'll do our absolute best to learn from what you all have to say in the comments each week and apply what we hear to this little pipeline we're putting together. We’re brand new to Toolkit, so tell us what we’re doing wrong. The more we know about how you all work, the kinds of solutions you’ve built, and some of the Toolkit best practices, the better we’ll be able to support you in the long run as members of the Toolkit team.

On the flip side, if you’re someone who is considering using Toolkit in your facility and you have questions about how certain pieces fit together, let us know that as well. If we can help answer those questions or address any concerns you may have, we’d love to do so. There’s a good chance that other readers have been, or are, in a similar situation, so please be vocal in the comments and we can get a conversation going.

We do expect that if you’re reading this you have some experience with Shotgun, or at least a general understanding of the role it serves in production. Throughout these posts we’ll do our best to link to pre-existing Shotgun or Toolkit documentation that we’ve used or that we think might be relevant to the discussion. Along those lines, here are some links that we found useful as we were getting up to speed.

Shotgun Support

Getting Started with Shotgun

Shotgun Community Forums

Toolkit Support

Toolkit Overview

Toolkit Example Videos

Other technologies we’ll be talking about and using during this series include Python, Github, Maya, Nuke, and Alembic.

Still with us? Good! Let’s get to work!

Planning

The first thing on our to-do list was to map out how we wanted our pipeline to work. We both came from the visual effects side of production and have had success in the past building subscription-based, pull pipelines. That kind of workflow requires features that are outside of the scope of Toolkit as it exists out of the box, but we would like to explore some of those ideas in the coming weeks. With that in mind, our initial goal has been to implement the broad strokes of our pipeline in such a way that we have a solid foundation upon which to build more complex features in the coming weeks.

We also wanted to minimize the number of content-creation packages we would be dealing with. So far we’re using Maya for everything except for compositing where we’re using Nuke. The reason for minimizing the number of DCCs is that we wanted to put the focus on learning and exploring Toolkit, not the content-creation software. We realize this isn’t a luxury many of you have in the real world, but the focus here is on Pipeline. And while the concepts and features we implement will often be DCC-specific, they should translate well, from a design standpoint, to other software packages.

A very simple representation of the pipeline 

We decided to stick pretty closely to the default Shotgun configuration for assets and shots. On the asset side we wanted to pull the model into rigging (if needed) and surfacing. On the shot side, we wanted to pull either the model (prop) or rig into layout, which would then be pulled into animation. Lighting would pull an Alembic cache from animation and it would then render out frames to be pulled in by comp. Our shot camera would be generated in layout and pulled by each of animation, lighting, and compositing. Surfacing was a bit of a question mark as neither of us are Maya experts. We decided to see if we could get shader networks pulled into lighting (not pictured) and attached to the Alembic caches as a first pass. Another requirement was that we needed to be able to generate reviewable material at each step in the pipeline.

Toolkit Setup

The first step in making this all work was to get Shotgun and Toolkit up and running. We got our Shotgun site set up and started customizing it based on our pipeline design. Jeff set up a couple of assets and a shot and once we had the tasks created for each, we were ready to get rolling with Toolkit.

It should be mentioned that we are in different physical locations, on opposite sides of the U.S. In the coming weeks we might formally explore how our pipeline would work with users in different locations, but for now we’re simulating a shared file system using an auto-synced folder via a free cloud-storage service. It’s not ideal, but suitable for our needs.

Getting toolkit set up for our project was fairly straight forward. There is some excellent documentation for installing and running Shotgun Desktop which walks you through setting up your project’s Toolkit configuration.

One small issue we hit during the process was getting locked out of the being able to define the primary storage location for our project.


Locked paths in the Setup Wizard 

What we didn’t realize was that these paths are shared across all projects and had already been set in the Shotgun site preferences. The docs state this clearly enough (we got a little ahead of ourselves), but this particular page of the setup wizard was confusing. We’ve created an internal ticket for this one to add a little more polish to the UI to clarify why all the paths are locked and to not suggest you can enter new paths.

We ended up creating a github repo (which we’ll be opening up to you guys in the coming weeks) based off of the default configuration and pointing Desktop to that during the install. We knew we were going to be making many changes, so being able to track those in a git repo was a must. We noticed that after Desktop finished with the setup process, there was a working clone of our repo in the newly created configuration directory. This seemed really nice! The next thing we did was go to the Pipeline Configurations page in Shotgun and clone the Primary configuration so that we could each have our own sandbox to play in. As a bonus, the cloned configuration directories also retained the necessary git repo structure, so we were ready to start independent development.

We noticed that the cloned configurations you have access to in Shotgun show up as additional items in the action menus.

Repeated configurations in the Action Menus 

It would be nice if you could customize the Shotgun action menus to some extent or just have cloned configurations show up as sub menus by default. We can see cases where some users, especially support folks, may clone configurations often which could make the menu really long. We created an internal ticket for this as well.

Next up was customizing the launchers for the content-creation software we’d make available to the users of our pipeline. This is typically managed within the pipeline configuration in these two files:

config/env/includes/app_launchers.yml 

config/env/includes/paths.yml


In the paths.yml file we entered the paths to the software we’re making available in the pipeline and cleaned out anything we weren’t using. In the app_launchers.yml file we configured instances of the tk-multi-launchapp with additional launch settings and again cleaned out the configurations for software we weren’t planning on using.

One thing that might be a bit confusing or misleading to folks is the use of the word app. In Toolkit, there’s the concept of an app which is a functional bit of a workflow that runs within one or more engines. The word app is also being used to refer to the content-creation packages. The app_launchers.yml file drives the launching of the content-creation software, but houses various configuration blocks for the tk-multi-launch Toolkit app. There’s another file in the includes directory called common_apps.yml which defines common configurations of Toolkit apps. Long term it might be nice to be more consistent with the terminology. Perhaps it would make sense for the file to be named dcc_launchers.yml and for the Toolkit app to be named tk-multi-launchdcc. What do you guys think?

Next we found the toggle in Desktop that allows you to set the pipeline configuration you want to use when launching content creation software.

Set the configuration to use within Desktop 

This was really convenient, but it did make us realize that desktop only allows launching the content creation software within a project context. It would be nice to be able to set an asset or shot context via Desktop and launch Maya or Nuke within that context; maybe even on a workfile displayed in Desktop. From our internal discussions, it sounds like the Toolkit team anticipated this as well; see all the extra space next to Apps at the to of the Desktop UI? There didn't seem to be a ticket for this internally, so we created one.

Conclusion 

That’s it for this first week. Hopefully you’re able to get an idea of where we’re going with this series. This entry was more about the setup and preparation for building a pipeline, but we’ll certainly be getting more into the guts of Toolkit in the coming weeks. Once again, if there’s anything you want us to cover or you have questions or thoughts on what we’re doing, please let us know in the comments.

Next week we’re going to talk quite a bit about the configuration system and how we think it might be improved. We’ll look at how we integrated our “studio” code with Toolkit and how that could be bundled and referenced with pipeline configurations.

We hope you all have a fantastic week!

- Jeff & Josh

About Jeff & Josh

Jeff was a Pipeline TD, Lead Pipeline TD, and Pipeline Supervisor at Rhythm and Hues Studios over a span of 9 years. After that, he spent 2+ years at Blur Studio as Pipeline Supervisor. Between R&H and Blur, he has experienced studios large and small, and a wide variety of both proprietary and third-party software. He also really enjoys writing about himself in the third person.

Josh followed the same career path as Jeff in the Pipeline department at R&H, going back to 2003. In 2010 he migrated to the Software group at R&H and helped develop the studio’s proprietary toolset. In 2014 he took a job as Senior Pipeline Engineer in the Digital Production Arts MFA program at Clemson University where he worked with students to develop an open source production pipeline framework.

Jeff & Josh joined the Toolkit team in August of 2015.

Labels: , ,

13 Comments


At September 17, 2015 at 11:43 AM , Blogger raitch said...

"Perhaps it would make sense for the file to be named dcc_launchers.yml and for the Toolkit app to be named tk-multi-launchdcc. What do you guys think?"

Yes, please.

 

At September 17, 2015 at 1:09 PM , Blogger Jean Mistrot said...

Hi Josh and Jeff,

Great to see you guys getting started. Jeff I am glad to see you are doing well. Perfect timing for this blog, at least for us anyway! I have basically taken the same steps as listed above as part of an evaluation process that I started on Mon. I also ran into the locked primary dialog confusion as well as some issues with missing authentication credentials during setup but I eventually got things installed. A suggestion for a swap in terminology: perhaps call 'apps' wrappers as that seems to essentially be their function. They wrap executables ( DCCs, or any custom application) in a context specific environment. That is the terminology we use internally and it seems to fit pretty well.

Please excuse my ignorance as I have only been at this for a couple days on and off and I am just gonna stream-o-consciousness blurt out my ramblings....

One of our biggest concerns with integrating our "asset-artifact" tracking with SG db is how slow queries are in general from the python api. As local hosting of SG db's is generally discouraged we are currently caching our production data in a very crude way to get around the latency issues we are experiencing. In support of this methodology we are actively developing a couchbase / memcache solution moving forward. This speeds things up substantially and affords continued work when our SG instance is unreachable

One outstanding question I have regarding the Toolkit is the integration of tools that require configuration but run on the farm or in a batch process. I am assuming we will need to pull out the Tank stuff along with the configs and place them in a central file store to drive configuration of these tools outside of the Shotgun Desktop environment? Further if we are to use a caching solution I noticed that Tank and the Toolkit embed the shotgun_api. Which leads me to think we would have to branch Tank and the Core to wrap sg_api to communicate with our cache?

We also like to context switch inside of our DCCs as we have many projects on the fly and share staff across departments and productions We use a launcher to configure our DCC's which then load an embedded utility that sets production context inside the app itself. This is not a workflow currently supported by the toolkit. It would be useful to create a division in these two different types of configuration.

As an aside: I also noticed that all the SGtk configuration is using yaml files. Is there any reason to employ yaml, ie are there any real data-types in these files that require yaml parsing? I just finished building yet another tool deployment / configuration system here at Atomic and I have found for this sort of thing json using ujson or simplejson to decode is orders of magnitude quicker than decoding yaml and when you are merging a bunch of configs you notice..:) An added benefit is that we can also use our memcache for this information as well.

Anyway I would love to chat with you guys about our experiences and requirements here at Atomic and I look forward to updates to this blog!!! :)

P.S. Josh did you run into Dr. House while at Clemson? He was my thesis chair when he was at Texas A&M.

 

At September 18, 2015 at 8:37 AM , Blogger raitch said...

I would like to see Admin functionality moved right to the Shotgun Desktop UI - to remove the default apps that are not in use like MotionBuilder, in the RMB menu, I would like to see "Remove" Is this clear?

 

At September 18, 2015 at 10:53 AM , Blogger TheBee said...

Hi Raitch,

One of the projects that the Toolkit team will be working on this coming year will be to reduce the amount of config yml file modifications that need to be performed by hand, in a text editor. One of the areas that it seems like we can smooth out is DCC configuration, so that will be one component of what's tackled.

The ultimate goal is to make it so that you can install Shotgun Desktop and get to the point where you can launch DCCs, load work files, and publish without having to touch a config file. A lofty goal, but one we think we can get to.

 

At September 18, 2015 at 12:02 PM , Blogger Jeff Beeland said...

It's great to hear from you, Jean! Josh and I are going through this for the first time, as my experiences with Shotgun at Blur were mostly limited to the web interface for production management, as opposed to artist workflow and pipeline. All of that to say that our collective ignorances are likely very much aligned at this point, as we are all learning this together as we go.

For query speeds, I totally get you. At Blur we dealt with it, as I'm sure everyone else has. Our solution there, which came about for more reasons than just speed, was to build a sync daemon that operated off of the Shotgun event framework to sync all of the data we were interested in to/from Blur's local postgres database. This was put together by Brendan Abel, a long-time TD with lots of Shotgun experience, and then maintained by a number of us at Blur. What it meant was that we didn't ever need to hit Shotgun directly for anything, and instead used the studio's own ORM to access our local database. This is obviously much, much faster than traversing the internet for every query, and it also meant that Blur's DBA could tune the database to suit our needs. The downside, which is the elephant in the room, is the enormous amount of complexity inherent in any system like this. It's a LOT of code with a LOT of moving parts, and so required a great deal of attention on a regular basis. Blur also relies very heavily on the daemon, and so any downtime is immediately noticed and a huge problem. It's a solution, but one that comes with a lot of stress.

Ultimately, there are two things at play: database performance and latency. At the end of every sprint that I've been a part of, the team handling the database talks at length about what they're doing to improve performance and stability of Shotgun. Things will continue to improve on those fronts, I have no doubt. The issue of latency, though, is something that will always be a problem for studios. For data that changes infrequently, a memcache solution of some sort is a possible solution, regardless of where your database resides, but it's not a COMPLETE solution, and never will be.

As for context switching within a DCC, I couldn't agree more. Again going back to my time at Blur, we operated much the way that you're describing. Where a large studio is often going to have dedicated teams for a given project, small- or medium-size studios are required to be much more nimble. At Blur, many of us were working on everything all at once. It is not uncommon there to have 5-6 projects in house all at a stage where artists are lighting/rendering/comping shots, which necessitates individuals have shot assignments on multiple projects at one time. It goes so far as to require assignment changes and additions throughout a given week, so everyone often seems to be working on EVERYTHING. This is not something I had considered when Josh and I were planning how to handle things, simply because we've been operating out of a single project. I think it's worth looking at, though, and so will make a ticket for it as something to explore in the coming weeks.

This is too long, so I have a bit more in the next comment...

 

At September 18, 2015 at 12:04 PM , Blogger Jeff Beeland said...

Continuing...

For farm and batch processing, I think that there are a lot of potential ways to work. One thing that Josh and I have discussed, and that is something we used heavily at R&H, is defining a context via the environment. This came in two primary flavors: commandline context switching, and Perl/Python API context switching. Underneath the commandline access sat an API that could be used from within code/tools to set a context, that then other code that relied on a context would reference. The farm acted in exactly the same way, where the job submitted provided a spec for the context in which to run the job, and the farm manager used the previously-mentioned API to set that context prior to running the job. This is something that Josh and I want to discuss and explore more as we proceed.

I'm going to stop there for now, as Josh will be better equipped to talk about configs. He has handled much of the heavy lifting for us there, which proves how scrappy and resourceful I am when it comes to avoiding the difficult work involved in any development project.

It would be great to keep in touch as you're going through your evaluation, so please drop us an email when you get a chance. We'll do everything we can to give you the support you need, so don't hesitate to ask.

 

At September 21, 2015 at 8:32 PM , Blogger Josh Tomlinson said...

Hi Jean,

Apologies for not responding sooner. As to why the configs are YAML, my best guess is that it's simply about readability. Without a proper UI to manage the tons of environment configurations, I think it makes sense to make the files as human readable as possible. In that sense it's hard to beat YAML. In the long run, I'd love to have a really slick UI for managing all the pieces of the pipeline configuration. At that point, how the data is stored is less important so long as it is fast! :) I touch on this a little bit in the post this week too.

And I do know Dr. House very well! He was my boss the first year I was at Clemson. :)

-Josh T.

 

At October 2, 2015 at 1:03 PM , Blogger Jeremy Brown said...

Hey all,
I definitely agree with the bit about renaming app_launchers etc. to something more fitting with the tk nomenclature. That definitely tripped me up as I was getting into everything.

I'm also happy about eventually being able to bury cloned config items in pull-down menus to avoid clutter.
On that note, is it possible to control who is able to see/execute those cloned items? Maybe this is covered in another doc somewhere (or in a subsequent blog post... I'm working my way through them now) but I definitely wouldn't want another artist to accidentally use a half-finished function I'm working on.

Thanks for all the work and assistance!

 

At October 5, 2015 at 7:01 PM , Blogger Josh Tomlinson said...

Hi Jeremy!

Check out the "User Restrictions" field on the Pipeline Configurations page. I believe the users specified in that field are the only ones who have access to that particular config. If it's empty, then everyone has access.

-Josh T.

 

At October 9, 2015 at 12:43 PM , Blogger Raphael Matto said...

Hi Jeff & Josh,

Thanks for this blog -- it's helping lots as I familiarize myself with Toolkit. Here are a few notes I jotted down while reading through week 1:

* I wasn't confused about the locked storage paths, but I agree a tooltip would help.

* Good idea cleaning up the action menus into submenus. What about having them only show up for whoever is named under User Restrictions for that config?

* I think the dcc/app name-change suggestion is good.

* I wouldn't have found how to switch configs w/Shotgun desktop, thanks for that!

* Hi Mistrot!

Really nice job, good writing style & I like how you're referencing and tying together disparate docs.

Cheers,
Raphael Matto
Shotgun Pipeline Team

 

At October 15, 2015 at 11:54 AM , Blogger Josh Tomlinson said...

Hey Raphael!

Thanks for the feedback!

Just to follow up, I believe the cloned config menus are driven based on the User Restrictions field. Let me know if you find otherwise.

Glad you're liking the posts. Keep the feedback coming!

-Josh T.

 

At November 30, 2015 at 2:39 PM , Blogger Unknown said...

Hey Jeff & Josh,

Awesome info for sure!! Very much appreciated!!

One suggestion I have for dealing with the multiple cloned menu's would be to implement a "filter" style access to these menu's. One could be user based and the others could be [group, department, permission group, etc...] based. This way you could be a little more granular in who has access to what. If it's strictly user login based and you have loads of artists on-board, things get a little more complex especially when adding or removing individuals.

Another thing I'd like to point out is how useful it is to enlist more examples and not assume the community as a whole gets it, whatever "it" may be. A lot of users are visual based and don't have the capacity or patience to sift through loads of legacy and new docs or forum posts that pertain to the beginings of SGTK. Think of it as "give a man a fish and you feed him for a day; teach a man to fish and you feed him for a lifetime".

You guys are totally on the right track!!!!!

I also worked for Alias/Wavefront/SGI/Autodesk for 6 years in a support and teaching roll so my experience with informing the masses with examples came in super handy.

This would also drastically reduce support tickets that come in. Just my .02

Keep up the awesome work guys!

Best,

Dan

 

At December 14, 2016 at 9:46 AM , Blogger Darragh said...

"In the app_launchers.yml file we configured instances of the tk-multi-launchapp with additional launch settings "

Could you please expand a bit on what exactly you did in terms of additional settings in the app_launchers.yml file please?

Best,
Darragh

 

Post a Comment

<< Home

<< Older Posts     Newer Posts >>

Our Story

We are industry folk who love production. A handful of us met while building...
Read More

Subscribe to Our Blog

Follow Us!