Two Guys and a Toolkit - Week 2: Configuration

New to Shotgun or Toolkit? Check out Our Story, learn about Our Product, or get a Quick Toolkit Overview. If you have specific questions, please visit our Support Page!

Configuration

Hi everyone! Josh here. Welcome back for part two of our series dedicated to building a simple pipeline using Toolkit.

Last week we gave a quick introduction of the project and talked about the early phases of design and set up. This week I want to go a little deeper into the setup and management of a Toolkit project and talk about the configuration system. Since I’ve been responsible for the pipeline configuration for our little pipeline, this will be a brain dump of my thoughts and experiences regarding the configs. I’d love to find out what you all have done as far as config customizations go and how you manage them on a daily basis across multiple projects and/or locations.  We got some great feedback in the comments last week, so let’s keep the conversation going!

Here are the links I used to get up to speed with the configuration system and to better understand how it works within Toolkit:

Basic Configuration

Managing your Project Configuration

Also, if you are interested in following along on the development side, you can find our simple pipeline config repository here. We don’t recommend anyone actually use our config yet, but we’ll be referencing it throughout the series.

Exploration

As we mentioned last week, getting the initial configuration installed and working for the pipeline was relatively painless. Using Shotgun Desktop was very convenient and we were able to launch DCCs in no time.

The next step was actually configuring the pipeline to behave the way we wanted. After reading the docs, I decided to just dive into the config directory structure, take a look at all the files, and try to get the “lay of the land,” so to speak.

In the core directory I found the templates.yml file and the schema directory tree, which are used to map Shotgun entities into the desired file-system structure. There was a lot to take in as far as the documentation for this portion of the config. To keep things simple for us, I cleaned out some of the references to DCCs we wouldn’t be using and left the rest of the structure largely unchanged.

The one thing I did add in the schema hierarchy was deferred folder creation files for the work area DCC directories. The default config does not include these files by default, so when you create the directories for an asset or shot on disk, you get many more directories than you likely need. Adding these little configuration files in there defers creation of the DCC directories until the engine is launched within that context. My preference would be to have this as the default behavior, but I’m curious what you guys think.

Deferred folder creation file

The other big chunk of the configuration structure is the env directory, which houses the various environment configs that drive what engines and apps are available for a given context. A core hook called pick_environment is used by Toolkit to map a context to one of these files (with the exception of the shotgun_*.yml files which are mapped by convention from within shotgun).

One of the things I’ve found really convenient about working with the environment configurations is the location setting available on each of the engine, app, and framework bundles. The location setting makes it really easy to pull the bundle from a local directory, a github repo, or the Toolkit app store. This makes development and testing changes to an environment configuration very convenient.

After browsing through these environment files some more, the thing that stuck out the most was the amount of repetition in the settings. Take look at the asset_step.yml and shot_step.yml files from the default config, for example.


Repetition in the config files

You’ll notice quite a bit of repetition in the engine configurations of each file. You’ll also see the tk-multi-* apps being used by several engines in each file, again with a very repetitive list of settings. After looking through these, I had a some thoughts about how to make managing these files a better experience.

Inheritance Experiment

My first thought was that I’d much prefer an inheritance scheme where I could define common settings in one file that I could include into an asset or shot environment. From there I could override the default settings with only the settings that differed for a given context. The config system already has an include mechanism, but it updates at the key level of the config dictionary rather than doing a merge with value-level overrides. So I decided to see if I could hack into the Toolkit core (which sounds really dangerous and awesome) to see if I could get the includes to do what I wanted. After fiddling around with it for a bit, I was able to get it working. You can see the changes in this branch of the tk-core github repo.

I didn't take the time to map the entire default config into a new organizational structure, but I did run some tests on a simple inheritance-based config, and it seems to work as expected. The changes allow you to define an engine and its apps with default values in one file, and then include that into an environment config that only overrides the settings that are unique to that context.

Includes overrides

You can browse these modified environment configs in this branch of the tk-config-simple repository.

I think this type of organization could make environment configs easier to manage. For example, when you want a specialized environment for one sequence on your project or for an animation step on one shot, you wouldn’t need to duplicate an entire config.

I should also note that I haven’t fully tested these ideas, and I’m not exactly sure how bad this would break the tank command. It was an interesting experiment, and it helped me learn more about how the includes worked and what goes on inside of core. I’m curious if any of you have done similar experiments or if you have an environment config setup that works well for you.

Management UI and API

Having looked through the configs, and played around with their organization, it seems to me that the best long-term solution would be to not have to manage them by hand or via the tank command at all. It seems what Toolkit really needs is a slick configuration management interface. It would be neat to have features like drag and drop installation of engines/apps/frameworks from the app store, push-button configuration deployment across multiple projects or locations, and visual indications when apps/engines/frameworks are out of date, just to name a few.

I’d also like to think about an environment configuration API that wasn’t file-centric. From a conceptual standpoint, how the data is stored under-the-hood shouldn’t really matter and we should be interacting with an API rather than files on disk. I’m guessing that if there was such an API, and perhaps some modifications to the pick_environment hook to accept an environment object, some of you might build your own slick UIs or come up with clever ways to organize and store your environment configurations. Then again, maybe you’ve already done that. If so, I’d love to hear about it!

Taking things a bit farther, in the long term I’d really like it if we could all share the bundled configurations (and other pipeline-related tools, scripts, widgets, gizmos, etc) that we’re building in a pipeline marketplace. Shotgun has the Toolkit App Store, and Autodesk owns Creative Market; what if we could build a community around pipeline-related content? Imagine working in Maya and being able to install a toolkit app, try it out, comment on it, rate it, or even suggest improvements for it, all within a single session!

Conclusion

This seems like a good place to wrap up this week’s post. It was fun getting the config into a semi-working state with the stripped down engines and apps, and the deferred directory creation. Tinkering with the Toolkit core was also an interesting experiment that we may end up fully fleshing out in the coming weeks.

Right now, I’m still down in the guts of the config system, making changes and trying to learn how some of the more subtle bits and pieces work. Feel free to watch the tk-config-simple repo if you want to follow along with the development over the coming weeks.

I hope this week’s post gave you some insight or perhaps sparked an idea in your mind about how we might move forward. I’m excited to hear your thoughts on and experiences with Toolkit configurations and to learn how you manage them at your facility.

I know last week we teased our studio code integration, but that’ll have to wait for a future post. Next week, Jeff will be giving you the low down on his experiences with file publishing and how data flows down the pipe from one step to another.

Updates from Last Week

Hey gang, this is Jeff. Last week’s post elicited a handful of comments in the blog post itself, but also a bit of feedback in support ticket conversations that revolved around some of the same things that we are trying to achieve with this project.

In last week’s post, I think we could have provided a little more detail in the section concerning cloning a pipeline configuration to create a development environment. It is a very useful feature and one that we are using daily, but there is some information that we failed to mention. After cloning the primary pipeline configuration, things should be up and running. However, when I set this up for myself, I immediately removed the cloned config directory  and created a config symlink to a clone of our tk-config-simple repo in my development area. This mostly worked, but there were autogenerated files that were part of the project’s primary config that were created on project setup.. These files had to be copied across from the primary config into the repo that I had linked to, and we added them to our .gitignore file since they shouldn’t be kept under revision control. This step is not immediately made obvious by the documentation, and so it could be a bit of a gotcha for new users.

Below is the list of ignored files that I copied across for reference. Hopefully it will help some of you get past that gotcha with a minimum amount of frustration.

Our .gitignore which includes the auto-generated files from the primary config

About Jeff & Josh 

Jeff was a Pipeline TD, Lead Pipeline TD, and Pipeline Supervisor at Rhythm and Hues Studios over a span of 9 years. After that, he spent 2+ years at Blur Studio as Pipeline Supervisor. Between R&H and Blur, he has experienced studios large and small, and a wide variety of both proprietary and third-party software. He also really enjoys writing about himself in the third person.

Josh followed the same career path as Jeff in the Pipeline department at R&H, going back to 2003. In 2010 he migrated to the Software group at R&H and helped develop the studio’s proprietary toolset. In 2014 he took a job as Senior Pipeline Engineer in the Digital Production Arts MFA program at Clemson University where he worked with students to develop an open source production pipeline framework.

Jeff & Josh joined the Toolkit team in August of 2015.

Labels: , , ,

9 Comments


At September 29, 2015 at 3:01 AM , Blogger Marijn Eken said...

"The one thing I did add in the schema hierarchy was deferred folder creation files for the work area DCC directories. [...] My preference would be to have this as the default behavior, but I’m curious what you guys think."

Yes, that's the first thing I did as well. Who needs a slew of empty directories? So you have my vote on making this the default. But how would that work? You would need to define somewhere which directory maps to which dcc, right?

 

At October 1, 2015 at 12:01 PM , Blogger Josh Tomlinson said...

Hi Marijn!

Great to get your input!

I was thinking of just having a deferred creation file for each DCC in the default config schema tree's work directories. The one in the blog post is maya.yml, I was thinking of just including that one (and nuke.yml, houdini.yml, etc) in there by default. Each would have the defer_creation field set to the corresponding engine. Would that be enough or am I missing something? :)

This just made me think that it would be slick if there was an easy way to identify the DCCs you planned on using for a project as you're setting up desktop. Maybe just a checklist of DCCs you could select and then the configuration and schemas were setup accordingly. Would be nice if you could go back and add new ones dynamically as well.

Thanks again!

 

At October 2, 2015 at 2:46 AM , Blogger Unknown said...

Your modified include with overrides looks very cool. We're using a lot of includes in our config and one annoying thing is that it breaks the "tank install" command.
Does your solution break it as well?
Also, the "tank updates" command still sees a lot of different environments and ends up checking again and again the same files through the includes. In the end, it takes 10-15 minutes to perform a check so we had to rewrite a more clever version that just reads all the versions at once and then check them one by one.
Thanks for your posts!

 

At October 5, 2015 at 11:23 AM , Blogger Marijn Eken said...

Hey Josh,
I get what you mean with the 'default settings' now. Yes, the way you explained it, is what I would expect to find. Automation of the configuration would be most welcome. I was just adding a new launcher right now, and couldn't figure out why it wasn't showing up in the web interface. Until I realized I was adding it in the wrong file. I think a UI would help prevent that kind of mistakes as well.

 

At October 5, 2015 at 7:10 PM , Blogger Josh Tomlinson said...

Hi Unknown! :)

I have a hunch that my test implementation does not play nicely with the tank command. I tried my best not to change the API, but with different chunks of a config being split up that way, I think the best case scenario is that it dumps the entire block being installed into the last resolved file - which sort of defeats the purpose. I didn't do much testing with the tank command itself other than 'cache_apps' and 'launch_app'. Ideally it would have some smarts to only add/remove bits from appropriate places on install, but I didn't have the time to go that deep.

10-15 minutes, wow! We should use your config setup as a stress test case! :) Your version sounds cool. I need to dig into the updates action to see exactly what its doing. Thanks for the tip!

-Josh T.

 

At October 9, 2015 at 12:45 PM , Blogger Raphael Matto said...

Hey guys,

A few more notes for you:

* I like deferred folder creation being on by default, seems obvious so I wonder why they didn't do it ...

* I like the config inheritance idea a lot, seems obvious so I wonder why they didn't do it ...

* I'm not sure why you're setting up config symlinks to a dev area, why not just work directly in the clone?

Cheers,
Raphael Matto
Shotgun Pipeline Team

 

At October 15, 2015 at 12:05 PM , Blogger Josh Tomlinson said...

Hi Raphael!

With regard to the deferred folder creation, keep an eye on the tk-config-default repo. Some changes may be coming... ;)

The config inheritance idea I think would work well for people who enjoy manually editing configs and feel comfortable doing that. I think it also introduces a level of complexity when it comes to writing an API that supports that type of thing. I think in the long run a nice UI will make everyone happy. :)

The symlink thing I think was just a personal preference. I cloned tk-config-default into my pipeline configuration area and work from there directly. I typically have one directory where all my dev repos live, so this is a bit out of character for me. I'm guessing Jeff works that way and just decided to symlink to the repo elsewhere. He can correct me if I'm wrong. Seems like the end result is the same though. :)

Thanks again for the feedback! Love it!

-Josh T.

 

At October 23, 2015 at 10:07 AM , Blogger slip said...

Hi Jeff and Josh,
I am enjoying reading your blog posts, its great to have you two as a part of the Toolkit team. I am currently implementing the "Global Configuration Using Includes" technique as documented here:

https://support.shotgunsoftware.com/entries/95441337-Managing-your-Project-Configuration#A%20global%20configuration%20using%20includes

and your ideas on configuration inheritance are very much in-line with what I was hoping to achieve. I took a look at your git tk-config-simple repo - and, even as only a proof of concept, its very exciting. I was wondering if there has been any internal discussions as far as supporting it in the release branch of the core? Did you end up breaking anything in your tests :) ? For my current production release, I am going to continue to use the current 'Repetition in the config files' setup, but moving forward, I would really like to be able to trim down the yaml files and use a truly inherited paradigm.

Thanks again for the blog posts,

Seth

 

At October 29, 2015 at 12:09 PM , Blogger Josh Tomlinson said...

Hi Seth!

Glad you're enjoying the posts! I have created an internal ticket for discussion of the config inheritance. I will definitely report back if that ends up on the roadmap. My expectation is that the longer term solution will be to keep people out of the configs all together and provide a nice interface for setting up your project. So I guess it's really a cost/benefit question for the short term feature vs development resources toward the long term goals. Like I said to Raphael above, it does seem very powerful but also adds some complexity as well. I'll will follow up with the team and report back if anything changes.

Thanks again for reading!
Josh T.

 

Post a Comment

<< Home

<< Older Posts     Newer Posts >>

Our Story

We are industry folk who love production. A handful of us met while building...
Read More

Subscribe to Our Blog

Follow Us!