Two Guys and a Toolkit - Week 7: Multi-Location Workflows

New to Shotgun or Toolkit? Check out Our Story, learn about Our Product, or get a Quick Toolkit Overview. If you have specific questions, please visit our Support Page!

Multi-Location Workflows

Hi everyone! Welcome back for part seven of our series dedicated to building a simple pipeline using Toolkit.

Up to this point in the series, we’ve been looking at setting up and building the fundamental pieces of our simple pipeline. Here are links to the previous posts, just in case you missed anything:

        Introduction, Planning, & Toolkit Setup
        Configuration
        Publishing
        Grouping
        Publishing from Maya to Nuke
        Dataflow and Workflow

As always, please let us know if you have any questions or want to tell us how you’re using Toolkit in exciting and unique ways. We’ve had a really great response from you in previous posts and we look forward to keeping that discussion going. So keep the feedback flowing!

This week we’re going to take a look at multi-location workflows. This could refer to something as structured as a studio with two physical locations, or as amorphous as a completely decentralized workforce where each artist is working from their own remote location. While the implementations of these two cases might differ, some of the core philosophies and designs will remain the same. We will be discussing some of these philosophies and the features they require. Toolkit provides the customizability necessary to act as the platform upon which a multi-location pipeline can be built, but a specific implementation is not provided out of the box. This week’s discussion will hopefully fill in some of those gaps from a design standpoint.

Scenarios

Let’s discuss two scenarios that are becoming more common in today’s visual effects and animation studios: studios with multiple physical locations, and decentralized workforces without a physical studio location. The former scenario has been common in the visual effects industry for a number of years, and the latter is becoming more popular as cloud technologies evolve. In fact, a large portion of the Shotgun team works remotely, with developers and other staff spread all around the world.

Multiple Physical Locations:

A typical multi-location workflow that involves multiple physical locations usually utilizes local storage at each location, and either a push- or pull-based synchronization of data between each.

Decentralized Workforce:

Decentralized workforces are relatively new in the visual effects industry, but are becoming more common as technology has evolved to allow for the possibility of having a distributed workflow with no centralized, physical studio location. Most of the discussions around this type of workflow revolve around the use of cloud storage combined with web services to build a location-agnostic pipeline. Shotgun is well positioned to act as the backbone for this type of workflow, as the use of a hosted, web-accessible database is a core feature of Shotgun Toolkit.

It makes time travel possible.

This setup would likely involve a series of pushes and pulls to and from some cloud-storage service. Paired with that storage would be an internet-accessible database that is used to track who has produced what data, where it lives within the cloud, and where it would be if it were to be cached on a local disk.

Problems and Solutions

As things relate to Shotgun and Toolkit, we have a few problems to solve. There are likely many different approaches to each, so our ideas should not be taken as gospel. If you have implemented any of this in the past, or if you have a good idea of how you would implement it, please let us know. Your ideas surrounding possible solutions to these problems could very well help set the direction that Toolkit takes in the future, as these are workflows that we would love to better support going forward.

What data should be shared?

There are three types of data associated with a project: configuration data, user generated “work files”, and tool produced, “published” data. Each one of these types of data should be sharable between users in different locations, but when and how that data is shared is different for each.

Configuration Data:

Configuration data, as it relates to Toolkit, would include all of the data within the project’s config root directory. This is mostly comprised of YAML files and all of the apps, engines, and frameworks that act as the backbone of the pipeline. Active project configurations should always be kept in sync for all users in all locations.

As things currently stand, this is a real challenge for Toolkit-driven pipelines. The project’s config typically lives in some centralized location that all users reference, and attempting to distribute that out to multiple locations raises synchronization issues and introduces possible race conditions when multiple people are modifying the project’s config at the same time in different physical locations.

The solution to this problem is a development project that we internally call “zipped config.” The goal is to store a project’s configuration in Shotgun, and put into place a push/pull system for modifying and distributing changes to that configuration.
In the above diagram, if user A were to make a change to the project’s configuration, Shotgun Desktop would be used to upload those changes to Shotgun. Users B and C would be notified of the config changes and given the opportunity to bring their local configs up to date. Should a conflict arise where user A uploads config changes and user B also made changes at the same time, user B would be notified of this when they attempt to upload their changes.

We don’t yet have a release date scheduled for the “zipped config” project, but it is a high priority development task. It is core to Toolkit’s ability to support multi-location workflows, and we will work hard to make it as robust a solution as possible.

User Work Files:

An individual user’s work files will likely need to be shared with other users, but only under certain circumstances. There are definitely situations where artists will share their work files, including the need to reassign work from one user to another, as well as for simple debugging of problems by support teams. Because the need is based on situation, transfer of this data from one physical location to another could be handled as a manual push or pull process. Taking this approach will reduce disk usage in remote locations (or in the cloud) and also reserve available bandwidth for the transferring of data that is known to be needed in remote locations.

Toolkit’s Perforce integration already implements a checkin/checkout system for user work files. It illustrates the flexibility of Toolkit and how it can be used to build similar workflows; that of managing access to shared files. The challenge becomes knowing what workfiles live where, and how to get them. Since they’re not explicitly tracked in Shotgun the way that published files are, some way of tracking workfiles that doesn’t rely on asking the filesystem will likely be required. We’d love to hear your thoughts on this, as we think there might be multiple approaches here should some thought be put into it.

The Perforce integration is implemented as custom hooks for tk-multi-publish, tk-multi-loader2, and tk-multi-workfiles, all of which can be found here.

Published Data:

This is where things get interesting. All published data needs to be accessible to all users, regardless of location. However, when that data is pulled down to local storage can be controlled, since all of the available files are tracked in Shotgun as PublishedFile entities. Where these files should reside on local storage is known, so knowing whether to download from cloud storage (or a remote physical location) can be determined when the file is requested.

The database records tracking these published files will need some additional bits of information populated at publish time. I’m listing a few here, but there are certainly more possibilities out there, so feel free to suggest others and what they could be used for.

1) A checksum computed from the locally-exported file prior to upload to the cloud. This can be used for verification of pulled/downloaded data when a user in a different physical location requests access to the file.
2) The cloud-storage location of the file. The PublishedFile entity will already have a record of the local-disk path of the file, but we will also need to know where to get that file from should we not already have it on local storage.
3) The size on disk of the published file. It’s always good to know how much data is about to be pulled down to local storage before actually doing so.

The order of operations for a publish could look something like this:
The order of operations when importing a published file could look something like this:
The upside to a pull-based workflow for files published in remote locations is that only data that’s requested is housed on local storage, which is an efficient use of disk space. The downside is that users can be left waiting for large data sets to download from another location’s storage before they can begin their work. It’s a trade off, and there are likely mechanisms that can be put into place to ease the pain. If you have ideas, please be vocal!

What about the project root path?

One of Toolkit’s limitations is that it relies on all data being stored underneath a project root directory. When dealing with a single studio location this works quite well, but it can become a problem when distributing users across multiple locations. The requirement is that ALL locations will need to have the same root directory. If the config says that the project’s data is stored in /shotgun/projects/tk_pipeline, then every location needs to mirror that same structure. This will likely not be a problem for studios with multiple physical locations, as they can structure their filesystems the same way across all locations. However, distributed workflows where artists work remotely will also be held to this requirement. Without control over the hardware and OS setup by a central Systems/IT group, the onus is on the individual users to properly configure their workstation to work within the confines of the pipeline.

This restriction does have its benefits, however, because it also ensures that reference paths in user work files and publishes will be consistent across locations. This will act to head off potential pathing problems should work need to be shared.

An Interesting Idea: Per-Location Status

Josh and I have discussed the notion of tracking a published file’s status on a per-location basis. The idea of published file status hasn’t been well explored in Toolkit, but it has the potential to provide some interesting functionality. The way that we envisioned it working would be to have two new types of entities in Shotgun: PublishedFileStatus and Location. Each PublishedFile would would have a PublishedFileStatus entity for each Location that exists. The status entity itself would have a field for tracking the following statuses:

1) Online: The published file is online on frontline storage in this location.
2) Nearline: The published file is online on nearline storage in this location.
3) Archived: The published file was online in this location, but has been archived.
4) Deleted: The published file was online in this location, but has been deleted.
5) Transferring: The published file is in the process of being transferred to this location.

Any number of other statuses could be added to that list to serve the purposes of your pipeline. Having the ability to track these kinds of statuses on a per-location basis can act as the foundation for additional, location-aware functionality within the pipeline. It also provides a great deal of information about the data your pipeline is tracking on a global scale, all accessible from the database without the need to stat files on disk in remote locations.

Conclusion

That’s where we will end this week’s post. Multi-location pipelines is a massive topic, and we’ve only hit the very basics here. This post is the start of a set of deeper, more philosophical topics of discussion that we’ll be writing about going forward. Because we’re moving into this territory, we hope to continue to elicit discussion from all of you out there. The remaining topics cover workflows that are not provided out of the box by Toolkit, but cover things that we would love to better support in the future. As such, your input in these discussions will help drive the direction of future development.

See you next week!

About Jeff & Josh 

Jeff was a Pipeline TD, Lead Pipeline TD, and Pipeline Supervisor at Rhythm and Hues Studios over a span of 9 years. After that, he spent 2+ years at Blur Studio as Pipeline Supervisor. Between R&H and Blur, he has experienced studios large and small, and a wide variety of both proprietary and third-party software. He also really enjoys writing about himself in the third person.

Josh followed the same career path as Jeff in the Pipeline department at R&H, going back to 2003. In 2010 he migrated to the Software group at R&H and helped develop the studio’s proprietary toolset. In 2014 he took a job as Senior Pipeline Engineer in the Digital Production Arts MFA program at Clemson University where he worked with students to develop an open source production pipeline framework.

Jeff & Josh joined the Toolkit team in August of 2015.

Labels: , ,

5 Comments


At November 2, 2015 at 8:33 AM , Blogger raitch said...

Hi,

Keep up the great work on the series!

I have been tinkering with Toolkit & Desktop in parallel and want to suggest a potential topic to explore : How can I add a basic launcher for internal pipeline scripts that can be accessed by the Desktop interface? I would add a screenshot here for clarity but do not see an image upload capability... How can I get this to you?

Thanks!

 

At November 4, 2015 at 7:10 PM , Blogger Jeff Beeland said...

Hey Raitch!

I brought this up in our morning meeting today to see if we already had a tutorial somewhere (I couldn't find one). We do not currently, but I'm going to spend some time later this week or early next week writing up a knowledge base article on it. Once I do I'll post the link here and also send it out to the Shotgun dev email list.

Thanks for pointing it out!

 

At November 5, 2015 at 7:57 AM , Blogger raitch said...

Sounds great - thanks!

 

At November 6, 2015 at 8:17 AM , Blogger Unknown said...

Hi,
Interesting post as always!
How do you think it could also manage these "reference paths in publishes"?
ie. if you want to load a published nuke script, you need to bring the nuke script locally but also any image sequence read by the script. Potentially, you'll need a process looking recursively at the referenced files until everything is here, and could include additional plugins, etc.
Would you need to add a new "reference files" field, or do you think you could just use the existing "upstream_published_files" field for that purpose?
Benoit

 

At January 1, 2016 at 7:18 AM , Blogger Marijn Eken said...

[quote]The upside to a pull-based workflow for files published in remote locations is that only data that’s requested is housed on local storage, which is an efficient use of disk space. The downside is that users can be left waiting for large data sets to download from another location’s storage before they can begin their work. It’s a trade off, and there are likely mechanisms that can be put into place to ease the pain. If you have ideas, please be vocal![/quote]

Especially with your suggested work areas principle, it should be possible to 'predict' what pipeline steps would need input from other steps. Based on the tasks that are a assigned to a certain person, it should be possible to automatically schedule download of material that is likely needed. That how I'd like to implement it.


[quote] ... then every location needs to mirror that same structure. This will likely not be a problem for studios with multiple physical locations, as they can structure their filesystems the same way across all locations. However, distributed workflows where artists work remotely will also be held to this requirement. Without control over the hardware and OS setup by a central Systems/IT group, the onus is on the individual users to properly configure their workstation to work within the confines of the pipeline. [/quote]

At our end, we solved this very problem by using drive mapping. On Windows it's easy by using the 'subst' command to create a drive letter, so the absolute path will be the same on every artists station. On OS X it was a bit trickier, but we ended up using a local NFS share to map the physical location to the same absolute path on every machine. Works like a charm.

 

Post a Comment

<< Home

<< Older Posts     Newer Posts >>

Our Story

We are industry folk who love production. A handful of us met while building...
Read More

Subscribe to Our Blog

Follow Us!