Two Guys and a Toolkit - Week 8: Subscriptions


New to Shotgun or Toolkit? Check out Our Story, learn about Our Product, or get a Quick Toolkit Overview. If you have specific questions, please visit our Support Page!

Subscriptions

Hi everyone! Welcome back for part eight of our series dedicated to building a simple pipeline using Toolkit. Here are links to the previous posts, just in case you missed anything:

I’d like to start out this week’s post by looking at a few common production questions.

- What version of the rig is animation using?
- Who hasn’t pulled the latest model?
- What version of the plate should I be using?
- How can I revert to what I was doing last Thursday?
- Why don’t the Lighting and FX elements line up?
- What version of the animation was I looking at in dailies yesterday?
- Disk space is getting low! What can I clean up?
- This shot is going to be rendered in Vancouver, what files need to be transferred?

Do any of those sound familiar? I’m willing to bet that if you’ve been on production for a while you’ve heard at least a few questions like these on a fairly regular basis. 

The purpose of this post is to talk about the concept of Subscriptions and how they can be used to answer, or help answer, these kinds of common production questions, across multiple locations, and without hammering your production disks.

Jeff and I spent the majority of our careers working with some of the best pipeline engineers around building and supporting subscription-based pipelines. From experience, we know that there can be a lot of moving parts when it comes to subscription-based workflows. That's why we didn’t try to tackle them while implementing our simple pipeline. It’s safe to say that we each feel strongly that the benefits of tracking subscriptions far outweigh the overhead on a medium to large size production. Hopefully this post will outline some of the benefits we see and make it clear why we are subscription fans. It will also be interesting to consider as we go along how subscriptions-based workflows could be implemented using Shotgun and Toolkit.

As always, these posts are aimed at starting a discussion with you all. After reading the post, if you have a strong opinion either way regarding subscriptions, or if you have any questions or comments, we’d love to hear them - especially if you’ve implemented a subscription-based workflow already! 

We linked to it earlier in the series, but If you want some more information about subscriptions and how they can be used to build sophisticated workflows, see this talk from SIGGRAPH 2014.

Work Area Versioning

Before we get into the guts of how subscriptions work and their benefits, I’d like to talk a little bit about versioning. Toolkit handles versions at the file level. You can version up your work file, and you can publish new versions of your work for folks downstream. This can work well when you have a single file that contains all of your work and when you generate all your publishes from that single file, but I think it’s also fairly common for an artist to be doing their work in multiple files and multiple DCC applications simultaneously. 

An FX artist may go back and forth between Maya and Houdini, and a matte painter may bounce between Photoshop and Nuke. An artist may have auxiliary files with shot-specific Python or MEL that they’re reading in and executing as a part of their own day-to-day workflow. An artist may also have multiple work files for a single DCC referencing each other within the work area that they're using to generate various outputs. 

In these scenarios, having to manage versions on individual files is not something that an artist should have to deal with. At the same time, keeping track of the state of those files on production, and having accurate snapshots of what an artist was doing at a given time, is extremely valuable. 

What I’d like to be able to do is to version the artist’s work area and all of its associated files as a whole. It should be possible to snapshot a Maya file, a Houdini file, and anything else in the work area directory together. This way, if a user models something in Maya, deforms it procedurally in Houdini, then versions and publishes it, they have a saved state for all the files they used to generate that output.



In the diagram you can see that when you track versions on individual files, you don’t know how they’re related over time. If you version the work area as a whole, you can see which files are associated with each other and when files were added or removed from the work area.

Tracking the changes across a collection of files should sound familiar if you’ve used Git for revision control. In fact, it would be a really interesting exercise to think about a Git-backed system for managing the state of a user’s work area over time. Anybody doing that on production?

But I digress a little bit. To implement this in Shotgun, we would add a custom WorkAreaVersion entity. It would include metadata such as:

- A description of what changed for that version
- Who created the version
- When the version was created
- The Location the version was created in (Using the custom entity Jeff proposed last week)

We would link our publishes to the new entity as well so that we could properly traverse our production data once we started populating subscription data.


As you see in the diagram, the version number of the publish matches the version number of the work area. Whether this is implicit or explicit in your data model, it is an important association that helps artists quickly map between the publish file and the state of the work area it was created from. 

The rest of the post assumes we’ve put the WorkAreaVersion entity in place and we have a strong association between the work area and publish versions.

What are Subscriptions?

We define a subscription as an association between a version of a work area and a version of an upstream publish. 



Subscriptions track the usage of upstream publishes over time:

In this example, we see the history of the hero comp’s subscription to the camera from layout. You can see the compositor was using version 7 of the camera in version 23 and 24 of her comp, but updated her subscription to use version 10 of the camera in version 25 of her comp. 

In Shotgun, this could be modeled as a custom Subscription entity with fields for the PublishedFile being subscribed to and the WorkAreaVersion that is using it

Once we begin building subscription records for each version of our work areas, we can start building very powerful views into our data. 

Subscription Dataflow

High-level Views

Without subscriptions, we might be forced to parse work files to see what the inputs are for a given work area or collection of work areas. That type of pipeline introspection does not scale well and becomes problematic when you have a multi-location setup. With our populated subscription data, we can look at snapshots of our pipeline’s dataflow at a high level. 

We could do a simple query to ask what version of the hero rig shot ab013_13’s hero animation is using. We could also run that query against all the animation tasks within the sequence to quickly see who isn’t using the approved version of the hero rig for the main character. Going even further, since we know what the latest versions of published files are (or perhaps we track other publish file approval statuses), we could simply ask the system to provide a list of all the work areas on the project that aren’t using the latest versions of things they’re subscribed to. 

Upstream Conflict Identification

Review sessions can take advantage of subscriptions as well. It is possible to write tools that build the complete upstream dependency graph for a given work area. 


If, in Comp dailies, the supervisor wants to know why some elements don’t line up, you might query the dependency graph, tracing the subscriptions upstream to see that lighting was using version 11 of the layout camera but FX was using version 13. After hitting this situation a few times, you might consider writing a tool to warn about potential conflicts before the compositor renders. You might even go as far as writing a tool that does a conflict resolution pass to warn artists as the subscriptions are created. 

Identifying Potential Issues Downstream

We can also look up data in the other direction. Starting with a version of a published file, we can ask what work areas are using it.


A layout artist may see that the latest version of the camera for shot ab013_14 is being subscribed to by the animator on shot ab013_15. There may be a perfectly good reason for this, but it would certainly raise a red flag. Being able to see exactly where publish data is going can be extremely useful in identifying potential problems. 

Cleanup Candidates

Another benefit to seeing what versions of a published file work areas are subscribed to is for identifying things to be cleaned up. 


If I see that version 16 of a very large Alembic cache from animation isn’t being used by any recent versions of lighting, then it may be a good candidate to be cleaned up. Jeff is going to go deeper into cleanup next week, but it’s worth pointing out the power of being able to ask your pipeline what is and what isn’t being used currently across all of the studio’s physical locations.

Subscription Dataflow with Shotgun

From a data standpoint, I don’t think there’s much in the way of preventing someone from modeling subscriptions in Shotgun. The concept is fairly simple assuming you have the work area versioning in place. I’m curious what you all think about tracking subscription data in Shotgun. Do you think it could help your studio? Does it seem overwhelming based on what you’ve read so far? 

Next, let’s look at subscriptions from a workflow perspective.

Subscription Workflows

Managing Subscriptions

So how do artists actually populate and manage subscriptions as they work? We looked at Toolkit’s Loader in week 6, and I think it is reasonable to imagine adding a hook to create a subscription as an artist references a publish into their work file. It’s also easy to envision the Breakdown app updating subscriptions for the current version of the work area as it updates references. 

But there are problems with using the current set of tools if we’re working in a versioned work area scheme. Since subscriptions are at the work area level, updating a subscription to version 10 of the camera would imply that all my work files should now be using version 10. Ideally, a subscription update would immediately update the references across all my DCCs.


There are a couple of ways to approach this issue depending on the scenario. You could have a way to manage subscriptions outside of the DCCs in the way of an interface that shows you the current version of your work area, what it’s subscriptions are and which ones are out of date. Since managing subscriptions at this level only affects the database and not the working files, there would need to be a way to auto-update the references as the DCCs were loaded. This could be implemented by having custom startup code that checked references and compared them to the subscribed version. 

A more ambitious solution might be to implement a custom url scheme for referencing subscribed files. A custom url evaluation phase would attempt to do a best match on the work area’s subscriptions and forward that path onto the DCC for loading. This would allow you to reference your subscriptions without an explicit version number, meaning as long as you are subscribed to a matching publish, the DCC should load the correct version. In other words, you wouldn’t need to change the reference when the subscription was updated. Just be wary of doing this type of thing on the render farm as you probably don’t want subscriptions changing mid-render. You also don’t want to hammer the database with queries while trying to evaluate subscriptions for each frame being rendered. You’d want to consider a cached subscription layer for evaluating references in these scenarios.

Update Notifications

In situations where a subscription was updated externally, a new version of a published file has become available, or an existing subscription exists to a publish file that has been deprecated, a messaging system could be used to notify the users already in the DCC that they need to update the references in their file. 


It might also be nice to provide users with the ability to lock subscriptions. Locking a subscription would imply that the user knows they may be out of date but they do not want to be prompted again to update the subscription. 

Usage Suggestions

When tracking subscriptions, eventually there will be enough data in the system to start making predictions about dataflow. You may see that lighting almost always subscribes to all of the Alembic publishes from animation. 


You could build a rule into your code somewhere that would automatically suggest newly published Alembic caches from animation when the lighter opened their work file. 

Subscription Overrides

Subscriptions can also act as overrides in alternative workflows when using publish file groupings as discussed in week 4. A lighter in a shot might subscribe to a lookdev publish group that includes the base model, textures, and shaders. 


A subscription to the Alembic cache from animation could automatically override the base model packaged with the lookdev when everything is loaded in the DCC or at render time. 

Push on Publish 

As mentioned before, subscriptions can play a big role in helping you manage data across multiple locations. When a user in location B subscribes to a publish created in location A, we know that we need to initiate a transfer from A to B. Further, you might decide that your pipeline should automatically push subsequent versions of the publish file to location B even before the artist in location B has updated their subscription.


In this example, versions of 18 and 19 of the animation cache have preemptively been transferred to location B to prevent the lighter from having to wait when they update their subscription. Subscriptions allow you to make more educated decisions about what files need to be where which can drastically improve iteration efficiency and turnaround time in multi location setups.

Remote Rendering

In scenarios where you’re rendering in the cloud or sending work to remote locations, the combination of the work area version and the subscriptions should give you a complete manifest of the files that need to be transferred to the new location. If you combine this idea with the per-location publish file status that Jeff mentioned last week, you can see how it makes it much easier to be smart about what files need to be moved where in order to execute a render or get a remote user up and running.

Reverting State

Since the subscriptions are tied to a version of the work area, it should be possible to restore a work area and its subscriptions, to a previous state. If the director says they preferred the camera move from last Tuesday, it should be possible to identify the version of lighting that was shown on that day and query the system to determine what version of the camera was being subscribed to at that time. The lighter can then update their subscription to the older version and re-render if necessary. 

Subscription Workflow with Toolkit

When it comes to building subscription-based workflows with Toolkit, I believe it is very much possible. I think I’ve mentioned it before, but one of the strengths of Toolkit, in my opinion, is that it’s a platform for building consistent workflows across every stage of production. This is a critical component when it comes to Subscriptions.

Like I said above, Toolkit’s current mode of operating at the file level for versioning would be the biggest hurdle for implementing the types of workflows described. But, with the openness and flexibility of the platform, outside of time and resources, I don’t see any real roadblocks to adding subscription-tracking interfaces and workflows using Toolkit.

Subscription-based Workflow Suggestions

Finally, I wanted to make a few suggestions about building subscription-based workflows that will help keep the history of the dataflow intact. Some of these might cause consternation among artists on production depending on how rigid you adhere to them. But the more you can adhere to them, the more accurate your representation of the production dataflow will be. 

Once a work area has been published and versioned up, the subscriptions of the previous version should not change. 

If subscriptions change after the fact, the inputs that created the published files are no longer an accurate representation of what the artist was using at that time. 


In this example, changing the subscription of version 23 of the Lighting work area to a different version of the Alembic cache from animation would create an inaccurate history of the geometry used to generate version 23 of the lighter’s renders. Since the renders are published and being consumed by the compositor, the subscription should not change.

A work area version should only be subscribed to one version of a published file at a time.

There are some cases on production where this is a really tough sell, but it does create a clearer view into how data flows through the pipeline.


If you allow scenarios like this, where a work area can subscribe to multiple versions of a publish, you create ambiguity. In this example, without diving into the work files, how do you know which version of the animation was used to render the output frames?

Every external input to your work area should be represented by a subscription. 

If you’re referencing something externally that is not a subscription, then you don’t have a complete view of the dataflow.


In this example, the lighter is using an image file from his home directory and referencing it in his renders. This creates a situation where data is being used and not tracked. If this shot needs to be transferred and rendered remotely, there’s a good chance the render will break. 

Publish file versions should match the version of the work area from which they were generated. 

This makes it easy to tell at a glance what version of the work area created which published files. I mentioned this earlier in the post, but it is worth reiterating. Quickly being able to identify at a glance exactly where something was generated is extremely useful, and quick little wins like this add up on production. 

Whether or not you enforce these rules in a subscription-based workflow, and to what extent, is totally dependent on the goals of your pipeline. There are situations where these rules just aren’t practical given time constraints, disk resources, etc. In my opinion, though, the more you stick to these rules, the more reliable view you’ll have into your production data.

Conclusion

That wraps up week 8! I hope you’ve enjoyed my very quick overview of subscriptions and how they can be beneficial on production. If you have any questions about the details of any of the subtopics or want to give us your thoughts on subscriptions or other ways of tracking these kinds relationships, we would absolutely love to hear from you. Maybe you have ideas about how to tap into subscription data to get at some of the information you’ve always wanted. Please kick off a conversation in the comments section!

Next week Jeff and Jesse are going to dive deeper into multiple-location scenarios, specifically with respect to PublishedFile statuses, location awareness, and cleanup strategies. 

Have a great week!

About Jeff & Josh 

Jeff was a Pipeline TD, Lead Pipeline TD, and Pipeline Supervisor at Rhythm and Hues Studios over a span of 9 years. After that, he spent 2+ years at Blur Studio as Pipeline Supervisor. Between R&H and Blur, he has experienced studios large and small, and a wide variety of both proprietary and third-party software. He also really enjoys writing about himself in the third person. 

Josh followed the same career path as Jeff in the Pipeline department at R&H, going back to 2003. In 2010 he migrated to the Software group at R&H and helped develop the studio’s proprietary toolset. In 2014 he took a job as Senior Pipeline Engineer in the Digital Production Arts MFA program at Clemson University where he worked with students to develop an open source production pipeline framework. 

Jeff & Josh joined the Toolkit team in August of 2015.

Labels: , ,

4 Comments


At November 13, 2015 at 5:06 PM , Blogger Patrick Wolf said...

Thanks for the article. I'm very intrigued by the concept of subscriptions!

Couple of questions please:
1) How does using the new WorkAreaVersion entity differ from just publishing the WorkArea as a PublishedFile and making use of the "Upstream/Downstream Published Files" field to track the subscriptions? Seems like this would give us a more consistent to traverse the graph by just using the PublishedFileEntity.
2) Did you ever run into the issue of files within the workarea referencing each other and then that link being broken when the whole folder is versioned up ie from v001\filea_v001.ext to v002\filea_v002.ext?

Thanks!
Patrick

 

At November 21, 2015 at 8:56 PM , Blogger Josh Tomlinson said...

Hey Patrick!

Apologies for not seeing this comment sooner!

If I were to try to implement this with a SG backend, I probably wouldn't try to overload the PublishedFile entity just because of the different purposes work areas and publish files serve. They would each be versioned, but otherwise the APIs would be very specific to each. I'd also wouldn't want to worry about distinguishing the two when querying SG. Does that answer your first question or am I misunderstanding?

Regarding files referencing each other, yes this was pretty common. Typically, we'd use relative path referencing (avoiding explicit version numbers) to keep things as portable as possible. But that does bring up something I didn't mention in the post which is what a work area might look like on disk. We had a couple of approaches that evolved over time and differed depending on the department/workflow. Here's a really simplistic overview:

1) simple directory versioning, artist works in explicit version directory

area/0001/
area/0002/ # artist works here

2) non-versioned, artist stays in 'work' directory

area/0001/
area/0002/
area/work/ # artist works here

In either of these cases, internal file references might look like this:

{work_area_path}/maya/file_A.ma:
reference_1: ../maya/file_B.ma
reference_2: ../images/file_C.jpg

I hope all that answers your questions. Please let me know if not!

-Josh T.

 

At November 25, 2015 at 9:51 PM , Blogger Shalin Siriwaradhana said...

What is the software you used to create these data flow diagram? Is it creately ?

 

At December 3, 2015 at 1:54 PM , Blogger Josh Tomlinson said...

Hi Shalin!

It's OmniGraffle! https://www.omnigroup.com/omnigraffle#create

This was my first time using it. Really intuitive!

Thanks for reading!
-Josh T.

 

Post a Comment

<< Home

<< Older Posts     Newer Posts >>

Our Story

We are industry folk who love production. A handful of us met while building...
Read More

Subscribe to Our Blog

Follow Us!