Two Guys and a Toolkit - Week 5: Publishing from Maya to Nuke

New to Shotgun or Toolkit? Check out Our Story, learn about Our Product, or get a Quick Toolkit Overview. If you have specific questions, please visit our Support Page!

Publishing from Maya to Nuke
Hi everyone! Josh here. Welcome back for part five of our series dedicated to building a simple pipeline using Toolkit.

Last week Jeff talked about PublishedFile grouping as a way of associating individual publishes as an organizational or representation tool within the pipeline. If you haven’t read it yet, I highly recommend it.

Are any of you doing anything similar to package rigs and models together? Or are you presenting a collection of render publishes to compositors as a single item in the loader? How are you leveraging Shotgun entities in unique ways to store your pipeline’s metadata and to present information to your users in a clean, easy-to-understand fashion? Let us know in the comments!

This week we’re going to stay in the realm of publishing and try to answer one of the most common questions I’ve heard recently:
“How do I publish my Maya renders to Nuke?”
We had 3 support tickets in the past two weeks asking for information on how to set up this hand off. We also heard from Toolkit veteran and recent Pipeline Award winner, Benoît Leveau in the comment section of the week 3 blog post.
“What was really missing from the publishing hooks when I installed Toolkit was a way to publish renders from Maya. This sounds like the most basic thing a pipeline should do, yet it's not mentioned anywhere. I had to look at how this was done in Nuke and replicate a similar behavior in Maya, which made me understand a lot about the template and context systems in Toolkit (which is a good thing) but having examples of it would be great when starting.”
-Benoît Leveau
The lighting to compositing hand off was something we needed to build for our simple pipeline as well. So I think it makes a lot of sense to dedicate this week’s post to taking a look at what is necessary to get published frames from Maya into Nuke using Toolkit, and putting together an example implementation that new Toolkit users can reference.


How do we get rendered images from Maya to Nuke?

For this post, you can find all of the necessary pieces in our tk-config-simple and tk-framework-simple repos. I’ve put plenty of comments in the code to help explain each little bit, but if you have any questions, don’t hesitate to ask. If you’ve built this type of workflow for your studio and you have ideas about how to improve this example, I would absolutely love to hear your thoughts!

Identifying what is needed

When the tickets came in asking how to setup publishing frames to Nuke, I have to admit that I was surprised there wasn’t much information in the Toolkit docs. I found this demo video showing how to use Toolkit in Nuke and this video talking about the Loader app. The first video showed how to load external Asset publishes and update render passes, but there was only a passing mention of loading shot renders. In the second video I could see in the filters a publish type called Rendered Image. I thought that maybe the loader was already setup to receive rendered frame publishes by default in Nuke.

So I took a look at the default configuration of the Nuke engine, and sure enough I came across this line in the shot_step.yml:

Rendered Image in Nuke engine configuration

It appeared as though the multi loader app running inside of Nuke would accept PublishedFiles of type Rendered Image, and would probably create Nuke Read nodes from them. To double check, I did a quick search through the multi loader code to see what the behavior was. I found the description of the read_node action in the Nuke actions hook:

read_node action as defined in the loader

Bingo! Confirmed. That was exactly what we needed. The next step was to work backwards, and get some Rendered Image PublishedFiles created to feed into Nuke.

Rendering and Templates

Before I went any further, I needed to answer a couple of workflow questions:

  • Where would the frames end up on disk? 
  • How would the artists be publishing?

One of the support tickets I was responding to when writing up this example wanted to include the render layer name and camera name in the path to the frames on disk. Their studio was initiating renders from within maya and wanted to publish from there as well. So those requirements helped drive how I answered the questions, and how I decided to build the example that follows.

The answers to these questions will vary greatly across studios, so it’s important to keep in mind that what follows is just an example of one way to setup Rendered Image publishes to Nuke.

Publishing from within a DCC like Maya, where your render setup lives, has a couple of benefits. First, if the DCC has Toolkit support, you can leverage the Publisher app and customize the hooks to meet your needs. You’ll also have access to the cameras, render layers, and passes as the hooks execute; this makes identifying the rendered frames to publish fairly straight-forward.

One downside to publishing within the DCC is that the artist may not have the session open when the render completes. Perhaps their file takes a long time to load, or it could be that it’s just more convenient for them to publish from within some other working context. Maybe launching the DCC is just not an optimal solution for publishing in your studio. In these scenarios, you’ll probably find yourself writing your own method for discovery of the rendered frames to publish, but the underlying code will probably look similar to the multi publish hooks we’re going to override for this example.

The next step I took was to define a template path for the frames to be rendered. As I mentioned before, I wanted the camera and layer names in the path, so I added the following to the keys section in core/templates.yml within the tk-config-simple repo :

Maya camera and layer name template keys

Next I added the actual render template:

Simple template for rendered frames

I only defined a single template, but I can imagine a production facility having several different conventions for rendered frames. In that case, you’d need to create templates for each convention and replicate some of the logic you’ll see in the hooks below.

Publish hook modifications

To generate the Rendered Image publishes, I needed to create a new secondary publish type like we discussed in previous weeks with cameras, alembic caches, and shader networks. This required adding the definition of the new publish type to the secondary_outputs list in the maya engine block of the shot_step.yml environment config:

Secondary output definition in environment config

The secondary output definition for the Rendered Image type help drives how the rendered frames will be displayed in the publisher. The publish_template field is slightly misleading here since the frames will have already been rendered when the publisher is visible. This is unlike other secondary Maya publishes which use the template to drive where to write those additional files as the publish hooks execute. For this example, since I was only using one template for rendered images, I put it in there for easy access inside the hook.

The first hook that needed to be modified was the scan_scene hook. You can see the changes I made to it here. The purpose of this particular hook is to identify the things in the session that need to be published. So I needed to find out if there were any rendered frames on disk matching the maya_shot_render template.

The important thing I think I really started to understand while working with this code is how the templating system works. The logic I wrote was basically to iterate over the cameras and render layers in the maya session and evaluate the render template with those names. If there were frames on disk that matched the evaluated path, then I had something to publish! I found a method in the Toolkit API called abstract_paths_from_template that did all the hard work for me. Given a template and a dictionary of fields, it will returns matching paths. Once I had the paths I was looking for, I added them to the list of items to be published.

Something that was less than ideal was having to hard code the template names in the scan_scene hook. Ideally, the hook would have access to the publish_template setting defined in the app’s config, but I don’t think it does. That’s not sufficient anyway in the case where you have multiple render templates to evaluate. Perhaps you could pass in a list of templates via the setting, but I didn’t think to try it. Another idea would be to establish a convention for the names of your render path templates and key off of those to find the appropriate templates to use for publishing. Anyway, this is something to consider improving upon in your implementation.

One other personal point to make here is that there is a {version} key in the maya_shot_render template. I retrieved the version from the maya_shot_work file to populate in the render template. I don’t know if this is the best way to determine the version number in Toolkit, but I do like the fact that I can tie my rendered frames to a scene file. If my main render pass is version 003, then I know it was rendered from version 003 of my work file. Tying versions together like this is something I’ve always been a big proponent of.

Along those lines, how does your studio manage versions across workfiles, published files, etc? Do you have any custom logic that keeps versions in sync or ties versions together in some way? It would be awesome to hear if anyone has abstracted the concept of a version in a unique and interesting way.

Now that I had secondary items to publish, I needed to actually publish them. I didn’t add any custom validation for the Rendered Image publishes, but I would highly recommend it for real production pipelines. I stubbed it out in the secondary_pre_publish hook here and added some comments. Some suggestions for validation include:

  • All frames exist
  • File sizes are consistent
  • Permissions are correct
  • Secondary files exist (thumbnails or quicktimes for example)

And finally, I modified the publish code itself, which was relatively minimal for this example. In the secondary_publish hook all I really had to do was get the publish path from the other_params dictionary I supplied from the scan hook and populate the data needed to register the publish with Shotgun. As I mentioned before, the other Maya secondary publish types we’ve covered would be exported from the session to a new file on disk in this portion of the hook. I relied on the fact that the rendered images are already on disk to simply forward the path onto the registration code. In a true production studio, you might do additional processing here, such as setting file permissions, creating symlinks, etc.

Maya to Nuke

Now let’s look at how it all fits together. Here are some example frames on disk.

Rendered frames on disk

You can see the 2 layers on disk showing up as check boxes in the Publish app.

Secondary publish of rendered images

After I publish, I can see the PublishedFile show up in Shotgun.

Publish file in Shotgun

One thing that would make this much nicer would be to upload one of the frames as the thumbnail to use. This wouldn’t be too hard to whip up, I just didn’t get to it. My example just uses the thumbnail path that comes with the publish.

The great news is that I made absolutely no changes on the Nuke side. Once I had the published Rendered Images, they showed up immediately in the Nuke loader:

Rendered Images in the Nuke loader
I was able to use the Action button to create read nodes for each of the layers, and it worked great!

Conclusion

That’s as far as I took my example. I hope this post provided enough information to help some of you who might be considering building a Maya to Nuke handoff for your pipeline. There is certainly a lot of room for customization with Toolkit and within the publish hooks specifically; which is great in this particular case because every studio will have their own unique setup and their own unique requirements for handling rendered frames. As I said earlier, I’d love to hear about your experiences getting rendered images published from Maya and into Nuke (or any other software for that matter), so please let me know what you’ve been up to in the comments section.

This marks the halfway point in the series, and I wanted to say thank you all for sticking with us. As Jeff mentioned in last week’s post, we’re going to try to start steering away from the “here’s what we did” posts and move toward the “wouldn’t it be cool if” discussions. If there’s anything you all are interested in exploring, we’d love to hear about it.

Jeff teased this in a previous post, but I think, spec’ing out how to build a Product-based, Subscription pipeline within a Shotgun/Toolkit context would be a really fun exploration for a future post. I think Subscriptions are critical to getting immediate answers to some of the toughest, most common questions people ask in the middle of production. I also believe Subscriptions provide a key ingredient to building efficient multi-site pipelines. If this is something you’d be interested in exploring as well, please let us know.

About Jeff & Josh 

Jeff was a Pipeline TD, Lead Pipeline TD, and Pipeline Supervisor at Rhythm and Hues Studios over a span of 9 years. After that, he spent 2+ years at Blur Studio as Pipeline Supervisor. Between R&H and Blur, he has experienced studios large and small, and a wide variety of both proprietary and third-party software. He also really enjoys writing about himself in the third person.

Josh followed the same career path as Jeff in the Pipeline department at R&H, going back to 2003. In 2010 he migrated to the Software group at R&H and helped develop the studio’s proprietary toolset. In 2014 he took a job as Senior Pipeline Engineer in the Digital Production Arts MFA program at Clemson University where he worked with students to develop an open source production pipeline framework.

Jeff & Josh joined the Toolkit team in August of 2015.

Labels: , ,

10 Comments


At October 20, 2015 at 6:45 AM , Blogger raphe said...

Another great post, thanks! I'll have to write my first Maya publish hook in the next week or two & this will jumpstart the learning process.

A few notes:

* In response to "In these scenarios, you’ll probably find yourself writing your own method for discovery of the rendered frames to publish": in these cases I'll note that you can usually either run the dcc app in command line mode locally or on a remote machine & the env context information will be available. It might be a little heavy handed, but at least you don't have to manage two different sets of "discovery" code.

* Yes, you definitely want the maya shot version number in the rendered frame name imho. The one bit there that always throws me for a loop is: if a studio outputs to stereo you have to either 1) always include the camera name in the rendering image name or 2) use two different rendered image templates. If you go with #2, then you add complexity to any tool that ingests those rendered images. Neither is a great solution.

* I've only set up the Maya to Nuke transfer once. It was a simple/brute force solution but the comp artists liked it. Basically we just had a php webpage that quickly scanned the latest Nuke project file for every shot in a seq & then reported anything worth reporting from the seq level: if the comp artist wasn't using an available cache, didn't have the latest image seq, etc (there were a bunch of little features that made it smart enough to not over-report). It was good because producers could check the page from their laptops during dailies if questions came up & lighters could check to see if the comp artists were using their latest, & the comp lead could quickly check on his minions. Then inside Nuke we had a few brute force scripts: update everything to latest, update selected to latest, import missing elements. There was also a automatic bash comp handoff from lighting that slapped everything together based on render layer order -- so they'd have a starting point for complex comps. Of course this whole setup assumes you're pretty much alwasy after the latest, so comp peeps were left to manage manually if things got hairy. Besides that downside it was simple and gave people enough control to feel comfortable. Looking foward to the more in-depth post on subscription workflows!

Cheers,
Raphael

 

At October 20, 2015 at 10:09 AM , Blogger Jeremy Brown said...

Sweet! I'm following along and starting to implement this (at long last).

One thing I wanted to chime in about was where you mentioned having to hard code the template name in the hook. I ran in to the exact same issue when I was working on another hook.

During an asset publish, I wanted it to not only copy the versioned .ma file, but to also copy an UN-versioned "master" file to a different location. I decided to make a new template called 'maya_asset_master' and I did what you did and hard-coded it.

In another completely different situation, I actually ended up redoing the tk-multi-reviewsubmission app to have greater control over the burnins, slate and QT format. I also have it render out .jpg sequences of publishes as well. Anyway, the point is that since I was free to hack away at the app as I wanted, I ended up adding configuration options like 'jpg_path_template' where (just like all the built-in sgtk apps) I define a default in the info.py, but I'm free to over-ride in the config if I wish. I also added a new hook which I called 'hook_publish_setup' in which I took the heavy lifting logic out of the Render() class and stuffed it in the hook. This allows me to define different behavior per-project. For instance, some times our clients require lens info on slates, or sometimes different CDL and color space handling.

Anyway, the point I guess I'm trying to make is that it would be SWEET if instead of having to as you say "take ownership" of an app to simply add another piece of configuration, you could add to an apps configuration on the fly without having to have that option defined in the app's manifest. In other words, if it finds a setting without a definition, it just passes it right along as if it were in there. Or something like that... you guys are the experts =)

 

At October 21, 2015 at 8:10 AM , Blogger Unknown said...

Thanks for adressing my comment! It's really interesting to see different approaches to the same problem. I didn't know you could use dots in the token names (as in your "maya.layer" example), that's neat!

We didn't want to render directly to the publish folder, as there would be no easy way for the compers to identify what's published and what's still a work in progress. Also, I'm wondering what happens in your case when importing wip files in Nuke and opening the Scene Breakdown, as the files will match the publish template but are not registered in Shotgun?

Anyway, we wanted a clear separation between the work area (per-user and open permissions) and the published area (where only selected scripts have write permissions to it) to avoid files (or even whole folders) being moved/deleted/overwritten. So we ended up using two templates: the work template is used to dictate where Maya renders its files, and is also used by the scan hook to detect the image sequences. The secondary publish hook then has to transform these paths to a publish template, and move the files accordingly.

Benoit

 

At October 26, 2015 at 7:06 AM , Blogger Josh Tomlinson said...

Hi guys! Thanks again for the feedback! It's great to hear from you!

Raphe -

I'm glad you're liking the posts, and I'm glad they're helping you get started! Regarding the discovery, that totally makes sense. I guess you could write a custom tk app to do this too. I wonder how often people write tk apps targeted at tk-shell and if they do anything special to prevent users from having to find a specific configuration to launch from. Would be interesting to hear.

I love hearing that you had a Nuke workflow that the artists liked. The reporting tools sound very nice and the fact that it was simple makes it easier on everyone involved. I've definitely been on the 'too complex' side of a workflow a time or two, and it stinks having to scramble to get everybody back on board. I'm looking forward to the subscription post as well. It's a difficult thing to get right I think, and it definitely takes an investment, but they can pay off in big ways. Hopefully we'll be able to get that across. :)

Jeremy -

This post is for you! Well, at least partly. ;)

I was curious about adding a non-manifest key/value to an app's config to see what would happen. I was able to add it into the app's configuration and access it from the hook at runtime! I'm not suggesting that's what people should be doing as it makes the configuration not portable and not documented. Perhaps toolkit shouldn't allow it, but should instead provide one or two settings that all apps understand by default. Something like a *args and **kwargs catch-all that is forwarded on to hooks.

Benoit -

Manne taught me about namespacing the keys that way. :)

Regarding the frame locations, I've personally been a fan of writing images to their final destination and letting the compositors have access to them as soon as they're published, even if all of the frames aren't done. So I guess that's why my validation hook was empty. ;)

The ability to push things out early and then update/backfill already published frames can be convenient if people are distributed geographically and when you have long renders going on.

I do like that toolkit allows for either kind of setup though. I should run some tests with Breakdown in Nuke to see what the behavior is for the case you mention, but I suspect it would allow you to update to a non-published set of frames on disk. IMO, updating should be db driven rather than using what's on disk. We'll definitely be talking about this in our subscriptions post.

Thanks guys!
-Josh

 

At November 18, 2015 at 1:03 PM , Blogger David vH said...

Hey guys,

There's one part that I miss aparently.
In Nuke we have the shotgun write node that pulls it's render location + naming from the template.
As far as I can see there's no such functionality in Maya out of the box.
So did you guys just fill in the Maya render layers/folders/location in manually in Maya? Or is there some logic that I'm not aware of?

Cheers and keep up the good work!

David

 

At November 21, 2015 at 9:13 PM , Blogger Josh Tomlinson said...

Hi David!

You are correct. I did just manually set the render path in maya while testing the setup. I don't know of anything out of the box either. Seems like it should be possible to write some code to populate the render settings at some point during the artist workflow, but I didn't go that far. If you end up doing that, I'd love to hear what your solution is!

Thanks!
Josh T.

 

At November 23, 2015 at 2:29 AM , Blogger Unknown said...

Hi David,
We wrote a function that sets the "defaultRenderGlobals.imageFilePrefix" attribute based on the current context and the proper template (depending if we're in an Asset step or a Shot step).
This function is called in the post_publish hook, but also in the scene operation hook for the "open" and "save_as" operations, so we make sure it's always up-to-date.
Benoit

 

At December 9, 2015 at 6:00 PM , Blogger rulosmuchos said...

Mr. Unknown, could you give a lil bit more about setting output_path function?

 

At July 5, 2016 at 4:21 AM , Blogger Abdul Raheem said...

Thanks for sharing this valuble information and it is very useful for me and also who wants,
Thanks for sharing the details!
i would like to more information from your side!
please added more then tips!Am working in
used laptops in hyderabad

 

At July 8, 2016 at 10:33 PM , Blogger Abdul Raheem said...

Good info. Thanks for following up and posting back.
i would like to more information from your side!
please added more then tips!Am working in
used laptops in hyderabad

 

Post a Comment

<< Home

<< Older Posts     Newer Posts >>

Our Story

We are industry folk who love production. A handful of us met while building...
Read More

Subscribe to Our Blog

Follow Us!