aws september webinar series - visual effects rendering in the aws cloud with thinkbox

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Usman Shakeel – Amazon Web Services

Ryan Russell – Thinkbox Software

Cloud Rendering at Scale with

Thinkbox Deadline on AWS

Visual Effects and

Animation1

Who is using AWS for Rendering?

2

3

Theme Parks

Gaming

Marketing Departments4

5

6

Manufacturing

Bio-Informatics &

Drug Design

7 Engineering and Architecture Design

VFX/Animation Rendering - workflow components

CompostingModeling Backend

Rendering

Asset Management

Collaboration and Task Management

Boutique Studios

• No or very little on-premise capacity

• Limited Licenses

• Project based work from customers

(large studios)

• Very tight budgets

• Some projects need very large scale

(more compute)

• No or very little infrastructure

optimization

• Content Security (mandated by their customers)

• Mostly All-in Cloud Workflows

A tale of two customers

Large VFX/Animation Houses/Studios

• A decent on-premise investment

• Unlimited Licenses

• Project based work from different Studio projects

• Tight Deadlines and producers decide the budgets

• Often under immense pressure for higher quality (more compute) and faster renders (more compute)

• Mastered the on-premise infrastructure optimization

• Content Security

• Hybrid Workflows

They both ask us the same thing …

The Cloud Scale

The ability to spin up thousands of cores on-demand

Turning CapEx into OpEx

No need to invest in infrastructure that may not be needed

after projects are finished

The most up-to-date configurations

Latest CPU and memory configurations

Large bandwidth between instances

Fast Storage options

A project based “disposable” infrastructure

Flexible (Cloud Elastic) Licensing

They both tell us the same thing …

=< $0.01per core/hour

Access to thousands of

cores whenever needed

No upfront investments in infrastructure

Easier collaboration

Ecosystem of

software providers

Access to large memory configs to do 6K/10K renders

Project based “disposable” infrastructure

Rendering in the Cloud - State of the Union

Scale at a very cheap price

• EC2 Spot

– Spot Fleet

– Termination notification

– Further Automation with Lambda

• What is large scale?

– 10s/100s/1000s/10000s cores


Data/Content Migration

• Aspera, Signiant, File Catalyst, Expedat

Storage

• Shared File Systems

– Amazon Elastic File System (EFS)

– Intel Lustre

• Storage Cache for Hybrid Workloads

– Avere vFXT

– Penzura

– Netapp Cloud-onTap


Licensing at Cloud Scale

• SaaS

• AWS Marketplace/Usage based licensing


3D Modeling in the Cloud

• GPU based EC2 instances

• Terdici PCoIP, AWS Appstream, Nice, Frame, Otoy

• Windows and Linux Support (VNC+VirtualGL)


Content Security

• AWS alignment with the latest MPAA cloud based application

guidelines for content security – August 2015

• Amazon VPC private endpoint for S3 – enables a true private

workflow capability

• Encryption & key management capabilities using Amazon Key

management Service (KMS)

• Amazon Glacier Vault for high-value media/originals

Introduction to Deadline

• What is Deadline?

– Cross-platform render farm management.

– Manages on-premise, Cloud-based, or hybrid render farms of all sizes.

– Supports over 70 content creation applications out of the box.

and more!

– Used in VFX, Design, Broadcast, AEC (Architecture, Engineering,

Construction), Education, etc.

Introduction to Deadline

• What is Deadline used for?

– Distribute frames of an animation

sequence across multiple nodes.

– Perform other types of processing, including particle simulations, Quicktime

movie generation, file conversion, etc.

– Distribute regions of a still frame

across multiple nodes (Jigsaw).

Use Deadline to automate your pipeline!

• Dependencies

– Allows jobs to start automatically.

– Job, frame, script, and asset

dependencies.

• Custom plugins

– Write Custom plugins to perform

any type of custom processing.

– Control anything with an API

(command line, scripting, etc).

Use Deadline to automate your pipeline!

• Event triggers

– Automate tasks when jobs enter certain states.

• Update internal pipeline mgmt software when a job starts and finishes.

• Do some post processing on the rendered output after a job finishes (Draft).

• Draft

– Compositing Python module that is

included with Deadline.

– Automate compositing operations on rendered images.

• Quicktime movie generation (multiple movies at different resolutions).

• File conversion (convert EXRs to JPGs before downloading for review).

Deadline Components

• Deadline Repository

– Consists of a MongoDB database and a shared file server.

– One Repository per render farm.

• Deadline Client

– Includes the Slave and Monitor applications (in addition to others).

– Slaves run on the render nodes to control the rendering process.

– Monitor is used to view render jobs and render nodes.

– Client applications connect directly to the database and file server.

Rendering with Deadline in AWS Cloud

• Rendering in AWS Cloud with Deadline 7

– AWS can be an extension of your on-premise farm, or it can host your

entire render farm.

– Think of it as a remote office that you connect to over VPN.

Rendering with Deadline in AWS Cloud

• What tools are currently available?

– Amazon Cloud Plugin for Deadline:

• View and control existing AWS instances

directly from the Deadline Monitor.

– Deadline VMX:

• Automates the launching / termination of

AWS instances based on jobs in queue.

– Deadline AWS Cloud Wizard:

• Set up a basic Deadline render farm in

the AWS Cloud in about 30 minutes.

Deadline 8 makes Cloud rendering easier!

• Deadline 8

– Currently accepting beta requests (more info later).

– Introduces Usage Based Licensing for metered licensing.

– New Proxy Server application removes VPN requirement.


• Usage Based Licensing

– Render nodes pull metered licenses from Thinkbox’s Cloud-based

license server.

– Usage is tracked per minute.

– Bulk minutes will be available via Thinkbox’s online store.

– Store will eventually host 3rd party licensing (Nuke, Vray, etc).

– Let’s check out the store!


• Deadline Proxy Server

– Facilitates communication between the Deadline applications and the

Deadline Database and Repository.

– Implements REST over HTTP(s), and can be combined with Nginx or

Apache for SSL security, removing the requirement of a VPN.

Deadline 8 hybrid render farm

Deadline 8 hybrid render farm

Local and Cloud render nodes in

the same render farm

Local and Cloud render nodes

working on the same job

Connected to Proxy Server

What’s coming in the future?

• Deadline VMX Improvements

– Improved budget and cap flexibility.

• Budgets based on amount to spend

per day, per month, etc.

• Budgets based on number of concurrent AWS instances.

– Support for Amazon EC2 Spot Instances.

• Spot Instances are a great alternative for VFX rendering resources.

• Render more frames across more nodes concurrently at a reduced cost.

• If Spot Instances are terminated while rendering, Deadline can recover

those frames and assign to other render nodes.

What’s coming in the future?

• AWS CloudFormation Templates

– Create an entire Deadline render farm infrastructure on AWS Cloud.

– Create AWS Cloud render nodes to connect to an existing on-premise

Deadline render farm.

• Asset Transferring

– Build asset awareness into Deadline jobs.

– Jobs won’t render until all assets are available.

– Missing assets will be automatically transferred.

Further Reference

• Join the Deadline 8 beta!

– Email [email protected] to join the beta.

– Free usage based licensing during the beta.

• Check out M&E solutions on AWS at:

http://aws.amazon.com/digital-media/solutions/

• Questions?

aws september webinar series - visual effects rendering in the aws cloud with thinkbox

Technology