4k media workflows on aws

4K Media Workflows on AWS

Usman ShakeelPrincipal Solutions Architect Lead (M&E)

Amazon Web Services

©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

Where is the problem?

More Bandwidth$$$$$

More PowerfulCompute $$$$$

Way more Storage$$$$$

Some Progress(ABR, HEVC, VP10)

Does it matter?Screen Size

Limited number of 4K mobile devices

Viewing Distance

HDMI 2.0

and there is more …

Where is the sliding scale on my Infrastructure?

A Concept - The Content LakeInspired from Data Lake (Coined by James Dixon in 2010)

A single store of all of digital content that you create and acquire in any form or factor•Don’t assume any resolutions/formats (for now or future)

•It is up to the consumer (application consuming the content) to use the appropriate infrastructure for processing

Amazon Simple Storage Service (S3) : the Content Lake

• Durable, cost-effective and fast

• Highly scalable front-end – Multi-part uploads (parallel writes)

– Range-gets (parallel reads)

• Use Amazon S3 with on-premises storage in a hybrid model

• No Storage capacity Upper-bounds

• Secure

Hydrating the Content Lake

Amazon S3

AWS Import/ ExportAmazon S3

(multi-part Upload)

AWS Storage Gateway

Direct Connect

N x 1G | 10G

Massively Scalable Front-end

Edge Locations

Availability Zone

Region

Dallas (2)

St.Louis

Miami

JacksonvilleLos Angeles (2)

Seattle

Ashburn (3)

Newark

New York (3)

Dublin

London (2)

Amsterdam (2)

Stockholm

Frankfurt (2)Paris (2)

Singapore(2)

Hong Kong (2)

Tokyo (2)

Sao Paulo

South Bend

San JosePalo AltoHayward

OsakaMilan

Sydney

MadridSeoul

Mumbai

Chennai

Regional Lakes …

Content Life-Cycle

Amazon Glacier (Life Cycle Policies)

Amazon S3

Near-term Storage

Long Term Archive

Both tiers accessible through a single API

Oldest content trickles down to glacier automatically to save cost

Consuming the Content Lake

Amazon S3

Amazon S3 (range-gets)

Direct Connect

N x 1G | 10G

Massively Scalable S3 Front-end

EBS

Instance Store

cMassively Scalable Compute on AWS Cloud

On-Prem Apps

Consuming the Content Lake

Compute IntensiveIntel ES-2666 v3 (Haswell) optimized specifically for EC2

Memory IntensiveLowest price point per GiB of RAM

GPUs 1,536 CUDA cores4GB of video memory

Enhanced NetworkingHigher PPS, Lower network jitter, low latency

IO IntensiveSSD Storage, EBS Optimized

High Storage24 x 2000 GiB per instance

Massively Scalable Compute Layer

Amazon S3

c4g2

m3

HEVC Encode

GPU Transcode

HDi2

Package

Queues (orchestrated by Amazon SWF)

Content Lake

Task Specific

• Spot Instances(no charge if spot is terminated)

• Reserved Instances

• Pay for only what you use

• Buy for your need– Short term/Long term

– Real-time/Batch workloads

• AWS Marketplace – Pay for what licenses you use

Cost Effective Compute

VS. $2.80/hr

Launch a CloudFormation stack

with all the infrastructure

resources for a specific project

Autoscale the stack as

appropriate

AMI

CloudFormation

TemplateCloudFormation

Terminate

Template

Automated Infrastructure Layers based on Project Scope

AWS Ecosystem (License included in hourly* pricing)

INGEST STORE MANAGE SECUREPROCESS

CREATEMONETIZE

INTEGRATEDELIVER

AWS Import/ Export

S3 multi-part API

For Source, Renditions, Metadata Sidecar Files

Ingest

AWS ElasticBeanstalk

DAMWeb Service

CloudFront Distribution

Amazon DynamoDB

Amazon CloudSearch

DAM Catalog

Rendition Workers

EC2 ASG

Amazon SNS Topic

Amazon SQS QueueRendition Jobs

Amazon SQS QueueMetadata Processing Jobs

S3 Notification

Elastic TranscoderProxy / Thumbnail

Generation

Processing

Metadata Workers

EC2 ASG

A Simple DAM at AWS

AWS Import/ Export

S3 multi-part API


Ingest


DAMWeb Service


Amazon DynamoDB

Amazon CloudSearch

DAM Catalog

Rendition Workers

EC2 ASG

Amazon SNS Topic



S3 Notification


Generation

Processing

Metadata Workers

EC2 ASG

A Simple DAM at AWS (Cost)

$

$

©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved.

AWS Import/ Export

S3 multi-part API


Ingest


DAMWeb Service


Amazon DynamoDB

Amazon CloudSearch

DAM Catalog

Rendition Workers

EC2 ASG

Amazon SNS Topic



S3 Notification


Generation

Processing

Metadata Workers

EC2 ASG

A Simple DAM at AWS (Using Spot)

0.$

0.$

AWS Import/ Export

S3 multi-part API


Ingest

AWS Elastic

Beanstalk

DAM

Web Service

CloudFront

Distribution

S3 Notification

Elastic Transcoder

Proxy / Thumbnail

Generation

Processing

Lambda

Metadata

Extract & Persist

Amazon

DynamoDB

Amazon

CloudSearch

DAM Catalog

Server-less Applications using AWS Lambda

0.0$

yeah more science…

Remote application visualization for production tools, VFX modeling

Content stays in the Cloud

Powerful workstations in the cloud when needed

Amazon S3

GPU

Content Lake

Amazon Appstream

RemoteDumb Terminal

Amazon EC2

Remember…

The Content Lake on AWS

Process

User Delivery/ConsumptionVFX/Production

On-Prem Apps

Archive

Content LakePartner/Affiliate/Service Provider


EBS

Instance Store

Amazon Glacier (Life Cycle Policies)

Amazon S3

Amazon EBS/EC2 Instance Store

Process


On-Prem Apps

Archive

Partner/Affiliate/Service Provider

Content Access Transfer


EBS

Instance Store

Amazon Glacier (Life Cycle Policies) Amazon EBS/EC2 Instance Store

Process

Partner/Affiliate/Service Provider


On-Prem Apps

Archive

Amazon Glacier (Life Cycle Policies) Amazon EBS/EC2 Instance Store

c

c

Direct Connect

Content Access Transfer

Disposable Infrastructure

AutoscalingWorkload specific

Amazon S3

South Lower Hall: SL9016http://aws.amazon.com/digital-media/

Come check out…Presentation Theater, Meet the AWS experts, cool demos

and Solutions Ecosystem

Thank you ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved.

4k media workflows on aws

Technology