scalable media workflows in the cloud
DESCRIPTION
This session will cover the approaches for a cloud-based workflow: media ingest, storage, processing and delivery scenarios on the AWS cloud. We will cover solutions for high speed file transfer, cloud-based transcoding, tiered storage, content processing, application deployment and global low-latency delivery, as well as the orchestration and management of the entire media workflow.TRANSCRIPT
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Scalable Media Workflows in the Cloud
Bhavik Vyas - AWS Stephane Houet - EVS
Media &Entertainment Product Manager
Partner Ecosystem Manager C-Cast
Media Workflows are Complex
Featurettes
Interviews
2D Movie
3D Movie
Archive Materials
Stills
Networks
Theatrical
DVD/BD
Online
Mobile Apps
Archive
MSOs
Process
Store
Deliver
Tools Need to Scale with Media …
Elasticity
Time tomarket
UndifferentiatedHeavy lifting
Pay as you go
Managed IT Services
Secure
High AvailabilityGlobal Scale
A Typical Media WorkflowIn
gest
Inde
x
Pro
cess
Pac
kage
Pro
tect
QC
Aut
h.
Tra
ck
Del
iver
Orchestrate
Store
Where AWS Fits
Amazon Web Services
Inge
st
Inde
x
Pro
cess
Pac
kage
Pro
tect
QC
Aut
h.
Tra
ck
Pla
ybac
k
Media Asset Management
Analytics and Monetization
Mapping the Workflow to AWS Services
AWS Direct Connect
Elastic LoadBalancing
AWS Import/ Export
Amazon S3
AWS Storage Gateway
Amazon Glacier
Amazon EBS
CloudFront
Amazon CloudSearch
Amazon SNS
Amazon SQS
Amazon SWF
Amazon Elastic
Transcoder
Amazon EC2 Amazon EMR
DynamoDB
Amazon VPC
Amazon RDS
Amazon Redshift
ElastiCache
AMI
Route 53
Ingest Store DeliverProcess/ Manage
Amazon EC2
Amazon EC2
Amazon Workspaces
Amazon Kinesis
Elastic LoadBalancing
Elastic Beanstalk CloudWatch CloudFormation
AWS SDKs
Java Python (boto) PHP .NET Ruby Node.js
iOS Android AWS Toolkit for Visual
Studio
AWS Toolkit for Eclipse
AWS Tools for Windows
PowerShell
AWS CLI
JavaScript
INGEST STORE MANAGE SECUREPROCESS
CREATEMONETIZE
INTEGRATEDELIVER
AWS Media Partners: Powering the Workflow
Media S/W in the AWS MarketplaceAWS Marketplace: Online Software Store
• Find, research, try & buy software• 1-Click launch – application running minutes• S/W Billing fully integrated into AWS account• 1700 products across 25 categories
Developer Tool Categories Include• Media Servers• Transcoding• Media Publishing• Digital Asset Management• Image Resizing
Let’s Dive Deeper.
Media Ingest for AWS
AWS Import/ Export
Amazon S3(multi-part Upload)
AWS Storage Gateway Service
AWS Direct Connect
Media Storage on AWS
AWS Cloud
Amazon Glacier
Gateway Appliance/ AWS Storage Gateway
Amazon S3
Block File
On-premises Data Center
Archive Backup Disaster Recovery
Amazon EBS
Choosing the Right Storage
>Durability >>Scale >>>Cost
Object Storage - http(s)
Amazon S3
Amazon Glacier
EBS
Elastic Block Storage
Object Storage - http(s)Block (CIFS/NFS)
EBS Options:• Three types:
– General Purpose (SSD) volumes• Ideal for a broad range of use cases such as boot volumes, small and
medium size databases, and development and test
– Provisioned IOPS (SSD) volumes• You specify the I/O performance, up to 4000 IOPS per volume
– Magnetic volumes• Ideal for workloads where data is accessed infrequently, and
applications where the lowest storage cost is important.
• Encrypted Amazon EBS volumes to meet a wide range of data-at-rest encryption requirements
EBS
Elastic Block Storage
Block (CIFS/NFS)
Reducing Object Storage Costs
Amazon S3
Amazon Glacier
Bucket lifecycle
Policies
Single API access (regardless of the
storage class)
> Durability
>> Scale
>>> Cost Amazon S3 (RRS)
Bucket lifecycle
Policies
<$0.03/GB
<$0.024/GB
<$0.01/GB
S3 & Working with Large Files
S3 Bucket
S3 SDK
AWS S3
S3 multi-part Upload
Files
corporate data center S3 Object <= 5TB
S3 Encryption Options
1. Sever-Side Encryption (SSE)– Provides an integrated solution where Amazon handles key
management
2. SSE-Customer (SSE-C)– Perform the encryption and decryption of your objects while retaining
control of the keys used to encrypt objects.
3. Client library (e.g. Amazon S3 Encryption Client)– Objects encrypted before they are sent to Amazon S3 for storage.
Media Processing
Amazon S3
Instance Store
EC2 Instance
AMI
EBS
Ensure Flexibility: EBS Backed Instances
AMIEBS
Instance Store
EC2 Instance
SIZE INSTANCE BY APPLICATION NEED
Size AWS Instance Type to Application
EC2 Instance Type
CPU’smemory
netw
ork
I/O
loca
l sto
rag
e
Scale Capacity Up & Down as Needed
Elastic
CLOUD capacity
Traditional
IT capacity
Your needs
Time
Ca
pa
city
Use What You Need, When You Need To
Capa
city
Time
Launch of new device
1000’s of transcoders
Amazon EC2 Instance Types
2006 2007 2008 2009 2010 2011 2012 2013m1.small
m1.xlarge
m1.large
m1.small
m2.2xlarge
m2.4xlarge
c1.medium
c1.xlarge
m1.xlarge
m1.large
m1.small
cc2.8xlarge
cc1.4xlarge
cg1.4xlarge
t1.micro
m2.xlarge
m2.2xlarge
m2.4xlarge
c1.medium
c1.xlarge
m1.xlarge
m1.large
m1.small
hs1.8xlarge
m3.xlarge
m3.2xlarge
hi1.4xlarge
m1.medium
cc2.8xlarge
cc1.4xlarge
cg1.4xlarge
t1.micro
m2.xlarge
m2.2xlarge
m2.4xlarge
c1.medium
c1.xlarge
m1.xlarge
m1.large
m1.small
cc1.4xlarge
cg1.4xlarge
t1.micro
m2.xlarge
m2.2xlarge
m2.4xlarge
c1.medium
c1.xlarge
m1.xlarge
m1.large
m1.small
c3.large
c3.xlarge
c3.2xlarge
c3.4xlarge
c3.8xlarge
i2.large
i2.xlarge
i2.2xlarge
i2.4xlarge
i2.8xlarge
g2.2xlarge
cr1.8xlarge
hs1.8xlarge
m3.xlarge
m3.2xlarge
hi1.4xlarge
m1.medium
cc2.8xlarge
cc1.4xlarge
cg1.4xlarge
t1.micro
m2.xlarge
m2.2xlarge
m2.4xlarge
c1.medium
c1.xlarge
m1.xlarge
m1.large
m1.small
c1.medium
c1.xlarge
m1.xlarge
m1.large
m1.small
new
existing Entry into GPU space
G2 Instances
2014 EC2 Instance Highlights• HI1 – 120,000 random read IOPS on SSDs• I2 – Next generation I/O optimized (350,000+ random read IOPS)• M3 – Next generation general purpose• HS1 – 48 TB of storage per instance• CR1 – 244 GiB of RAM &Intel Xeon E5 2670 (Sandy Bridge) CPUs • C3 – 2.8 GHz Intel Xeon E5-2680v2 (Ivy Bridge) CPUs (SSD)• R3 – Next generation memory optimized
EC2 GPU Instances (G2)
1 NVIDIA GK104 GPU (Kepler / GRID)
2.6 GHz Sandy Bridge CPU w/ Turbo enabled
8 vCPUs, 15 GiB of RAM
60GB SSD storage
Frame Capture and Encoding APIs
g2.2xlarge
Unix/Linux instances start at $0.02/hour
Pay as you go for compute power
Low cost and flexibility
Pay only for what you use, no up-front commitments or long-term contracts
Use Cases:
Applications with short term, spiky, or unpredictable workloads;
Application development or testing
On-demand instances
1- or 3-year terms
Pay low up-front fee, receive significant hourly discount
Low Cost / Predictability
Helps ensure compute capacity is available when needed
Use Cases:
Applications with steady state or predictable usage
Applications that require reserved capacity, including disaster recovery
Reserved instances
Bid on unused EC2 capacity
Spot Price based on supply/demand, determined automatically
Cost / Large Scale, dynamic workload handling
Use Cases:
Applications with flexible start and end times
Applications only feasible at very low compute prices
Spot instances
EC2 Instance Options
Managed Database ServicesAmazon
RDSAmazon
DynamoDB
Managed relational database service
Managed NoSQL database service
Amazon ElastiCache
In-Memory Caching Service
Workflow Building Blocks
Amazon Simple Queue Service
(SQS)
Amazon SWF
Amazon Simple Email Service
(SES)
Amazon Simple Notification
Service (SNS)
Amazon CloudSearch
CloudWatch
AWS CloudTrail
Amazon ElastiCache
Amazon Elastic
MapReduce
Elastic Load
Balancing
Amazon Relational Database
Service (RDS)Process
Store
Deliver
AWS Media Delivery
Amazon S3
Processed Media Files HTTP/HTTPS
Streaming or Distribution
Amazon CloudFront
ABR VOD & Live Streaming
Content: Consumption & Reporting
Amazon CloudFront
Access Logs
Amazon S3
EMR
Amazon Redshift
BusinessSmarts
Mapping the Workflow to AWS Services
AWS Direct Connect
Elastic LoadBalancing
AWS Import/ Export
Amazon S3
AWS Storage Gateway
Amazon Glacier
Amazon EBS
CloudFront
Amazon CloudSearch
Amazon SNS
Amazon SQS
Amazon SWF
Amazon Elastic Transcoder
Amazon EC2 Amazon EMR
DynamoDB
Amazon VPC
Amazon RDS
Amazon Redshift
ElastiCache
AMI
Route 53
Ingest Store DeliverProcess/ Manage
Amazon EC2
Amazon EC2
Amazon Workspaces
Amazon Kinesis
Elastic LoadBalancing
Elastic Beanstalk CloudWatch
AWS CloudFormation
INGEST STORE MANAGE SECUREPROCESS
CREATEMONETIZE
INTEGRATEDELIVER
AWS Media Partners: Powering the Workflow
Media Workflows in the Cloud:Customer Examples
in the Cloud
AKA “Video Factory”
Video Factory – Workflow
SDI Broadcast Video Feed
x 24
Playout Data Feed
Broadcast Encoder
Live Ingest Logic
Amazon Elastic Transcoder
ElementalCloud
DRM
QC
Editorial Clipping
MAM
Amazon S3Mezzanine
Time AddressableMedia Store
Amazon S3Distribution Renditions
RTPChunker
Transcode Abstraction
Layer
Mezzanine
Playout Video
Transcoded Video
Metadata
SMPTE Timecode
Mezzanine Video Capture
• The UK’s biggest video & audio on-demand service– And it’s free!
• Over 7 million requests every day– ~2% of overall consumption of BBC output
• Over 500 unique hours of content every week– Available immediately after broadcast, for at least 7 days
• Available on over 1000 devices including– PC, iOS, Android, Windows Phone, Smart TVs, Cable Boxes…
• Both streaming and download (iOS, Android, PC)
• > 20 million app downloads
Sources: BBC iPlayer Performance Pack August 2013http://www.bbc.co.uk/blogs/internet/posts/Video-Factory
www.evs.com
POWERING MULTIMEDIA CONTENT DELIVERYAT 2014 FIFA WORLD CUP BRAZIL™
EVSSPORTS
June 12 - July 13 2014- 12 Stadiums
- 64 Games
IBC1 in Rio de Janeiro
EVS appointed for multilateral production- On site2 live production- IBC file-based video management- Multimedia production and distribution
FIFA WORLD CUP 2014
Notes:1. The International Broadcast Centre (IBC) is a temporary hub for broadcasters during major sport
events.2. Remote broadcast television studio where the event is produced and then beamed to
broadcasters via satellite.
EVS ENABLED A NEVER-SEEN-BEFORE MULTIMEDIA EXPERIENCE
6 Live Streams
HLS streaming of 6 HD streams to tablets & mobiles per match
24 Replay cameras
On demand replay of selected events from any of the 20+ cameras on the field.
+4000 VoD elements
On demand multimedia exclusive edits.
EVSSPORTS
Max of 12 streams at 10Mbps each.
Max of 36 streams at 3,5-10Mbps
9 HLS encoding, from 0.4 to 3.5Mbps + 2 thumbs 1 RTMP
EVS C-CAST LIVE STREAMING WORKFLOW
EVS C-CAST VODWORKFLOW
9 HLS encoding, from 0.4 to 3.5Mbps + 2 thumbs1 RTMP
EVSSPORTS
Cloud PlatformLinking mobile trucks, studios and viewersContent selection and approval processMetadata editing stationAutomatic translations in 20+ languagesAutomatic HD backup on topWorkflow engine, including- VoD file transfers and transcoding management- Live streaming activation and transcoding management
Output: API mode or white label embedded
EVS C-CAST MANAGES ALL FROM PRODUCTION TO VIEWERS
MULTIMEDIA PRODUCTIONHBS OVERVIEW MAP
MULTIMEDIA PRODUCTION IN THE AWS CLOUD
Aspera for transfers into S3Store and deliver data using S3Create image thumbnails (EC2, SQS & S3)Transcode VoD files using ZencoderTranscode Live using Elemental Cloud- Up to 128 concurrent live streams encoded
Operator GUI for Rio and international sites- S3, Cloudfront, EC2 and RDS
Manage all workflows at IBC and Venue- EC2, SQS & RDS
1Gbps Direct Connect from Rio to Dublin
EVSSPORTS
6 MULTILATERAL STREAMS
17 INDIVIDUAL MRL STREAM
4 INDIVIDUAL COMMENTARY
243 DIFFERENT LIVE STREAMS /GAME
2,799,360 MINUTES OF ENCODED STREAMS
LIVE STREAMING
MULTI-ANGLE CLIPS
On average 60 CLIPS RECORDED PER MATCH, overall
UP TO 20,000 MULTILATERAL CLIPS throughout the tournament
Roughly a third of clips from multicam with 14 DIFFERENT ANGLES on average
COMBINED DATA FOR FIFA WORLD CUP WEB PLAYER AND APP AFTER 64 MATCHES FOR ALL MARKETS
TOTAL HOURS WATCHED:
>15 MILLION
UNIQUE USERS: >35 MILLION
Available in more than 10 TERRITORIES
Peak ACTIVE CONCURRENT USERS
500,000
FIN & THANK YOU!
Hall #3.B16
Hall #7.G30
Hall #4.B75
Hall #8.B90
Hall #5.B20