TRANSFORMING CLOUD INFRASTRUCTURE

TO SUPPORT BIG DATA STORAGE AND WORKFLOWS

PRESENTER AND AGENDA

PRESENTER

Jay Migliaccio

Director

Cloud platforms and Services jay@asperasoft.com

• Aspera Intro

• Big data and cloud storage challenges

• FASP protocol

• Deployment options / use cases

• Customer success

• Q&A

AGENDA

ASPERA’S MISSION

Creating next-generation transport technologies

that move the world’s digital assets at maximum speed,

regardless of file size, transfer distance and network conditions.

3

TRENDS

Big Data Explosion

• 90% of data today file-based or unstructured

• Mix of file sizes—but larger and larger files the norm

Diversity of IP Networks—Media, Bandwidth Rates, and Conditions

• Variable bandwidth rates (slow to super-fast)

• Bandwidth rates increasing—costs decreasing

• Network media remains diverse (terrestrial, satellite, wireless)

• Conditions vary—all networks prone to degradation over distance

Global Workflows—moving Big Data over WANs

• Teams are geographically dispersed

• Over distance, network conditions degrade

• Contemporary TCP acceleration solutions not designed for big data transfer and replication

Cloud Computing Grows Up

• Amazon Web Services (AWS) S3 cloud storage – 2010: 262 billion objects, 2012: 1.3 trillion objects

• More choices: IBM SoftLayer, Microsoft Azure, OpenStack, HP Cloud

• No longer a niche – Netflix (transcoding), MTV (global video distribution), BGI (genomic sequencing)

4

WHY CLOUD COMPUTING?

5

• Eliminates the need to plan ahead

• Allows companies to quickly meet changes in demand

• Without the lead-time bottleneck

THE POTENTIAL OF INFINITE COMPUTING RESOURCES, ON DEMAND

• Reduce capital outlay and investment risk

• Start small & increase hardware resources to match need

• Auto-scale to meet demand

THE ELIMINATION OF AN UP-FRONT COMMITMENT

• Virtual machines by the hour

• Storage by the month

• Bandwidth by the GB

PAY-FOR-USE RESOURCE MODEL

BIG DATA CLOUD STORAGE CHALLENGE

6

CHALLENGES OF STORING BIG FILES IN THE CLOUD?

DESIGNED AS “SCALABLE DISTRIBUTED OBJECT STORES”

• Google File System, Amazon S3, Hadoop Distributed File System

• Simple read/write of data “blobs”, indexed by a key

• Multiple replicas are distributed across storage for durability and optimized for high throughput access

• Should work well for storing large numbers of files

BUT • “Blob" sizes are small (64 MB/128 MB)

• Large media files must be “chunked”

• 1TB file = transporting and reassembling 10,000+ chunks!

• Data I/O use the standard HTTP protocol

• VERY SLOW at distance

• Single HTTP stream slow even locally (<100 Mbps).

• Multi-chunk APIs impede workflow and are complex

M&E/ BIG-DATA SERVICES REQUIRE HIGH-SPEED SOFTWARE BRIDGE OVER THE WAN

• Large files moved at full bandwidth capacity with global access

• Overcome the WAN and the I/O bottleneck

• Files of any size

• Transparent to the end user (GUI, command line, browser, etc.)

CHALLENGES FOR STORING BIG FILES IN DISTRIBUTED OBJECT “CLOUD” STORAGE

APIs require Large files are divided into chunks (typically 64 MB - 128 MB)

and multiple replicas are distributed across the storage for durability

• 1 TB file requires 10,000 chunks at 100MB per chunk!

Optimized for high throughput access BUT only for *chunk* sizes in parallel,

i.e. no high throughput, whole file read/write

I/O protocol is HTTP only!

• HTTP PUT or GET by chunk

• SLOW over the WAN (due to TCP throughput bottleneck over WAN), e.g. <1-10s

of Megabits/s depending on distance

• Even local I/O is slow unless a parallel HTTP stream write/read is used, e.g. local

file system drivers, S3 fuse, are notoriously slow, e.g. 8-10 Megabytes/s

Security and access control is only as good as the application

Simply no tools for inter-cloud data transfer - Lock In!

Most use cases need high speed transfer, virtually unlimited size, robust

performance & security

HTTP

BIG DATA CLOUD STORAGE CHALLENGE

9

CHALLENGES WITH TCP AND ALTERNATIVE TECHNOLOGIES

Distance degrades conditions on all networks

• Latency (or Round Trip Times) increase

• Packet losses increase

• Fast networks just as prone to degradation

TCP performance degrades with distance

• Throughput bottleneck becomes more severe with

increased latency and packet loss

TCP does not scale with bandwidth

• TCP designed for low bandwidth

• Adding more bandwidth does not improve throughput

Alternative Technologies

• TCP-based - Network latency and packet loss must be low

• Modified TCP – Improves TCP performance but insufficient for fast networks

• UDP traffic blasters - Inefficient and waste bandwidth

• Data caching - Inappropriate for many large file transfer workflows

• Data compression - Time consuming and impractical for certain file types

• CDNs & co-lo build outs - High overhead and expensive to scale

10

FASP — HIGH-PERFORMANCE DATA TRANSPORT

Maximum line-rate WAN transfer speed

• Transfer performance scales with bandwidth independent

of transfer distance and resilient to packet loss

• Optimal end-to-end throughput efficiency

Congestion Avoidance and Policy Control

• Automatic, full utilization of available bandwidth

• On-the-fly prioritization and bandwidth allocation

Uncompromising security and reliability

• Secure, user/endpoint authentication

• AES-128 cryptography in transit & at-rest

Scalable management, monitoring and control

• Real-time progress, performance and bandwidth utilization

• Detailed transfer history, logging, and manifest

Enterprise-Class File Delivery

• Transfers up to thousands of times faster than FTP/HTTP(S)

• Precise and predictable transfer times

• Extreme scalability (concurrency and throughput) 11

FASP – MANAGEMENT WITH ADAPTIVE RATE CONTROL

Extraordinary bandwidth control

• Automatic, full utilization of available

bandwidth

• Protection of other network traffic

• On-the-fly, per flow, user and job

prioritization

• Highly-concurrent transfer stacking

System-wide monitoring and reporting

• Real-time progress and performance

analysis

• Real-time bandwidth utilization

• Detailed transfer history, logging and

manifest

Centralized command and control

• Per transfer, user, group and node

• Manage and create global transfer policies

• Remotely initiate, schedule and automate

transfer jobs

12

BIG DATA CLOUD STORAGE CHALLENGE

13

PARTIAL SOLUTION – MULTI-PART UPLOAD

ASPERA ON-DEMAND WITH DIRECT-TO-CLOUD

Cloud

ASPERA TRANSFER PLATFORM

16

• High-speed Transport

• Unified access to storage

• Security & Access Control

• Tracking and Reporting

• REST and SOAP APIs

ASPERA TRANSFER

PLATFORM

Automation • Aspera Orchestrator

• Telestream

Transformation • Elemental

• encoding.com

• Sony CI

• thePlatform

• Windows Azure

• Zencoder

Media Management • Harmonic

• Media Beacon

• Sony

• Vidispine

Production • Avid

• EVS

Sharing / Collaboration • Aspera Shares

• Aspera faspex

PARTNER

ECOSYSTEM

TRANSFERRING LARGE FILES WITH THE CLOUD FASP VS MULTIPART HTTP

TRANSFER DATA TO CLOUD OVER WAN EFFECTIVE THROUGHPUT

• Typical internet conditions

• 50–250ms latency & 0.1–3% packet loss

• 15 parallel http streams

<10 to 100 Mbps depending on distance

• Aspera FASP transfer over WAN to Cloud up to 1Gbps (per EC2 Extra Large Instance) independent of distance!

10 TB transferred per 24 hours

17

ASPERA ON DEMAND ADVANTAGES

Maximum speed

• Enables large data set transfers over any network at maximum speed, regardless of network conditions or distance

• Transfers large data sets of small files with the same efficiency as large single files

• Very lightweight - does not require specialized or powerful hardware to maintain high speeds

Adaptive rate control

• Provides precise rate control (pre-set and on-the-fly) for guaranteed transfer times

• Uses an adaptive rate control to fully utilize available bandwidth while remaining fair to other traffic

• Supports on-the-fly configurable bandwidth sharing policies. Users may pre-set and change individual transfer rates and finish

times as needed

Complete security

• Includes complete, built-in security using open standard cryptography for user authentication, data encryption and data integrity

verification

Robust, software only solution

• Uses standard, unmodified IP networking and is implemented in software as an application protocol..

• Automatically resumes partial transfers and retries failed transfers

Flexible open architecture

• Supports interoperable file and directory transfers between all major operating systems and provides a complete, modern

software API to build upon

USE CASE: TRANSFER DIRECT-TO-OBJECT STORAGE

FASP

HTTP –

multipart

HTTP –

multipart

Transfer

Server

Aspera

Client

Client, Emeryville, CA

1. Upload using typical multi-part HTTP client

2. FASP high-speed upload Direct-to-Object storage

1

2

Scale out

Cloud Platform

USE CASE: INTRA-CLOUD TRANSFERS ACROSS SAME OR DIFFERENT CLOUD INFRASTRUCTURE

Node

Node

THE SOLUTION

• Data migration from one region to

another or from one provider to another

• Transfer database or application logs

from one region to another for DR or

Business Continuity

Node

20

FASP

FASP

FASP

Shares

Node Node

DMZ

Datacenter, Emeryville, CA

Client, NY

THE SOLUTION

• Shares Web app transparently communicates with

Aspera server Nodes and displays content in a

single user interface

• User browses authorized content across multiple

shares

• Independent high-speed data transfers to/from

Datacenter, AWS S3, and Windows Azure BLOB,

transparent to user

USE CASE: SHARING AND COLLABORATION HYBRID ACROSS PUBLIC & PRIVATE CLOUDS

21

FASP

FASP

Cloud Platform

USE CASE: PERSON-TO-PERSON DELIVERY WITH DIRECT-TO-OBJECT

Connect

Browser

Plug-in

Direct to

Object

faspex

Person-to-

Person

Connect

Browser

Plug-in

Los Angeles New York

THE SOLUTION

• Faspex upload to cloud and notification

to recipient

• Faspex download direct from cloud

22

FASP

FASP

Cloud Platform

ASPERA TRANSFER SERVER INTEGRATED WITH CLOUD SERVICES

HTTP 1

2 3 4 Client, NY

Cloud Platform

HTTP –

multipart

Aspera

On Demand

• User authenticates to SaaS web application

• SaaS application communicates with Aspera via

Node API

• Content is uploaded to and downloaded from

Object storage via Aspera on Demand

• SaaS application accesses content in object

storage for processing

THE SOLUTION

FASP

ASPERA ON DEMAND: AUTO-SCALING PLATFORM

KEY

COMPONENTS

• Cluster Manager for Auto-scale and Scaled DB

• Console Management UI + Reporting API • Enhanced Client for Shares

Authorizations • Unified Access to Files/Directories

(Browser, GUI, Commend Line, SDK)

AUTO SCALING

• Min/max number of t/s • Utilization low/high watermark • Min number of t/s in “available” pool • Min number of idle t/s in ”available” pool

CLUSTER MGR

• Monitor cluster nodes • Determine eligibility for transfer scale up / down • Create / remove db with replicas • Add / remove node

CUSTOMER SUCCESS STORIES Aspera On Demand

ULTIMATE FIGHTING CHAMPIONSHIP (UFC)

About UFC

• UFC® is the world’s leading promoter of mixed martial arts (MMA), with

programming broadcast to half a billion homes throughout 150 countries.

Challenge

• Fights are located worldwide, from Ireland to Brazil to Las Vegas, Nevada,

frequently in locations with poor connectivity.

• Transferring high-resolution video content from venue to host site in the cloud for scale-out transcoding

Solution

• Aspera On Demand Application Platform running on AWS EC2 instances and Connect browser plug-in to transfer

video clips at high-speed from the different venues to AWS.

• Production teams transfer the content from their laptops using the install-on-demand Connect plug-in, and the

Aspera Application Platform stores the content directly in S3.

• Parallel encoding jobs are automatically started using encoding.com’s cloud-based transcoding service, which

also backstops on AWS, and the resulting output files are stored back into S3, ready for distribution.

• Final delivery of the device-specific content is accomplished using Amazon CloudFront, and all the URLs are

automatically created and available for insertion into the UFC website and can be sent to 3rd party content

providers.

26

ULTIMATE FIGHTING CHAMPIONSHIP (UFC)

27

HTTP –

multipart

Parallel Transcoding

14 EC2 instances: 3 min

`

Herndon, VA

THE SOLUTION

1. Video Broadcast Capture

2. High-speed upload of content directly into AWS S3

3. Transfer completion triggers parallel transcoding in the cloud and

device-specific videos are saved to S3

4. Using AWS CloudFront, content is delivered to viewers’ devices

1

2

3

4

Connect Server

running on EC2

Aspera Connect

browser plug-in

FASP

NETFLIX TRANSCODING IN THE CLOUD

About Netflix

• Netflix delivers streaming movies and TV shows to over 700 different devices

– PCs, internet-connected TVs, gaming consoles, tablets and smart phones.

Challenge

• Scale-out transcoding without large upfront investments in compute and storage

• Fast turn around times for end-to-end processing, from content ingest to distribution

• Usage-based licensing to match timing of large content purchases

• Smooth transition for partners, minimal changes to their workflow

Solution

• Aspera On Demand with Direct-to-S3 technology to move the content into the Amazon Web Services S3 storage

for transcoding and streaming out to its customers.

• With Aspera On Demand Direct-to-S3, Netflix can achieve transfer rate improvements of up to 10x the typical

transfer speeds of available cloud transfer solutions per virtual machine instance.

• Shares On Demand for content ingest to secure folders, abstracting the final location of the stored content

28

NETFLIX TRANSCODING IN THE CLOUD

Shares

Browser Client

THE SOLUTION

• Shares Web app transparently communicates with

Aspera server Nodes and displays content in a

single user interface

• Aspera Client uploads directly via Aspera Transfer

Servers

• CLI transfers also supported

• Independent high-speed data transfers to AWS S3

transparent to user

Node

29

Node Desktop Client

Node Node

FASP

FASP

About

• Rovio Entertainment Ltd is the creator of the globally successful

Angry Birds which became an international phenomenon and has

expanded rapidly into multifaceted entertainment, publishing and

licensing to become a beloved international brand.

Challenge

• Reliably move Angry Birds 52 part animation series to more

than 20 broadcast partners around the world

Solution

• Aspera faspex on Demand to accelerate content delivery to Rovio broadcast partners around the world

and to streamline the related operational processes

• Each episode is first produced to high-quality animation, then encoded for each broadcaster

• Final output is delivered to broadcasters via Aspera faspex on Demand, avoiding the need for

infrastructure build out and maintenance

30

ROVIO ENTERTAINMENT

THANK YOU Questions?

FOR MORE INFORMATION ON ANY ASPERA PRODUCT, PLEASE VISIT

WWW.ASPERASOFT.COM