transforming cloud infrastructure
Post on 03-Jan-2017
224 Views
Preview:
TRANSCRIPT
TRANSFORMING CLOUD INFRASTRUCTURE
TO SUPPORT BIG DATA STORAGE AND WORKFLOWS
PRESENTER AND AGENDA
PRESENTER
Jay Migliaccio
Director
Cloud platforms and Services jay@asperasoft.com
• Aspera Intro
• Big data and cloud storage challenges
• FASP protocol
• Deployment options / use cases
• Customer success
• Q&A
AGENDA
ASPERA’S MISSION
Creating next-generation transport technologies
that move the world’s digital assets at maximum speed,
regardless of file size, transfer distance and network conditions.
3
TRENDS
Big Data Explosion
• 90% of data today file-based or unstructured
• Mix of file sizes—but larger and larger files the norm
Diversity of IP Networks—Media, Bandwidth Rates, and Conditions
• Variable bandwidth rates (slow to super-fast)
• Bandwidth rates increasing—costs decreasing
• Network media remains diverse (terrestrial, satellite, wireless)
• Conditions vary—all networks prone to degradation over distance
Global Workflows—moving Big Data over WANs
• Teams are geographically dispersed
• Over distance, network conditions degrade
• Contemporary TCP acceleration solutions not designed for big data transfer and replication
Cloud Computing Grows Up
• Amazon Web Services (AWS) S3 cloud storage – 2010: 262 billion objects, 2012: 1.3 trillion objects
• More choices: IBM SoftLayer, Microsoft Azure, OpenStack, HP Cloud
• No longer a niche – Netflix (transcoding), MTV (global video distribution), BGI (genomic sequencing)
4
WHY CLOUD COMPUTING?
5
• Eliminates the need to plan ahead
• Allows companies to quickly meet changes in demand
• Without the lead-time bottleneck
THE POTENTIAL OF INFINITE COMPUTING RESOURCES, ON DEMAND
• Reduce capital outlay and investment risk
• Start small & increase hardware resources to match need
• Auto-scale to meet demand
THE ELIMINATION OF AN UP-FRONT COMMITMENT
• Virtual machines by the hour
• Storage by the month
• Bandwidth by the GB
PAY-FOR-USE RESOURCE MODEL
BIG DATA CLOUD STORAGE CHALLENGE
6
CHALLENGES OF STORING BIG FILES IN THE CLOUD?
DESIGNED AS “SCALABLE DISTRIBUTED OBJECT STORES”
• Google File System, Amazon S3, Hadoop Distributed File System
• Simple read/write of data “blobs”, indexed by a key
• Multiple replicas are distributed across storage for durability and optimized for high throughput access
• Should work well for storing large numbers of files
BUT • “Blob" sizes are small (64 MB/128 MB)
• Large media files must be “chunked”
• 1TB file = transporting and reassembling 10,000+ chunks!
• Data I/O use the standard HTTP protocol
• VERY SLOW at distance
• Single HTTP stream slow even locally (<100 Mbps).
• Multi-chunk APIs impede workflow and are complex
M&E/ BIG-DATA SERVICES REQUIRE HIGH-SPEED SOFTWARE BRIDGE OVER THE WAN
• Large files moved at full bandwidth capacity with global access
• Overcome the WAN and the I/O bottleneck
• Files of any size
• Transparent to the end user (GUI, command line, browser, etc.)
CHALLENGES FOR STORING BIG FILES IN DISTRIBUTED OBJECT “CLOUD” STORAGE
APIs require Large files are divided into chunks (typically 64 MB - 128 MB)
and multiple replicas are distributed across the storage for durability
• 1 TB file requires 10,000 chunks at 100MB per chunk!
Optimized for high throughput access BUT only for *chunk* sizes in parallel,
i.e. no high throughput, whole file read/write
I/O protocol is HTTP only!
• HTTP PUT or GET by chunk
• SLOW over the WAN (due to TCP throughput bottleneck over WAN), e.g. <1-10s
of Megabits/s depending on distance
• Even local I/O is slow unless a parallel HTTP stream write/read is used, e.g. local
file system drivers, S3 fuse, are notoriously slow, e.g. 8-10 Megabytes/s
Security and access control is only as good as the application
Simply no tools for inter-cloud data transfer - Lock In!
Most use cases need high speed transfer, virtually unlimited size, robust
performance & security
HTTP
BIG DATA CLOUD STORAGE CHALLENGE
9
CHALLENGES WITH TCP AND ALTERNATIVE TECHNOLOGIES
Distance degrades conditions on all networks
• Latency (or Round Trip Times) increase
• Packet losses increase
• Fast networks just as prone to degradation
TCP performance degrades with distance
• Throughput bottleneck becomes more severe with
increased latency and packet loss
TCP does not scale with bandwidth
• TCP designed for low bandwidth
• Adding more bandwidth does not improve throughput
Alternative Technologies
• TCP-based - Network latency and packet loss must be low
• Modified TCP – Improves TCP performance but insufficient for fast networks
• UDP traffic blasters - Inefficient and waste bandwidth
• Data caching - Inappropriate for many large file transfer workflows
• Data compression - Time consuming and impractical for certain file types
• CDNs & co-lo build outs - High overhead and expensive to scale
10
FASP — HIGH-PERFORMANCE DATA TRANSPORT
Maximum line-rate WAN transfer speed
• Transfer performance scales with bandwidth independent
of transfer distance and resilient to packet loss
• Optimal end-to-end throughput efficiency
Congestion Avoidance and Policy Control
• Automatic, full utilization of available bandwidth
• On-the-fly prioritization and bandwidth allocation
Uncompromising security and reliability
• Secure, user/endpoint authentication
• AES-128 cryptography in transit & at-rest
Scalable management, monitoring and control
• Real-time progress, performance and bandwidth utilization
• Detailed transfer history, logging, and manifest
Enterprise-Class File Delivery
• Transfers up to thousands of times faster than FTP/HTTP(S)
• Precise and predictable transfer times
• Extreme scalability (concurrency and throughput) 11
FASP – MANAGEMENT WITH ADAPTIVE RATE CONTROL
Extraordinary bandwidth control
• Automatic, full utilization of available
bandwidth
• Protection of other network traffic
• On-the-fly, per flow, user and job
prioritization
• Highly-concurrent transfer stacking
System-wide monitoring and reporting
• Real-time progress and performance
analysis
• Real-time bandwidth utilization
• Detailed transfer history, logging and
manifest
Centralized command and control
• Per transfer, user, group and node
• Manage and create global transfer policies
• Remotely initiate, schedule and automate
transfer jobs
12
BIG DATA CLOUD STORAGE CHALLENGE
13
PARTIAL SOLUTION – MULTI-PART UPLOAD
ASPERA ON-DEMAND WITH DIRECT-TO-CLOUD
Cloud
ASPERA TRANSFER PLATFORM
16
• High-speed Transport
• Unified access to storage
• Security & Access Control
• Tracking and Reporting
• REST and SOAP APIs
ASPERA TRANSFER
PLATFORM
Automation • Aspera Orchestrator
• Telestream
Transformation • Elemental
• encoding.com
• Sony CI
• thePlatform
• Windows Azure
• Zencoder
Media Management • Harmonic
• Media Beacon
• Sony
• Vidispine
Production • Avid
• EVS
Sharing / Collaboration • Aspera Shares
• Aspera faspex
PARTNER
ECOSYSTEM
TRANSFERRING LARGE FILES WITH THE CLOUD FASP VS MULTIPART HTTP
TRANSFER DATA TO CLOUD OVER WAN EFFECTIVE THROUGHPUT
• Typical internet conditions
• 50–250ms latency & 0.1–3% packet loss
• 15 parallel http streams
<10 to 100 Mbps depending on distance
• Aspera FASP transfer over WAN to Cloud up to 1Gbps (per EC2 Extra Large Instance) independent of distance!
10 TB transferred per 24 hours
17
ASPERA ON DEMAND ADVANTAGES
Maximum speed
• Enables large data set transfers over any network at maximum speed, regardless of network conditions or distance
• Transfers large data sets of small files with the same efficiency as large single files
• Very lightweight - does not require specialized or powerful hardware to maintain high speeds
Adaptive rate control
• Provides precise rate control (pre-set and on-the-fly) for guaranteed transfer times
• Uses an adaptive rate control to fully utilize available bandwidth while remaining fair to other traffic
• Supports on-the-fly configurable bandwidth sharing policies. Users may pre-set and change individual transfer rates and finish
times as needed
Complete security
• Includes complete, built-in security using open standard cryptography for user authentication, data encryption and data integrity
verification
Robust, software only solution
• Uses standard, unmodified IP networking and is implemented in software as an application protocol..
• Automatically resumes partial transfers and retries failed transfers
Flexible open architecture
• Supports interoperable file and directory transfers between all major operating systems and provides a complete, modern
software API to build upon
USE CASE: TRANSFER DIRECT-TO-OBJECT STORAGE
FASP
HTTP –
multipart
HTTP –
multipart
Transfer
Server
Aspera
Client
Client, Emeryville, CA
1. Upload using typical multi-part HTTP client
2. FASP high-speed upload Direct-to-Object storage
1
2
Scale out
Cloud Platform
USE CASE: INTRA-CLOUD TRANSFERS ACROSS SAME OR DIFFERENT CLOUD INFRASTRUCTURE
Node
Node
THE SOLUTION
• Data migration from one region to
another or from one provider to another
• Transfer database or application logs
from one region to another for DR or
Business Continuity
Node
20
FASP
FASP
FASP
Shares
Node Node
DMZ
Datacenter, Emeryville, CA
Client, NY
THE SOLUTION
• Shares Web app transparently communicates with
Aspera server Nodes and displays content in a
single user interface
• User browses authorized content across multiple
shares
• Independent high-speed data transfers to/from
Datacenter, AWS S3, and Windows Azure BLOB,
transparent to user
USE CASE: SHARING AND COLLABORATION HYBRID ACROSS PUBLIC & PRIVATE CLOUDS
21
FASP
FASP
Cloud Platform
USE CASE: PERSON-TO-PERSON DELIVERY WITH DIRECT-TO-OBJECT
Connect
Browser
Plug-in
Direct to
Object
faspex
Person-to-
Person
Connect
Browser
Plug-in
Los Angeles New York
THE SOLUTION
• Faspex upload to cloud and notification
to recipient
• Faspex download direct from cloud
22
FASP
FASP
Cloud Platform
ASPERA TRANSFER SERVER INTEGRATED WITH CLOUD SERVICES
HTTP 1
2 3 4 Client, NY
Cloud Platform
HTTP –
multipart
Aspera
On Demand
• User authenticates to SaaS web application
• SaaS application communicates with Aspera via
Node API
• Content is uploaded to and downloaded from
Object storage via Aspera on Demand
• SaaS application accesses content in object
storage for processing
THE SOLUTION
FASP
ASPERA ON DEMAND: AUTO-SCALING PLATFORM
KEY
COMPONENTS
• Cluster Manager for Auto-scale and Scaled DB
• Console Management UI + Reporting API • Enhanced Client for Shares
Authorizations • Unified Access to Files/Directories
(Browser, GUI, Commend Line, SDK)
AUTO SCALING
• Min/max number of t/s • Utilization low/high watermark • Min number of t/s in “available” pool • Min number of idle t/s in ”available” pool
CLUSTER MGR
• Monitor cluster nodes • Determine eligibility for transfer scale up / down • Create / remove db with replicas • Add / remove node
CUSTOMER SUCCESS STORIES Aspera On Demand
ULTIMATE FIGHTING CHAMPIONSHIP (UFC)
About UFC
• UFC® is the world’s leading promoter of mixed martial arts (MMA), with
programming broadcast to half a billion homes throughout 150 countries.
Challenge
• Fights are located worldwide, from Ireland to Brazil to Las Vegas, Nevada,
frequently in locations with poor connectivity.
• Transferring high-resolution video content from venue to host site in the cloud for scale-out transcoding
Solution
• Aspera On Demand Application Platform running on AWS EC2 instances and Connect browser plug-in to transfer
video clips at high-speed from the different venues to AWS.
• Production teams transfer the content from their laptops using the install-on-demand Connect plug-in, and the
Aspera Application Platform stores the content directly in S3.
• Parallel encoding jobs are automatically started using encoding.com’s cloud-based transcoding service, which
also backstops on AWS, and the resulting output files are stored back into S3, ready for distribution.
• Final delivery of the device-specific content is accomplished using Amazon CloudFront, and all the URLs are
automatically created and available for insertion into the UFC website and can be sent to 3rd party content
providers.
26
ULTIMATE FIGHTING CHAMPIONSHIP (UFC)
27
HTTP –
multipart
Parallel Transcoding
14 EC2 instances: 3 min
`
Herndon, VA
THE SOLUTION
1. Video Broadcast Capture
2. High-speed upload of content directly into AWS S3
3. Transfer completion triggers parallel transcoding in the cloud and
device-specific videos are saved to S3
4. Using AWS CloudFront, content is delivered to viewers’ devices
1
2
3
4
Connect Server
running on EC2
Aspera Connect
browser plug-in
FASP
NETFLIX TRANSCODING IN THE CLOUD
About Netflix
• Netflix delivers streaming movies and TV shows to over 700 different devices
– PCs, internet-connected TVs, gaming consoles, tablets and smart phones.
Challenge
• Scale-out transcoding without large upfront investments in compute and storage
• Fast turn around times for end-to-end processing, from content ingest to distribution
• Usage-based licensing to match timing of large content purchases
• Smooth transition for partners, minimal changes to their workflow
Solution
• Aspera On Demand with Direct-to-S3 technology to move the content into the Amazon Web Services S3 storage
for transcoding and streaming out to its customers.
• With Aspera On Demand Direct-to-S3, Netflix can achieve transfer rate improvements of up to 10x the typical
transfer speeds of available cloud transfer solutions per virtual machine instance.
• Shares On Demand for content ingest to secure folders, abstracting the final location of the stored content
28
NETFLIX TRANSCODING IN THE CLOUD
Shares
Browser Client
THE SOLUTION
• Shares Web app transparently communicates with
Aspera server Nodes and displays content in a
single user interface
• Aspera Client uploads directly via Aspera Transfer
Servers
• CLI transfers also supported
• Independent high-speed data transfers to AWS S3
transparent to user
Node
29
Node Desktop Client
Node Node
FASP
FASP
About
• Rovio Entertainment Ltd is the creator of the globally successful
Angry Birds which became an international phenomenon and has
expanded rapidly into multifaceted entertainment, publishing and
licensing to become a beloved international brand.
Challenge
• Reliably move Angry Birds 52 part animation series to more
than 20 broadcast partners around the world
Solution
• Aspera faspex on Demand to accelerate content delivery to Rovio broadcast partners around the world
and to streamline the related operational processes
• Each episode is first produced to high-quality animation, then encoded for each broadcaster
• Final output is delivered to broadcasters via Aspera faspex on Demand, avoiding the need for
infrastructure build out and maintenance
30
ROVIO ENTERTAINMENT
THANK YOU Questions?
FOR MORE INFORMATION ON ANY ASPERA PRODUCT, PLEASE VISIT
WWW.ASPERASOFT.COM
top related