Enabling The Big Data Cloud for HPC and Collaboration With High-Speed Data Transport

PRESENTER AND AGENDA

PRESENTER

Daniel KumiDirector, New Market Development daniel@asperasoft.com • Who and Why Aspera?

• WAN Transport

• Wireless Transport

• Cloud and Big Data – Data Transfer Challenges for HPC and Collaboration

• Aspera On Demand

• Aspera-HP Discussion

AGENDA

Trends

Big Data Explosion• 90% of data today file-based or unstructured• Mix of file sizes—but larger and larger files the norm

Diversity of IP Networks—Media, Bandwidth Rates, and Conditions• Variable bandwidth rates (slow to super-fast)• Bandwidth rates increasing—costs decreasing• Network media remains diverse (terrestrial, satellite, wireless) • Conditions vary—all networks prone to degradation over distance

Data Freighting Challenges—moving Big Data over WANs• Teams are geographically dispersed• Over distance, network conditions degrade• Contemporary TCP acceleration solutions not designed for big data transfer and replication

Cloud Computing Grows Up• Amazon Web Services (AWS) S3 cloud storage – 2010: 262 billion objects, 2012: 905 billion

objects• More choices: Microsoft Azure, OpenStack, HP Cloud• No longer a niche – Netflix (transcoding), MTV (global video distribution), BGI (genomic

sequencing)

ASPERA’S MISSION

Creating next-generation transport technologies

that move the world’s digital assets at maximum speed,

regardless of file size, transfer distance and network conditions.

Aspera: moving the world’s digital assets at maximum speed

Expanded to Asia PAC and Latin America through direct and channel

50% YOY growth in revenue and employees

Over 12,000 licenses sold, and over 1,500 customers world wide

Patents issued or pending in 32 countries

Continuing to innovate: fasp3™, fasp-MC™, mobile transport, cloud enablement

MARKETS SERVED

.Media and Entertainment, Federal Government, Life Sciences, Healthcare, Cloud Computing, Software and Gaming, Financial Services, Legal, eDiscovery, Engineering, Technology, Telecommunications, Service Providers, Architecture and Design, Enterprise IT, Oil and Gas

Aspera Ecosystem of Partners

Life SciencesAspera in Life Sciences

BIG DATA TRANSFER CHALLENGE

What Happened to my Bandwidth?

1000 Mbps• 170ms RTT• 0.001% packet loss rate Paris

Seattle

WAN Throughput is 1000Mbps

Max TCP Throughput ~29Mbps

Where’s my 970Mbps?

At 29Mbps50GB transfer will take 4 hrs1TB transfer will take 3.3 days

WAN

BIG-DATA and WAN TRANSFER WITH TCP

TCP WAS DESIGNED IN THE EARLY 80’S• When data was small & bandwidth was limited• Fantastic for reliable data delivery• Not fast enough for big-data

TCP IS THE ENGINE THAT DRIVES• FTP, HTTP & HTTPS• RSYNC, SCP & DICOM• CIFS & NFS

TCP DOES NOT LIKE NETWORK LATENCY/ RTT• Geographic distance increases latency• Network congestion increases latency

TCP DOES NOT LIKE PACKET LOSS• Loss is caused by congestion• Different network capacity• Wireless and satellite communications

The Aspera SolutionSo if TCP doesn’t work, what’s the answer?

WAN is 1000Mbps

Max TCP Throughput ~29Mbps

Max Aspera Throughput ~995Mbps (gain of x34)

ROI measured in $$ cost of not using 971Mbps

Same WAN Scenario with Aspera fasp

1000 Mbps• 170ms RTT• 0.001% packet loss rate ParisSeattle

WAN

At 995 Mbps• 50GB transfer will take ~4 hrs• 50GB transfer will take ~7 mins

• 1TB transfer will take 3.3 days• 1TB transfer will take 2.4 hrs

FASP™ — HIGH-PERFORMANCE DATA TRANSPORT

MAXIMUM LINE-RATE WAN TRANSFER SPEED• Transfer performance scales with bandwidth independent

of transfer distance and resilient to packet loss• Optimal end-to-end throughput efficiency

CONGESTION AVOIDANCE AND POLICY CONTROL• Automatic, full utilization of available bandwidth• On-the-fly prioritization and bandwidth allocation

UNCOMPROMISING SECURITY AND RELIABILITY• Secure, user/endpoint authentication • AES-128 cryptography in transit & at-rest

SCALABLE MANAGEMENT, MONITORING AND CONTROL• Real-time progress, performance and bandwidth utilization• Detailed transfer history, logging, and manifest

ENTERPRISE-CLASS FILE DELIVERY• Transfers up to thousands of times faster than FTP/HTTP(S)• Precise and predictable transfer times• Extreme scalability (concurrency and throughput)

fasp Bandwidth ROI

FTP Across US US – EU US – ASIA Satellite

1 GB 1 – 2 hrs 2 – 4 hrs 4 – 20 hrs 8 – 20 hrs

10 GB 15 – 20 hrs 20 – 40 hrs Impractical Impractical

100 GB Impractical Impractical Impractical Impractical

fasp™ 2 Mbps 10 Mbps 45 Mbps 100 Mbps 200 Mbps 1 Gbps

1 GB 70 min. 14 min. 3.2 min. 1.4 min. 42 sec. 8.4 sec.

10 GB 11.7 hrs 140 min. 32 min. 14 min. 7 min. 1.4 min.

100 GB 23.3 hrs 5.3 hrs 2.3 hrs 1.2 hrs 14 min.

FTP: Limited by Distance & Packet Loss, Not B/W

Aspera: Scales Linearly with Bandwidth

Distance & Packet Loss Independent

FASP vs TCP PERFORMANCE

6 Gbps Scalable WAN Throughput

~6Gbps Big-Data Throughput• Latency independent• Loss independent

x3000 improvement vs. TCP• 1TB data moved in 20 min• 2 days with TCP over LAN conditions

Scale to ~10Gbps with IQ Accelerator

Aspera mobile app SUITENew FEATURES AND CAPABILITIES

Technology Advantages• Up to 3x faster than 3G, up to 100x faster than 802.11n• Complete integration into native device environment

iOS faspex™ App for iPhone and iPad• High-speed faspex mobile transfers• Native iOS app look and feel, and Photo Gallery

integration• Familiar faspex™ workflow with visual style of iOS email

app• Supports faspex™ security models (e.g. encryption at

rest)

Android Client App• High-speed transfers to/from Enterprise or Connect

Servers• Upload content from camera, photo gallery, or file

system • Browse Connect Server’s listings and download to the

device• Upload geo-tagged content using device’s position

Application Uses• Remote news collection, upload and publishing to central

hub• Viewing and approving content as part of a file based

workflow• Remote medical image viewing and diagnosis• Use fasp-AIR SDK to embed high-speed transfers into

your mobile apps

fasp-AIR Benchmarks on Verizon 4G

In some cases (highlighted in orange), speeds will vary greatly, depending on available bandwidth and the underlying condition of the wireless network.

Our unique, patented faspTM transport technologies provide unparalleled speed, efficiency, and bandwidth control over any file size, transfer distance, network condition and storage

location (on-premise or cloud).

TRANSPORT

Aspera Software Product & Technology Portfolio

Complete portfolio of servers and clients for

high-speed data delivery and distribution.

DISTRIBUTE

Global person-to-person and project-based exchange and collaboration of files

and directories.

COLLABORATE

Web-based application and SDK for creating

and managing automated file-based

workflows.

AUTOMATE

fasp™ Software Environment

ASPERA DEVELOPER NETWORK

A complete set of SDKs provides developers with guides, reference information, and sample code to assist them with integrating Aspera technology into their own applications. Aspera fasp™ technology can be used in desktop, network-based, and web applications in place of FTP, HTTP, or custom TCP-based copy protocols.

ASPERA MOBILE APIs

Android SDKAspera Android SDK provides a Java API to transfer files using fasp-AIR™.

iPhone SDKAspera iPhone SDK provides an Objective C API to transfer files using fasp-AIR.

ASPERA APPLICATION APIsfaspex™ Web APIThe Aspera faspex Web API provides a set of services that enables users to create and receive digital deliveries via a Web interface, while taking advantage of fasp high-speed transfer technology.

OTHER INFORMATION

Supporting Tools and LibrariesSupporting tools and libraries let you perform other common tasks surrounding file transfers.

General ReferenceReference on error codes, log file locations, configuration files and more.

ASPERA TRANSFER APIs

Aspera Web ServicesA SOAP based web service API that allows initiation, monitoring and controlling of fasp based file transfers.

Aspera WebJavascript API exposed by Aspera Connect client. It allows integration of fasp based file transfers into web applications.

Connect 2.8 developer Preview 2Introducing the new Connect 2.8 developer preview! Integrate the functionality of Aspera Connect 2.8, a fasp-based file transfer client, into your own web applications, while customizing it to your unique brand.

fasp ManagerA class library that allows intiations, monitoring and controlling of fasp based file transfers.

Aspera Multicast SDKA Java class library that allows initiation and management of IP multicast based data transmissions using Aspera fasp-MC™.

CLOUD COMPUTING & BIG DATA

Cloud Computing – Why is it so compelling?

Pay-for-use resource model• CPU’s by the hour• Storage by the day• Bandwidth by the GB Cloud

The elimination of an up-front commitment• Reduce capital outlay and investment risk• Start small & increase h/w resources to match need• Auto-scale to meet demand

The potential of infinite computing resources, on demand

• Eliminates the need to plan ahead• Allows companies to meet demand• Without the lead-time bottleneck

SO? WHAT CAN I DO WITH IT?

• Near-line for editing, creative apps and processing• B2B / B2C data workflow• Offsite storage for disaster recovery and business

continuity

• OTT, play out, release, project & event specific marketing

• Collaborative data exchange• CDN and global delivery

• Compute Intensive: 10’s, 100’s, 1000’s of CPU cores• Transcoding, rendering, encoding, watermarking• Big-data analytics & HPC

DATA & CONTENTDISTRIBUTION

DATA PROCESSING & CONTENT CREATION

STORAGE FOR ARCHIVE & D/R

GETTING BIG DATA IN AND OUT OF THE CLOUD

KNOWING WHEN TO CHOSE THE RIGHT TOOL

CHALLENGES OF STORING BIG FILES IN THE CLOUD?

BEWARE THE OBJECT STORE:• Not like traditional NAS or SAN• Bigger, better, but possibly much more complex• a.k.a. Google File System, Amazon S3, Hadoop Distributed File System • Simple read/write of data “blobs”, indexed by a key• Multiple replicas are distributed across storage for durability and optimized for access • Should work well for storing large numbers of files

UNDERSTAND CHUNKS, BLOCKS and BLOBS• You need to deal with chunks, blocks and blobs• “Chunk” sizes are small (64 MB/128 MB)

• Large media files must be “chunked” (1TB file = transporting and reassembling 10,000+ chunks!)• Multi-chunk APIs impede workflow and are complex

• Data I/O use the standard HTTP(s) protocol • VERY SLOW at distance• Single HTTP stream slow even locally (<100 Mbps).

BIG-DATA SERVICES WILL NEED A HIGH-SPEED BRIDGE TO THE CLOUD• Large files moved at full bandwidth capacity with global access• Overcome the WAN and storage bottleneck• Support files of any size or quantity• Transparent to the end user/data owner (GUI, command line, API, browser, etc.)• No hardware to support B2B, B2C, C2B workflow

Big data cloud storage challenge

Cloud

S3 & BIG-DATA: TYPICAL APPLICATION OPTIONS

SOLUTION: ASPERA ON-DEMAND DIRECT-TO-S3

Cloud

OVERCOMING BOTH BOTTLENECKS

#1 — TRANSFER DATA TO EC2 OVER WAN EFFECTIVE THROUGHPUT

• http transfer over WAN (single stream)• Typical internet conditions

• 50–250ms latency & 0.1–3% packet loss• 15 parallel http streams

<10 Mbps

<10 to 100 Mbps

• Aspera fasp transfer over WAN to EC2 up to 1Gbps (per EC2 Extra Large Instance)

#2 — TRANSFER DATA FROM EC2 TO S3 EFFECTIVE THROUGHPUT

• Standard single stream http 10 to 100 Mbps

• Aspera S3 Proxy• With parallel I/O http streams

up to 1Gbps(per EC2 Extra Large Instance)

ASPERA + AWS | ~10 TB transferred per 24 hours | PER EC2 INSTANCE

ASPERA DIRECT-TO-S3 — LINE RATE ACCESS TO THE CLOUD

UNRIVALED ASPERA PERFORMANCE• Built on Aspera fasp™ technology for maximum transfer speed

• Regardless of file size, transfer distance and network conditions• Precise bandwidth control ensures the available bandwidth is utilized to achieve maximum transfer

speeds, while being fair to other business-critical network traffic

SEAMLESS INTEGRATION WITH S3• Integrated with S3 multi-part HTTP for maximum “last foot” performance• Simple configuration of S3 credentials, for both shared and dedicated docroot• Transfers directly into S3 are seamless and transparent to user

ENTERPRISE-GRADE SECURITY AND RELIABILITY• Secure authentication with encryption in transit & at rest (AES-128, FIPS 140-2, HIPPA Compliant)• Packet-level data integrity verification• Automatic resume of partial or failed transfers• Full support for AWS S3 Service-side-encryption at rest

INTEROPERATES WITH ALL ASPERA HOST OPTIONS• Any platform (Windows, Linux, MAC, UNIX, iOS, Android)• Any Aspera Clients (CLI, Desktop, Point-to-Point, Mobile, Web, Embedded)• Any Aspera Servers (Enterprise, Connect, faspex)

ASPERA FOR AWS: DIRECT-TO-S3 : SCALE OUT ARCHITECURE FOR ULTRA HIGH THROUGHPUT

fasp

HTTP – multipa

rt

HTTP – multipart

Aspera TransferServer

Aspera Client

Client, Dallas, TX

1. Upload using typical multi-part HTTP client

2. fasp high-speed upload Direct-to-S3

1

2

Herndon, VA

Scale out

ASPERA SOFTWARE ON DEMAND

KEY FEATURES• On demand high-performance data transport to and from remote infrastructures• Unlimited scale out of transfer capacity with additional AMIs• Support for all Aspera Server software and use cases• Additional Client Options: Mobile, Outlook Plug-in & Cargo (Aspera faspex)• Flexible Storage Options: Local, EBS, AWS S3 • Seamlessly interoperates with on-premise Aspera deployments• Integrated Management and Monitoring

APPLICATIONS AND USE CASE• High Performance Computing On Demand• Content Aggregation, Transformation and Distribution• Time-boxed event or project-based collaboration, ad-hoc distribution or content ingest

Aspera ConsoleGlobal transfer monitoring,

reporting & control

Aspera SharesGlobal Person-to-person file

transfer & exchange

Aspera faspexGlobal Person-to-person file

ingest & distribution

Aspera ServerUniversal file transfer server

supports desktop, web, mobile & embedded

ASPERA shares 1.0MULTI-NODE FILE SHARING

Global Multi-node, High-speed File Sharing• Easy-to-use web application with secure access to a

consolidated view of all shared content• Fully integrated Aspera Connect browser plug-in for high-

speed, secure file transfers• Access to content distributed across any locations

(enterprise servers, private, public and hybrid clouds)• Extremely scalable with web application being decoupled

from storage nodes• Powerful security model administered through a single

management point combining authorization, user management, and access control

• Search, filtering, and sorting capabilities make it easy to find individual files or folders in a very large content store

Application Examples• Internal distribution of digital assets within the enterprise • Collaboration enablement for geographically-distributed

teams • Project team and third-party content or data gathering • Global browser-based distribution of data across external

teams and partners

Aspera Shares Features and Benefits

Powerful security and access model• Secure access administered through a single

management point combining authorization, user management, and access control

• Add and manage users and groups locally, as well as through Active Directory and LDAP

• Complete control over access, such as which nodes are visible, and which nodes, directories and files are accessible

• Granular control over all end-user operations such as browsing, uploading, downloading, making new directories, renaming or deleting files and directories

Easy to use Web interface• Users can easily navigate across files and folders to

locate and initiate a high-speed file upload or download

• A single view consolidates all shared content making it easy to search and navigate across multiple nodes

• Powerful search, filtering, and sorting capabilities simplify finding individual files or folders in a very large content store

• Operations can be performed on a single file or folder, or directly on the search results

HYBRID CLOUD DEPLOYMENT (PUBLIC/PRIVATE)

fasp Shares

NodeNode

Shares app transparently communicates with Aspera server Nodes in cloud and in enterprise

User browses content across authorized shares

High-speed data transfers with Datacenter

High-speed data transfers with Direct-to-S3

DMZ

Herndon, VA

fasp

Datacenter, Emeryville, CA

Client, NY, NY

Aspera faspex™ Overview

Global Person-to-Person File Delivery• Built on Aspera Enterprise Server technology• High-performance fasp™ transport• Person-to-person and project-based file-exchange• Easy to use Web-based interface, email, mobile and

desktop client interfaces• Easily create and manage work groups for file-based

collaboration • Enterprise-scale user management and access

control

Application examples• Internal distribution of digital assets within the

enterprise • Collaboration enablement for geographically-

distributed teams • File-based review, approval and quality assurance

workflows • Digital delivery and collaborative file transfer with

external partners

Person-to-person File Exchange and Collaboration

Aspera faspex™

Aspera faspex™ key features and benefits

Unrivaled Aspera performance • Maximum transfer speed regardless of file size,

transfer distance and network conditions • Secure authentication, full data encryption in transit

and at rest • 100% reliable data delivery

Global File Exchange and Collaboration• Custom Metadata Collection with integrated reporting• Comprehensive and granular package access and

expiration policies• Integrated email notification• API and automation capabilities for file-based

process /workflow integration

Enterprise-scale solution • Comprehensive administration, user management &

access control • Highly concurrent user connection architecture• Remotely monitored and controlled via Aspera

Console – Aspera’s centralized web-based transfer management application.

Comprehensive Security

Full support of the Aspera fasp™ security model• Secure authentication, encryption of the data using strong cryptography, per-packet integrity

verification to protect against man-in-the-middle compromise, and is FIPS-140 2 compliant

Supports all LDAP directory services for import, synchronization, and direct authentication

• Open Directory, Open LDAP, Active Directory

Configurable security options • Automatic deactivation after set number of failed login attempts• Concurrent login prevention, session timeout, strong passwords• Configurable per IP and mask upload, download, login permission• Sending and receiving from 3rd party non registered users, with policy-based expiration

Configurable data encryption in flight and at rest• Package contents can be encrypted over-the-wire, or optionally at rest for complete security• Encrypted packages can be decrypted on the fly when downloaded by the recipient using one of the

faspex™ clients such as the iOS faspex™ Client, Aspera Add-in for Outlook, Aspera Connect browser plug-in, or Aspera Cargo downloader

• The sender may choose when and to whom to distribute the secret, and thus prevent unauthorized users from accessing the content, and control when recipients are able to decrypt the downloaded content

ASPERA faspex 3.0NEW FEATURES AND CAPABILITIES

Remote File Sources (local, enterprise, cloud) • Browse and publish from any storage location• RESTful API for integration and full process automation

Aspera On Demand• Available on demand via AWS subscription• Seamless integration with Amazon S3• Supports all faspex client and mobile options

Support for Active / Active and Tiered Architecture• Active-active high availability configuration with

shared storage (on Linux)• Transfer server and faspex web application can be

deployed on separate hosts for maximum scale out

New Client Applications• Faspex app for iOS support mobile use cases• MS Outlook add-in allows transfers directly from within

Outlook with transparent uploads/downloads via faspex

Self-registration • Enables rapid addition of third parties with both

moderated and un-moderated configuration options

Person-to-person File Exchange and Collaboration

Enterprise Performance, Scalability and Reliability

Unlimited scale-out• faspex™ application and Enterprise Server can be installed

on separate hosts• Allows clustering of Enterprise Server nodes for unlimited

scale-out of transfer capacity (I/O and network capacity)

Multi-server relay• Master server and Relay server configuration• Transfers from one user to another are relayed to

the users’ home server• Simple administration of relay server through the

master with automatic synchronization of user accounts between master and peers

High-availability configuration• faspex™ server is designed to be deployed in an active /

passive HA configuration. Now supports Active/Active• Ensures continuous availability of the faspex™

application• Seamless automatic retry and resume of transfer

sessions on failover assuming shared storage

Faspex On demand with Aspera direct-to-S3

Connect Browser Plug-in

HTTP – multipa

rt

fasp faspexPerson-to-

Person

fasp

Connect Browser Plug-in

1. fasp high-speed upload Direct-to-S3

2. faspex notification

3. fasp high-speed down load Direct-from-S3

1

2

3

Herndon, VA

Client, NY, NY Client, NY, NY

BIG-DATA: ACCESSED & DELIVERED BY ASPERA

File transfer

THANK YOU!

Daniel KumiDirector, New Market Developmentdaniel@asperasoft.com