tachyon: an open source memory-centric distributed storage system

Haoyuan Li, Tachyon Nexushaoyuan@tachyonnexus.com

September 30, 2015 @ Strata and Hadoop World NYC 2015

An Open Source Memory-Centric Distributed Storage System

Outline

•  Open Source

•  Introduction to Tachyon

•  New Features

•  Getting Involved

2

Outline

•  Open Source

•  Introduction to Tachyon

•  New Features

•  Getting Involved

3

History •  Started at UC Berkeley AMPLab –  From summer 2012 –  Same lab produced Apache Spark and Apache Mesos

•  Open sourced –  April 2013 –  Apache License 2.0 –  Latest Release: Version 0.7.1 (August 2015)

•  Deployed at > 100 companies

4

Contributors Growth

5

v0.4!Feb ‘14

v0.3!Oct ‘13

v0.2 Apr ‘13

v0.1 Dec ‘12

v0.6!Mar ‘15

v0.5!Jul ‘14

v0.7!Jul ‘15

1 3 15

30

46

70

111

Contributors Growth

6

> 150 Contributors (3x increment over the last Strata NYC)

> 50 Organizations

Contributors Growth

7

One of the Fastest Growing Big Data Open Source Project

Thanks to Contributors and Users!

8

One Tachyon ProductionDeployment Example

•  Baidu (Dominant Search Engine in China, ~ 50 Billion USD Market Cap)

•  Framework: SparkSQL •  Under Storage: Baidu’s File System •  Storage Media: MEM + HDD •  100+ nodes deployment •  1PB+ managed space •  30x Performance Improvement

9

Outline

•  Open Source

•  Introduction to Tachyon

•  New Features

•  Getting Involved

10

Tachyon is an Open Source

Memory-centricDistributed

Storage System 11

12

Why Tachyon?

Performance Trend: Memory is Fast

•  RAM throughput increasing exponentially

•  Disk throughput increasing slowly

13

Memory-locality key to interactive response times

Price Trend: Memory is Cheaper

source:  jcmit.com  14

Realized by many…

15

16

Is the Problem Solved?

17

Missing a Solution for the Storage Layer

A Use Case Example with -

•  Fast, in-memory data processing framework – Keep one in-memory copy inside JVM – Track lineage of operations used to derive data – Upon failure, use lineage to recompute data

map

filter map

join reduce

Lineage Tracking

18

Issue 1

19

Data Sharing is the bottleneck in analytics pipeline:Slow writes to disk

Spark Job1

Spark mem block manager

block 1

block 3

Spark Job2

Spark mem block manager

block 3

block 1

HDFS / Amazon S3 block 1

block 3

block 2

block 4

storage engine & execution engine same process (slow writes)

Issue 1

20

Spark Job

Spark mem block manager

block 1

block 3

Hadoop MR Job

YARN

HDFS / Amazon S3 block 1

block 3

block 2

block 4

Data Sharing is the bottleneck in analytics pipeline:Slow writes to disk

storage engine & execution engine same process (slow writes)

Issue 1 resolved with Tachyon

21

Memory-speed data sharingamong jobs in different

frameworks execution engine & storage engine same process (fast writes)

Spark Job

Spark mem

Hadoop MR Job

YARN

HDFS / Amazon S3 block 1

block 3

block 2

block 4

HDFS  disk  

block  1  

block  3  

block  2  

block  4  Tachyon!in-memory

block 1

block 3 block 4

Issue 2

22

Spark Task

Spark memory block manager

block 1

block 3

HDFS / Amazon S3 block 1

block 3

block 2

block 4

execution engine & storage engine same process

Cache loss when process crashes

Issue 2

23

crash

Spark memory block manager

block 1

block 3

HDFS / Amazon S3 block 1

block 3

block 2

block 4

execution engine & storage engine same process

Cache loss when process crashes

HDFS / Amazon S3

Issue 2

24

block 1

block 3

block 2

block 4

execution engine & storage engine same process

crash

Cache loss when process crashes

HDFS / Amazon S3 block 1

block 3

block 2

block 4 Tachyon!in-memory

block 1

block 3 block 4

Issue 2 resolved with Tachyon

25

Spark Task

Spark memory block manager

execution engine & storage engine same process

Keep in-memory data safe,even when a job crashes.

Issue 2 resolved with Tachyon

26

HDFS  disk  

block  1  

block  3  

block  2  

block  4  

execution engine & storage engine same process

Tachyon!in-memory

block 1

block 3 block 4

crash

HDFS / Amazon S3 block 1

block 3

block 2

block 4

Keep in-memory data safe,even when a job crashes.

HDFS / Amazon S3

Issue 3

27

In-memory Data Duplication & Java Garbage Collection

Spark Job1

Spark mem block manager

block 1

block 3

Spark Job2

Spark mem block manager

block 3

block 1

block 1

block 3

block 2

block 4

execution engine & storage engine same process (duplication & GC)

Issue 3 resolved with Tachyon

28

No in-memory data duplication,much less GC

Spark Job1

Spark mem

Spark Job2

Spark mem

HDFS / Amazon S3 block 1

block 3

block 2

block 4

execution engine & storage engine same process (no duplication & GC)

HDFS  disk  

block  1  

block  3  

block  2  

block  4  Tachyon!in-memory

block 1

block 3 block 4

Previously Mentioned

•  A memory-centric storage architecture

•  Push lineage down to storage layer

29

Tachyon Memory-Centric Architecture

30

Tachyon Memory-Centric Architecture

31

Lineage in Tachyon

32

Outline

•  Open Source

•  Introduction to Tachyon

•  New Features

•  Getting Involved

33

1) Eco-system: Enable new workload in any storage;

Work with the framework of your choice;

34

2) Tachyon running in production environment,

both in the Cloud and on Premise.

35

Use Case: Baidu

•  Framework: SparkSQL •  Under Storage: Baidu’s File System •  Storage Media: MEM + HDD •  100+ nodes deployment •  1PB+ managed space •  30x Performance Improvement

36

Use Case: a SAAS Company

•  Framework: Impala

•  Under Storage: S3

•  Storage Media: MEM + SSD

•  15x Performance Improvement

37

Use Case: an Oil Company

•  Framework: Spark

•  Under Storage: GlusterFS

•  Storage Media: MEM only

•  Analyzing data in traditional storage

38

Use Case: a SAAS Company

•  Framework: Spark

•  Under Storage: S3

•  Storage Media: SSD only

•  Elastic Tachyon deployment

39

40

What if data size exceeds memory capacity?

41

3) Tiered Storage:Tachyon Manages More Than DRAM

MEM SSD

HDD

Faster

Higher Capacity

42

Configurable Storage Tiers

MEM only

MEM + HHD

SSD only

43

4) Pluggable Data Management Policy

Evict stale data to lower tier

Promote hot data to upper tier

44

Pin Data in Memory

5) Transparent Naming

45

6) Unified Namespace

46

More Features

•  7) Remote Write Support •  8) Easy deployment with Mesos and Yarn •  9) Initial Security Support •  10) One Command Cluster Deployment •  11) Metrics Reporting for Clients, Workers,

and Master

47

12) More Under Storage Supports

48

Reported Tachyon Usage

49

Outline

•  Open Source

•  Introduction to Tachyon

•  New Features

•  Getting Involved

50

Memory-Centric Distributed Storage

Welcome to try, contact, and collaborate!

51

JIRA New Contributor Tasks

•  Team consists of Tachyon creators, top contributors

•  Series A ($7.5 million) from Andreessen Horowitz

•  Committed to Tachyon Open Source

52

53

Strata NYC 2015

•  Welcome to visit us at our booth #P18.

•  Check out other Tachyon related talks. –  First-ever scalable, distributed deep learning architecture

using Spark and Tachyon •  Christopher Nguyen (Adatao, Inc.), Vu Pham (Adatao, Inc) •  2:05pm–2:45pm Thursday, 10/01/2015

–  Faster time to insight using Spark, Tachyon, and Zeppelin •  Nirmal Ranganathan (Rackspace Hosting) •  2:05pm–2:45pm Thursday, 10/01/2015

54

•  Try Tachyon: http://tachyon-project.org

•  Develop Tachyon: https://github.com/amplab/tachyon

•  Meet Friends: http://www.meetup.com/Tachyon

•  Get News: http://goo.gl/mwB2sX

•  Tachyon Nexus: http://www.tachyonnexus.com •  Contact us: haoyuan@tachyonnexus.com

55