accessing data anywhere with unified namespace
TRANSCRIPT
![Page 1: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/1.jpg)
Alluxio (formerly Tachyon):Accessing Data Anywhere with Unified Namespace
Jiri Simsa
June 15, 2016 @ Alluxio Meetup (hosted by Intel)
![Page 2: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/2.jpg)
About Me
• Software Engineer @ Alluxio, Inc.
• PMC Member and Maintainer of Alluxio Open Source Project
• Ph.D. from Carnegie Mellon University (Parallel Data Lab)
• Worked at Google before joining Alluxio
• Twitter: @jsimsa, Github: jsimsa
2
![Page 3: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/3.jpg)
Outline
• Motivation
• Unified Namespace
• Use Cases
• Demo
3
![Page 4: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/4.jpg)
Big Data Ecosystem
4
![Page 5: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/5.jpg)
Big Data Ecosystem
5
![Page 6: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/6.jpg)
Big Data Ecosystem
6
![Page 7: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/7.jpg)
Alluxio Benefits
• Future-proofing your applications–applications can communicate with different storage systems, both existing and new, using the same namespace and interface–seamless integration between applications and new storage systems enables faster innovation
• Enabling new workloads–one-time effort to enable an application to access many different types of storage systems and a storage system to be accessed by many different types of applications
7
![Page 8: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/8.jpg)
Outline
• Motivation
• Unified Namespace
• Use Cases
• Demo
8
![Page 9: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/9.jpg)
Unified Namespace
an abstraction that makes it possible for
applications to access different storage
systems through the same interface
9
![Page 10: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/10.jpg)
Transparent Naming
•Operations over persisted Alluxio objects
mapped transparently to underlying storage
•Alluxio paths are preserved in storage layer
Alluxio Storage System (HDFS, S3, …)
alluxio://host:port/
Data Users
Reports Sales Alice Bob
hdfs://host:port/
Data Users
Reports Sales Alice Bob
10
![Page 11: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/11.jpg)
Multiple Storage Systems
•Unified namespace for multiple data sources
•Sharing of data across storage systems
•API for on-the-fly mounting / unmounting
AlluxioStorage System A
alluxio://host:port/
Data Users
Alice Bob
hdfs://host:port/
Users
Alice Bob
Storage System B
s3://host/bucket
Reports SalesReports Sales
11
![Page 12: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/12.jpg)
Outline
• Motivation
• Unified Namespace
• Use Cases
• Demo
12
![Page 13: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/13.jpg)
Multiple Storage / Compute
13
![Page 14: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/14.jpg)
Changing Storage Backend
14
![Page 15: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/15.jpg)
Changing Storage Backend
15
![Page 16: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/16.jpg)
Outline
• Motivation
• Unified Namespace
• Use Cases
• Demo
16
![Page 17: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/17.jpg)
Resources
• Alluxio Project: http://www.alluxio.org
• Development: https://github.com/Alluxio/alluxio
• Meet Friends: http://www.meetup.com/Alluxio
• Alluxio, Inc.: http://www.alluxio.com
• Contact us: [email protected]
![Page 18: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/18.jpg)
Backup Slides
18
![Page 19: Accessing Data Anywhere with Unified Namespace](https://reader036.vdocuments.net/reader036/viewer/2022062223/5875bb7d1a28ab33128b46a3/html5/thumbnails/19.jpg)
Architecture Overview
19