on the road - 情報処理学会fusion & improvement amazon dynamo query model, acid facebook...

Yosuke Hara Nov 14th, 2012

On the Road

1Wednesday, November 14, 12

Table of contents

1. Dive into the Web

2. The Seed And the Sower

3. To the Growth

4. To the Bloom (the Future)


1. Dive into the Web


The Internet of things: http://www.isgtw.org/visualization/internet-things4Wednesday, November 14, 12

http://www.isgtw.org/visualization/internet-things

http://www.isgtw.org/visualization/internet-things

As of 2011

60 SECONDS - THINGS THAT HAPPEN ON INTERNET EVERY SIXTY SECONDS: http://www.go-gulf.com/blog/60-seconds5Wednesday, November 14, 12

http://www.go-gulf.com/blog/60-seconds

http://www.go-gulf.com/blog/60-seconds

X3.0

Amazon S3 - 905 Billion Objects and 650,000 Requests/Second: http://bit.ly/HqAlYV

650,000 req/sec


http://bit.ly/HqAlYV

http://bit.ly/HqAlYV

2. The Seed and the Sower


Awareness


The Seed and the Sower

Set up a hypothesis

Awareness - “Why?”

Survey

Unclear Convince


1. Low ROI2. Possibility of SPOF3. Storage Expansion is difficult during increasing data

?



Will face the situation



Need to store and managehuge amount of files at low-cost.



3. To the Growth


Architecture


Architecture

法隆寺: http://bit.ly/UnzytRHōryū-ji: http://bit.ly/fbfOrF

“法隆寺 - Hōryū-ji”


https://www.google.com/search?q=%E6%B3%95%E9%9A%86%E5%AF%BA+wikipedia&aq=f&oq=%E6%B3%95%E9%9A%86%E5%AF%BA+wikipedia&sugexp=chrome,mod=14&sourceid=chrome&ie=UTF-8




Architecture is “POWER”


For whom?For what?

How?

Find!


Architecture

AvailabilityFault-tolerance, Automatically Recover

Split brain measures,I/O strategy,

Structure of the system layers

ScalabilityElastic storage-cluster

Able to join/leave nodes

PerformanceCache mechanism, I/O strategyStructure of the system layers

Need to # of servers

AdministrationManage and Monitor nodes in the cluster

Architecture

Use CaseFor the Web Applications

For the Cloud


HIGH Availability(Reliability)

HIGH Cost Performance HIGH Scalability

Architecture

To Realize


LeoFS-Manager

LeoFS-Gateway

LeoFS-Storage

REST over HTTP (80/443) RPC

(4369)

Request from Web Application(s) or Browser

META Object Store

Storage Engine/Router

META Object Store META Object Store

RPC (4369)

Storage Engine/Router Storage Engine/Router

Load Balancer

S3-API

Monitor(SNMP)

GUI Console

(4000,4010,4020)

(10020, 10021)

Architecture

Gateway (Stateless Proxy)

HTTP Request/Response Handling+

w/Object Cache

ManagerSystem Management

Ring MonitorNode State Monitor

StorageObject Storage, Meta data Storage

+Replicator/Recoverer, Qeueue


LeoFS-Manager

LeoFS-Gateway

LeoFS-Storage

REST over HTTP (80/443) RPC

(4369)

Request from Web Application(s) or Browser

META Object Store

Storage Engine/Router

META Object Store META Object Store

RPC (4369)

Storage Engine/Router Storage Engine/Router

Load Balancer

S3-API

Monitor(SNMP)

GUI Console

(4000,4010,4020)

(10020, 10021)

Architecture


Estimation / Review

Design

Simulation

Architecture

Basic Detail


GatewayStateless ProxyHTTP

Storage Cluster

Erlang RPC

Erlang RPC

Manager Cluster

State/Process MonitorErlang RPC

Object Cache

Architecture


LeoFS Architecture - Gateway / Storage

Storage Engine

Object Storage Metadata

StorageRPC RPC

membership (fault-detection)

redundant-manager replicator

queue

read-repairer

Gateway REST over HTTP (S3-API)

redundant-manager membership (fault-detection)

get put delete head

RPC


Cowboy: Erlang light-weight HTTP-Server - http://http://www.ninenines.eu/

Gateway

From ApplicationsS3-API

Object Cache

Replicate when using RPC

Consistent HashingHorizontal Distribution

Storage Nodes

[ LRU, Slab allocator, Skip graph ]“Cowboy”

“Stateless Proxy”

LeoFS Architecture - LeoFS Gateway


http://www.ninenines.eu/

http://www.ninenines.eu/

...

LeoFS Storage

leo-object-storage

LeoFS Architecture - LeoFS Storage Engine

Metadata : Keeps an in-memory index of all data.Object Storage : Log structured (append-only) object store.

Request From Gateway

replicatorrepairer

queue...


LeoFS Architecture - LeoFS Storage Engine - Data Structure

Offset Version Time-stamp{VNodeId, Key}

<Metadata>

Checksum

for Sync

KeySize CustomMeta Size File Size

for Retrieve an File (Object)

Footer (8B)

Checksum KeySize DataSize Offset Version Time-stamp

{VNodeId,Key} User-Meta Footer

<Object>

Header (Metadata - Fixed length) Body (Variable Length)

User-MetaSize

ActualFile

Supe

r-bl

ock

Obj

ect-1

Obj

ect-2

Obj

ect-3

<Object Container>

Obj

ect-4

Obj

ect-5


"Less is more" and "God is in the details"

Ludwig Mies van der Rohe: http://en.wikipedia.org/wiki/Ludwig_Mies_van_der_Rohe30Wednesday, November 14, 12

http://en.wikipedia.org/wiki/Ludwig_Mies_van_der_Rohe

http://en.wikipedia.org/wiki/Ludwig_Mies_van_der_Rohe

Strategy


“Gradual Development”


Estimation / Review

Implement / Fix / Improvement

Benchmark / TEST

Strategy

Incubation Production


Strategy

“SCRUM”

Scrum (development): http://en.wikipedia.org/wiki/Scrum_(development)34Wednesday, November 14, 12

http://en.wikipedia.org/wiki/Scrum_(development)

http://en.wikipedia.org/wiki/Scrum_(development)

Plan


Media Platform

Application / Log Collector

Search / Analysis

PaaS / IaaS

Plan

“Final image”

DATA-HUB


Media Platform

Application / Log Collector

Search / Analysis

PaaS / IaaS

Plan

“Final image”


Estimation

Implement / Fix / Improvement

Benchmark / TEST

Plan

Prototype1,2 Phase1 Phase2


ARIA (Photo Storage) released

LeoFS (Cloud Storage System) released

FROM “Photo Storage” TO “Cloud Storage”

Feb 2010 June 2011 July 2012 Nov 2012

Started R&D

a. Object Cache b. Compactionc. Rebalanced. S3 Compatibility

e. Large-object Supportf. Multi-tenantg. Multi-layer cache

Core Functions

Plan


5MB 100MB a few GB

From Photo Storage To Cloud Storage

1st step as Cloud StorageSpecialize in “Photo”

2011 - Phase1


5MB 100MB a few GB

From Photo Storage To Cloud Storage

Aim to “DATA-HUB” in the CloudHandle various unstructured data

2012 - Phase2


S3FS-C

Goal


Goal


4. To the Bloom


Fusion & Improvement

Amazon DynamoQuery Model, ACID

Facebook HaystackStorage Engine

SEDA: An Architecture for Well-Conditioned,

Scalable Internet Services

Hybrid = P2P + ManagerStorage Cluster / Manager

Split brain measures

Amazon S3 APIGateway, Manager

To the Bloom

SEDA: http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf

Amazon Dynamo: http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf

Facebook Haystack: http://www.facebook.com/note.php?note_id=76191543919


http://www.facebook.com/note.php?note_id=76191543919


http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf


http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf


Amazon DynamoQuery Model, ACID

Facebook HaystackStorage Engine

SEDA: An Architecture for Well-Conditioned,

Scalable Internet Services

Hybrid = P2P + ManagerStorage Cluster / Manager

Split brain measures

Amazon S3 APIGateway, Manager

To the Bloom

SEDA: http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf

Amazon Dynamo: http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf

Facebook Haystack: http://www.facebook.com/note.php?note_id=76191543919








“Connecting the dots”


Richard St. John's 8 secrets of success

Richard St. John's 8 secrets of success: http://www.ted.com/talks/richard_st_john_s_8_secrets_of_success.html


http://www.ted.com/talks/richard_st_john_s_8_secrets_of_success.html

http://www.ted.com/talks/richard_st_john_s_8_secrets_of_success.html

Wrap Up


Wrap Up

1. Awareness - Why? Why? Why?

2. Architecture is “POWER”

3. Gradual Development

4. Fusion and Improvement

5. 8 secrets of success


Thank you for your time

LeoFS - http://www.leofs.org


http://www.leofs.org

http://www.leofs.org

on the road - 情報処理学会fusion & improvement amazon dynamo query model, acid facebook...

Documents