Rakuten’s Private Cloud

Jonathan Levine

http://www.rakuten.co.jp/

2

What’s Rakuten?

Rakuten Facts • Founded in 1997 by Hiroshi Mikitani

• Over 9,000 staff worldwide

• Rakuten Ichiba and Rakuten Travel are largest businesses;

we have a total of more than 50 services

• Domestically, more than 130 million products offered by

more than 130,000 providers

Corporate History - Domestic 1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

Corporate History – Global 1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

Domestic Marketplace Sales Growth

Source: Rakuten

*: Domestic E-Commerce = Mall (Fixed-price, Group purchase), Mobile, Business, Golf, Books, Auction (Flea market), download, tickets,

Internet Supermarket, Check Out, Off-track betting, toto, ShowTime, Media rental, Dining , Salon (added from Q2/12), and Mart and

Kenko.com (added from Q3/12)

International Marketplace Sales Growth

Source: Rakuten

*: Marketplace = Taiwan Rakuten Ichiba + Tarad Premium Mall + Lekutian + Buy.com marketplace + PriceMinister marketplace +

Rakuten Belanja Online + Rakuten Deutschland + Play.com marketplace + Rakuten Brasil marketplace (from Q2/12).

8

Why is Rakuten Ichiba Different?

Our Mission

Empowerment

10

Unique E-Commerce Model

Rakuten’s Empowerment Model is Unique and Difficult to Copy

11

Typical E-Commerce Site

12

Rakuten’s E-Commerce Site

13

Why Cloud?

My First Computer

Moore’s Law

Source: Wikipedia

My Current “Computer”

<

17

Server CPU Utilization, by Service

新サービス系サーバー別 　CPU稼働率(2009年3月)

0

10

20

30

40

50

60

70

80

90

100

wfb

tool

wpl

aza

mis

ml

wde

li

wenq

wm

oney

wdi

gi

win

cu

woubo

wis

eek

ddin

wnik

ki

btm

ail

bpla

za

wis

dir

bdig

i

wsp

ace

ddeli

wtm

ail

bfbt

ool

wap

itic

wps

p

wga

me

ww

om

an

apde

licti

wpo

rtal

gl

bnik

ki

wdi

giad

m

wm

ile

wde

liim

g

bmedi

a

asear

chr

wbr

tic

wpo

rtal

adm

mti

c

wga

meap

p

bauto

サーバー群

CPU使用率(%）

CPU使用率最大値

CPU使用率平均値

１０％

Domestic Marketplace Sales Growth

Source: Rakuten

19

Cloud at Rakuten

20

Step 1: Virtualization

Virtual Server Pool

Server Server

Shared Storage

Hard Disk

Storage Network

Old： 1 physical == 1server New：1 physical == n servers

System

Benefits of Virtualization

•Faster Deployment Time for New Servers •Automatically Handle Downtime •Improve Equipment Utilization •Easy Replacement of Hardware

OS

App

OS

App

OS

App

OS

App

OS

App Virtual Server Pool

21

“Pcloud” production Environment

Server Farm:

104 Physical Storage Farm:

14 Physical

iSCSI

22

Private Cloud Progress – Stage 1

VMs

Why so Slow?

Technology.

25

“RIaaS" production Environment

Fabric Network

(10G, FCoE)

3PAR Storage

Blade Server VMware

26

Private Cloud Management Tool

・VM management

・Server Setup (with Chef)

・DNS setting

・SLB control

Not Just Technology.

Not Just Technology.

It’s an Internal Business.

Source: Amazon re:Invent

• Tools

• Training

• Support

• Roadmap

• Communication

== Product Management

RIaaS is crossing over the Chasm!!!

We are here!

Early Adopter Early Majority Late Majority

Proof of solution Willing to use Step by Step

Test -> Production

Case study

Sensitive

Communication Word of mouth Interactive Passive

SLA No Risk No Return Reasonable requirement Demanding

Priority Speed

Experience

Balanced Risk/Return Stability

PCloud

RIaaS

32

Private Cloud Progress – Stage 2

Jul-1

0

Aug-1

0

Sep-1

0

Oct-

10

No

v-1

0

De

c-1

0

Jan-1

1

Feb

-11

Ma

r-1

1

Apr-

11

Ma

y-1

1

Jun-1

1

Jul-1

1

Aug-1

1

Sep-1

1

Oct-

11

No

v-1

1

De

c-1

1

Jan-1

2

Feb

-12

Ma

r-1

2

Apr-

12

Ma

y-1

2

Jun-1

2

Jul-1

2

Aug-1

2

Sep-1

2

Oct-

12

No

v-1

2

De

c-1

2

Jan-1

3

Feb

-13

Ma

r-1

3

VMs

33

Case Study

34

Exciting Event …?

35

20

05/1

/1

20

05/6

/1

20

05/1

1/1

20

06/4

/1

20

06/9

/1

20

07/2

/1

20

07/7

/1

20

07/1

2/1

20

08/5

/1

20

08/1

0/1

20

09/3

/1

20

09/8

/1

20

10/1

/1

20

10/6

/1

20

10/1

1/1

20

11/4

/1

20

11/9

/1

20

12/2

/1

20

12/7

/1

20

12/1

2/1

SuperSale Network Effect

36

Private Cloud for SuperSale

0

1000

2000

3000

4000

5000

6000

Jul-1

0

Aug-1

0

Sep-1

0

Oct-

10

No

v-1

0

De

c-1

0

Jan-1

1

Feb

-11

Ma

r-1

1

Apr-

11

Ma

y-1

1

Jun-1

1

Jul-1

1

Aug-1

1

Sep-1

1

Oct-

11

No

v-1

1

De

c-1

1

Jan-1

2

Feb

-12

Ma

r-1

2

Apr-

12

Ma

y-1

2

Jun-1

2

Jul-1

2

Aug-1

2

Sep-1

2

Oct-

12

No

v-1

2

De

c-1

2

Jan-1

3

Feb

-13

Ma

r-1

3

VMs

37

Next Steps

High Capacity Racks

PatchPatch

EqualLogic

R610

R610

21

543

876

11109

141312

17 R6101615 R610

2019 R61018

27 R610

2423 R610

25 R610

31 R61030

28

353433 R61032

41 Patch4039383736

21 R61022

26

29 R610

42 Patch

39 Nexus 2232PP

2U4N32

42 Nexus 2232PP41 Nexus 2232PP40 Nexus 2232PP

383736353433

312U4N30

292U4N28

272U4N26

252U4N24

232U4N22

212U4N20

192U4N18

172U4N16

152U4N14

132U4N12

52U4N4

112U4N10

92U4N8

32U4N2

1Management 48port switch

72U4N6

Standard Rack

(３KVA)

Hicap Rack

(15KVA)

48 Nodes

9 Cables

12 Nodes

36 Cables

Site Recovery Manager for Migration

Old DC New DC

40

Global DC Expansion

EU DC

Japan DC

U.S. DC

Private Cloud Roadmap

• IaaS (Compute as a Service) improvements

– Add APIs

– Add App team access

• DBaaS (Database as a Service)

• STaaS (Storage as a Service)

• PaaS (Platform as a Service)

42

Final Thoughts

43

•12 Employees

•27,000,000 users

•2 years

44

•31 Employees

•#16 site in US (as of May, 2012; source: Alexa)

•30 months

45

But…

• Technology is really important…

• … but it isn’t everything

• Cost is really important…

• … but it isn’t everything

• Make sure you can have enough

scale to be competitive…

• Then compete to win!

Thank you!