Bare Metal Provisioning for Big Data Vol.01 　 2016/12/1Woosuk, ChoiEC Core Technology Department http://www.rakuten.co.jp/

2

At first, Simple Comparison Virtualization VS Bare Metal

3

What’s the best one for Big Data System ? • We’re used to need enough resource every time in order to

processing huge data in the Rakuten • In terms of performance/cost, winner is Bare Metal

Bare Metal Virtualization ( Cloud )

InfraManagement Difficult Easy

Performance Best Effort Bottleneck Storage/Network ..

Solutions Many legacy way .. AWS , OpenStack ..

Virtualization VS Bare Metal

4

H/W Selection Approach

Low Price

Micro Server

Local Storage

A Large number of servers

DAS

High Performance Storage

5

Quanta S910-X31E

Micro Server

FEATURE HIGHLIGHTS• Chassis

• Nodes : 9 nodes • Network - Inner Switch

• 1Set, No redundancy • 10Gbps x 2 uplink• 1Gbps x 24 internal

• Power • 2 redundant PSUs

• Per Node• Processor

• Xeon E3-1200 v3 (4C/1S/8T)• Memory

• DDR3 16GB/32GB

• Network • 1Gbps x 2 via Inner Switch

• Storage - No Raid • Controller : SATAIII• SSD : 2.5” 120GB/480GB• HDD : 2.5” 1TB

6

Quanta 210-X12RS/D51B-1U

1U type Server

FEATURE HIGHLIGHTS• Processor

• Xeon E5-2620/2640 v3 • 8C/1S/32T

• Memory • DDR3/4 128GB

• Network • 10Gbps x 1 , No redundancy

• Power • 1 PSU , No redundancy

• Storage • Raid Controller : MegaRAID • SSD : 2.5” 480/600GB x 4 • HDD : 2.5” 1TB x 2

7

Quanta 210-X22RQ/D51B-2U

2U type Server for Hadoop

FEATURE HIGHLIGHTS• Processor

• Xeon E5-2620/2640 v3• 8C/2S/32T

• Memory • DDR3/4 128GB/256GB

• Network • 10Gbps x 1 , No redundancy

• PSU• 1 PSU , No redundancy

• Storage • Controller : SATAIII ( no RAID )• SSD : 2.5” 120GB x 2• HDD : 3.5” 4/6/8TB x 12

8

Bare Metal, But is it really good for everything ? So, we can make Bare Metal management system

which is most likely to cloud solution

9

2014

2014 3Q2015 2Q

2015 4Q2016

Bare Metal System Project for Big Data

1st Revision Launch

2nd Phase Start

2nd Revision Launch

HistoryFor Next ….• More Global• Not only Big Data• Expand for others

• Scratch• By Admin• GUI/No API

• More Open Source• By User• Resource Management• GUI/API

10

US Data Center200 servers

Boston Office

EU Data Center300 Servers

Paris Office

JP Data Center5000 Physical Servers

India Operation Center24x7 Trouble Tier1 Support

US Data Center500 servers

Tokyo Office

Where are we ?

11

• Racking/Cabling• HW Check

• Stress Test• BIOS Configuration• Error on BMC

Acceptance

• Registration to OS Provisioning System

• Test / Building OS image for each type

• Base Chef recipes

Ready to Provisioning • Provisioning at once

• OS Installation• Application

Deployment• Set up monitoring

ProvisioningConnected System

Server Assign

Physical Server Delivery Flow

Bare Metal Management Facility

Infra Administrator ApplicationPlatform Admin

12

Rack = SW NAME : XXXX

3

3

1

1

Racking Rule – for Infra Hostname

Infra Hostname Rule

Region SW Host = Rack Name

Port Num = Position

Server Type

AJP XXXX 01 2U

JP-XXXX-01-2u

BJP XXXX 03 1U

JP-XXXX-03-1u

B

A

Region : JP

Mapping Rack position and Port number in Switch

OS provisioning engine can make infra hostname automatically

Rack Position

Port Number

13

OCP Server

DHCPTFTPSCP BDD Servers1.Get IP address and boot loader file name

2.Get boot loader file(pxelinux.0)

3.Get boot option configuration(pxelinux.cfg/default)

4.Start kernel5.Get boot image

7.Perform BDD OCP Scripts

8.Scp logs file

8.Analyze logs and get certificate result

Certificate the whole rack servers as new delivery

6.Start boot image

HW Check - Data Flow

14

No. Script Name Note

1 Check for initial delivery Collect HW information and check whether it is meet our expectation.

2 HW full stress test for DOA Check stability of HW

3 Initial FW / BIOS update Check BIOS version, downgrade / upgrade BIOS version when necessary

4 Initial standard BIOS configuration Mandatory configuration in BIOSEx) Dedicated network for BMC

5 Over provisioning for SSD Better write performance

6 Initial L2 Switch configuration Basic configuration on ToR / Inner Switch

HW Check - Function

15

Ready to Provisioning - Bare Metal Provisioning System

16

Role Mission Detail

Redundancy / Capacity Plaining

Self Provisioning

All provisioning including application deployment should be handled by users at once.

Full Automation Fire and Forget.

Pool/Resource Management

All user can control their own resource from pool. It can help your capacity planning as well as trouble shooting

Platform Quality

Full Stack Management

All system management, all user can do that by themselves

Easy Operation Support not only GUI but also API for easy maintenance

Fast Delivery Faster faster faster, like cloud

Mission

17

Installation Engine - MAAS Managem

ent Tool - Chef

OS provisioning system

App

OS

App

OS

GSP Hadoop

Bare Metal Management

App

OS

App

OS

Cluster for Redundancy

Pool

Platform Admin / Application Owner

Infra Admin

Self provisioningResource management

Developing and Maintenance

Concept

18

Component Detail

Provisioning ControllerThis was built up by Rakuten in order to control all provisioning process and data.It store own organization data in Rakuten as well as control MAAS and Chef , Power DNS, for all automation.

MAAS Metal as a Service from Canonical

Chef Management tool

PowerDNS Master DNS

BIND Slave DNS

Shinken Alerting system is compatible with Nagios.

Graphite Time Series DB

Components with Open Source

19

Self Provisioning with Full Automation

Dash Board

Organization

Role/Recipe

Host name

Custom data

Provisioning System

OS Provisioning with Chef

Easy Install

API

Worker

Controller

Installation Engine

Management

Monitoring

Operation System

Configuration

All Operation by Chef

App Deploy

Monitoring

Kick OS Provisioning for your system

With Recipes

Recipe for Application

Full Automation

Control Control

ConnectChef

MAASPower DNSShinken Graphite

Operation by each User

20

Pool

APP 1

Deployed

Pool

APP 2

Deployed

After Commissionin

g

Assignment

Provisioning

Destroy OK

Take over

OK

OK OK

OK

NG

OK

OK

Resource/Pool Management : Reservation

Infra Admin

Admin User 1 ADMIN

Authority

Authentication

Chef Organization

= Tenant APP 1 APP2

Admin User 1 User 2Only Admin

NGNGNG

OK

NG

User 2

Only possible by build user

21

APP 1

Deployed

APP 2

Deployed

Provisioning

Destroy OK

Take over

OK

OK OK

OK

NG

OK

OK

Admin User 1 ADMIN

Authority

Authentication

Chef Organization

= Tenant APP 1 APP2

Admin User 1 User 2

NGNGNG

OK

NG

User 2

Only possible by build user

After Commissioning

Shared Pool

Quotation for each group/platform

Self pick up from shared Pool

Resource/Pool Management : Shared Pool

22

• Designed by ApplicationApp Monitoring

• Designed by Application• Custom Package by Application App Deployment

• Custom Configuration on OS by Application• Server Account based on Chef OrganizationOS Configuration

• Default OS monitoring Infra Monitoring

• Default Configuration on OS• Basic Packages OS Configuration

• Simple image• Pattern base Partitioning / Raid configuration OS Installation

• Detail H/W Spec• Custom Information for BDDInventory Data

Full Stack Management

Role/Recipe

Infra Base

App XX Application/Platform Admin

Infra Admin

Responsibility Chef Organization

OS Provisioning

Criteria

App YY

Role/Recipe

App ZZ

MAAS

OS Images

23

Easy Provisioning

1st Step• Chose Server

2nd Step

• Chose Action • Install• Destroy

3rd Step

• Hostname what you want• Unnecessary DNS operation

• OS distribution/version • Tenant yours and environment• Chose recipes of your application

Final, click and get it

Hey, I want new server

Just Do It

24

InstallOS SetupOS SetupEnv

Provisioning Process and Fast Delivery

Provisioning System

Default Infra Role

App Role

Manage recipes for app

Default Infra Monitoring

App Monitoring

Basic Install DNS entry

OS / APP Configuration

MonitoringConfigurationTask

Worker

Approximately 30 min

Request via GUI/API

Operation System

Chef ChefMAASPowerDNS

25

Chef-Client

CollectedYum

Nrpe

Collector

Graphite

Graphite Web

TimeSeriseDB

Inventory DB

OCS Inventory

InventoryGLPI agent

Shiken

Thruk

Monitoring

Config files

Docker

PagerDutyAlert

Repository

Package

JIRA

Issue Tracking

Chef Enterprise

Configuration

DataBag

Account

Organization

CookbookCookbook

Cookbook

Elastic Search

Graylog2

Log Analysis

GIT

Source Management

Jenkins

Graylog Web

Graphana

Dash Board

Core Engine

OS Provisioning

MAAS

PowerDNS

DNS

MySQL

OS Install

Entry

Finally, You can get server with one click

OperationSupport

26

What is Next ? Is it done ? No…

27

Global Hosting for BCP

US 1 US 2 EU JP 1 JP 2

Zone 1

Zone 2

Zone 3

Global Zone

ProvSystem

Global Dash Board

ProvSystem

ProvSystem

ProvSystem

Prov System

Local System

Data Center

Local Zone

28

Platform : APP 1Environment : DEV

Management

API

Application

Application

HaProxy

Network Layer

Float IP

NAT

Private Network for User

Platform : APP 2Environment : PROD

.

.

Network ControlBare Metal

Management

XAnother Application

ACLBlock by ACL

Permit

Permit

Permit

Permit

XUser

API

Application

Application

Permit

29

Data Base

DB 1

DB XX

ApplicationLayer

.

.

Storage Layer

API

App 1

App 2

Internal SLB

HaProxy

AccessLayer

Access

Web 1

Web2

External SLB

HaProxy

Permit

Permit

ACL

Separated Network

30

Separating Functionality

Like OpenStack Concept • Current system only focus OS provisioning as one worker• Current system had been designed for only dedicated DC • Functionality requirement is getting increase

31

Installation Engine Managem

ent Tool - Chef OS provisioning system

App

OS

Bare Metal App

Bare Metal Management

App

OS

App

OS

Cluster for Redundancy

Pool

New feature for Cloud

Docker

OS OS OS

VM

Cloud

OpenStack/Sw

arm/M

esos ..

Global Dash Board

Cost Allocation

32

Thank you

33

We are hiring!

Let’s join us for Building the core part of Rakuten

Just mail to me, I can help you for your applicationWoosuk.choi@rakuten.com

http://global.rakuten.com/corp/careers/