baidu cloudfoundry english

A private cloud platform based on CloudFoundry TRANSLATED VERSION

@Weiyu Wang(王炜煜)，Operations Department @Baidu weibo.com/wwy1640 2013-7-19

Outline �

Background and Objectives

Practice and Reform（Part 1、2）

Processes and Standard

Reform operations

Future plans TRANSLATED VERSION

1. Background and Objectives

TRANSLATED VERSION

Operation and PaaS �

Storage

Servers

Networking

O/S

Middleware

Virtualization

Data

Applications

Runtime

OP(SRE)，operation

PaaS (and IaaS)

TRANSLATED VERSION

Objectives �Automation

Business life cycle management，for example, modification 、monitor、fault handling and so on.

Resource utilization is elastic.

Standardization Flow

Instance standard

System environment、runtime、framework

Unification Integrate the third-party service，for example DB、Cache、log、FS and so on.

Linkage with other system platform

TRANSLATED VERSION

Why CloudFoundry ？

Automation

Standard Unification

Machine Management （The downstream department）

Automation

Standard Unification

TRANSLATED VERSION

Why CF ？ �

Automation

Unification

Standard

TRANSLATED VERSION

2. Practice and Reform（Part1） Java，base on cf 1.0

TRANSLATED VERSION

Java Apps �

•  Number of Product Categories >100

•  APP >200

•  Instances>2000

•  Average single-instance 10G（Memory）

•  Average Daily total pv > 1billion

•  The numbers of developers and testers for APP > 700

•  Tomcat5/6/7、jdk1.5/1.6、Standalone

TRANSLATED VERSION

Implementation and Preparation �

•  Relevant modification based on CentOS ü Deploy each CF component independently

⁺  Analyze BOSH、chef，implementation based on physical machine

ü OS environment initialization

⁺  apt-get is changed to yum

ü Ubuntu-cmd to CentOS

⁺  DEA(v1.0)，agent.rb、secure.rb

yum install -y make gcc gcc-c++ kernel-devel.x86_64 openssl-devel.x86_64 libxml2.x86_64 libxml2-devel.x86_64 libxslt.x86_64 libxslt-devel.x86_64 git.x86_64 sqlite.x86_64 ruby-sqlite3.x86_64 sqlite-devel.x86_64 unzip.x86_64 zip.x86_64 ruby-devel.x86_64 ruby-mysql.x86_64 mysql-devel.x86_64 curl-devel.x86_64 postgresql-libs.x86_64 postgresql-devel.x86_64 zlib-devel.x86_64 readline-devel.x86_64 ImageMagick.x86_64 ImageMagick-devel.x86_64 php-magickwand.x86_64

TRANSLATED VERSION

Cluster capacity assessment �•  Number of instances，NATS capacity assessment

ü  Number of instances hosted by single DEA(<100），the pressure to NATS-Server has little

effect

ü  Single NATS-Server can host 330 DEAs by a conservative estimate，The number of single

instance is 5~30.

ü  Multiple NATS-Server，extendable

Deplay （ms）

Number of DEAs （10 ~ 340）

Number of Single DEA instances（5 ~ 30）

Critical line 330 DEAs

TRANSLATED VERSION

In cluster, component redundant, LB design �

•  NATS ü Cluster，multiple NATS, synchronous heartbeat ü Cache information from client side. If network is cut down，it

should keep to reconnect. ü Multiple NATS does load balance（Client > 0.5.beta.6）

NATS-Server1 NATS-Server2

NATS-Client (caching message)

NATS-Server1/2, Random list

TRANSLATED VERSION

Multiple cluster redundant design �•  Multiple independent cluster ，logic independent

ü  The first layer’s switch，modify DNS A record，for multiple domain names(CNAME to this A record), they will uniformly switch to to different clusters

ü  The second layer’s switch，modify “interface layer”（For its application layer’s function ，it can be simply understood as Nginx’s reverse proxy ）

ü  Ensure App (stateless) capacity，or expand the capacity quickly to prevent overload when the traffic switch back

Baidu GateWay Front End

Router

A记录

Baidu GateWay Front End

Router

app1 app1

CNAME（formal domain name）

CNAME（formal domain name）

www.baidu.com CNAME www.a.shifen.com. www.baidu.cn CNAME www.a.shifen.com. www.a.shifen.com. A 119.75.218.77 www.a.shifen.com. A 119.75.217.56

TRANSLATED VERSION

Core components, distributed �

Router_1

NATS_1

Router

NATS CC HM

Stager

DEA

PG_DB Redis

TRANSLATED VERSION

Framework（cf1.0） �

DEA

Logging Name Service Monitoring

jvm

Stager

File Persistence

HM

Router

CC

Baidu GateWay / Front End

jvm jvm

API Bridge

UAA

jvm

jvm jvm jvm jvm

Router（Cluster 02）

N A T S

DB

TRANSLATED VERSION

New features �•  Support RPC, Single instance with multiple

ports ü  One instance will open multiple ports，and provide API to search the

IP ,ports in real time

ü  Linkage with “name service”，synchronize dynamic IP/port’s

relationship with name.

ü  RPC caller will connect the instance directly according to name

TRANSLATED VERSION

DEA server

Support RPC、 Single instance with multiple ports �

Instance01:port

Instance02:port

API Bridge

NS server

TXT record ip:port ip:port

RPC caller

NS client

Domain ip:port ip:port

ip_local_port_range

10000 ~ 60000

Port pool（There is freeze

period after allocation）

61000 ~ 65000

TRANSLATED VERSION

New features �

•  Support JMX ü  API to search the IP and Jconsole port in real time, then implement to

collect JMX data in real time.

TRANSLATED VERSION

DEA

Support JMX �

Instance01: Jconsole 端口

Instance02: Jconsole 端口

{ "instances": [ { "index": 0, "state": "RUNNING", "since": 438249600, "jconsole_ip": "10.1.1.1", "jconsole_port": 61111 }, { "index": 1, "state": "RUNNING", "since": 438249600, "jconsole_ip": "10.1.1.1", "jconsole_port": 62222 }

Monitoring Metrics

CpuUseRateDaemonThreadCount MemPool_OldGen_UseRate

NonHeapMemoryUsage_used TotalCompilationTime

TotalPeakThreadCount TotalStartedThreadCount

UnloadedClassCount GC_Major_Frequency GC_Major_Time

… …

Stager： java \ -Dcom.sun.management.jmxremote.port={VCAP_JCONSOLE_PORT} -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

TRANSLATED VERSION

New features �

•  Enhancement to health monitor ü  Seven layers’ detection

ü  Number of file handler detection

TRANSLATED VERSION

DEA Server DEA agent.rb

Health Manger

instance

http avaliability

instance

CPU MEM DISK ……

report

Enhancement to health monitor �

handler

TRANSLATED VERSION

DEA(v1.0), logical enhancement �•  Ports Management

ü  Description ⁺  Single DEA, multiple instance，parallel to assign and start the port，there is no

critical line，but there is the port competition issue

ü  Solution ⁺  Reference DEA(v2.0)’s logic（Notes: it’s DEA_NG, not compatible with CF1.0）

⁺  Define ip_local_port_range as 10000~61000，it is dynamic ports’ range

⁺  Make 61001~65000 as DEA scheduling assigned ports

⁺  For assigned port，add “[release time、port num]” data structure

⁺  It resolve the port competition by delaying to release the port

ü  Note ⁺  CF2.0 has resolved this problem by the same method above.

TRANSLATED VERSION

DEA(v1.0),logical enhancement �

•  Instance resource information management ü Description

⁺  Du command takes long time to calculate the disk space, as a result, the

following commands’ calculation is not consistent

⁺  When calculate the CPU utilization, it doesn’t consider the number of cores

ü  Solution

⁺  Adjust the related command’s order

⁺  When calculate the CPU utilization, it should be divided by the number of

cores

ü Notes ⁺  CF2.0 has resolved this problem.

TRANSLATED VERSION

New features (Linkage with peripheral system） �

•  File persistent ü  Use MFS（Moose File System）

ü  DEA deply MFS-Client and mount /mfs/path to let instance use

ü  MFS service provide the HTTP interface to get the data

•  Route based on URL，distinguish APP ü  foo.baidu.com/app1 à app1.foo.baidu.com

ü  foo.baidu.com/app2 à app2.foo.baidu.com

•  Monitor linkage ü  APP’s life cycle，to interact with external monitor system’s API, to implement the

monitor item’s automatic modification.

•  The SDK ü  Automatic release（encapsulate vmc）

ü  View file

TRANSLATED VERSION

Summary of key reform point(CF V1.0) �•  Relevant reform based on CentOS

•  NATS-Cluster usage、NATS-Client retry and cache

•  Support RPC、single instance with multiple ports

•  Support dynamic JMX、Jconsole

•  Enhance the health monitor

•  Ports management

•  Instance resource information management

•  Peripheral component：File persistent、Monitor linkage、URI Route、The SDK

TRANSLATED VERSION

2. Practice and Reform（Part2） C/C++，base on cf 2.0

TRANSLATED VERSION

Several key problems of C/C++ Apps �•  Container’s runtime is isolated with resource

ü  Kernel/GNU

ü  Resource isolation

ü  Snapshot，Core Dump

•  Single instance, multiple processes ü  Health monitor

ü  The order of processes’ execution

ü  Communication within instance and among process

ü  Multiple ports

ü  The isomorphism of multiple instances

TRANSLATED VERSION

Several key problems of C/C++ Apps �•  Big instance

ü  Big instance number（100 thousands）

ü  Large amount of data（single instance，2TB）

ü  High memory usage（single instance，100G）

ü  Long start time（30mins）

ü  Large flow（single instance，daily total PV2 hundred million）

ü  When drift，to prevent insufficient resources

•  APP communication ü  Network layer communication，authorization、flow control

ü  Output file，need to get from outside

ü  Input file，need to push from outside

ü  RPC，none-HTTP protocol，not containing PATH info，can’t route

TRANSLATED VERSION

Instance’s OS-Level environment preparation �•  Container’s runtime environment

ü Kernel is consistent with host machine

ü Make Container’s file environment

warden/warden/root/linux/rootfs/setup.sh

if grep -q -i centos /etc/issue then exec $(dirname $0)/centos.sh $@ fi

TRANSLATED VERSION

Relationship between Container and host machine �

Warden

Networking，Bridge / NAT / Firewall / FlowControl

DEA

init─┬─xxx ├─xxx─xxx ├─xxx

mount r usr/ lib/ etc/ mount rw xxx/

network interface（sub net）

Cgroup – CPU / MEM

Name space init─┬─xxx ├─xxx─xxx ├─xxx

mount r usr/ lib/ etc/ mount rw xxx/

network interface（sub net）

Cgroup – CPU / MEM

Name space

TRANSLATED VERSION

Package management �•  Buildpack API

ü  detect ， check

ü  Compile，environment preparation

⁺  Directory structure

⁺  Program files，and relevant supporting program

⁺  Startup script, and ensure the startup order of process …

⁺  Monitor script，it can periodically execute and check the whole instance’s health

ü  Release，information to publish

ü  Procfile，parameter passing（e.g. port）

ü  .profile.d，environment variable

TRANSLATED VERSION

Point to enhance health monitor �•  Self-defined monitor scripts

ü  self-defined monitor scripts, which is published together with instance and periodically to modify the content of stat_file

ü  DEA will check the stat_file periodically

Instance

stat_file

monitor.sh

process-1

process-2

DEA

HM

TRANSLATED VERSION

Reform to APP �•  For RPC，support NS Client

ü  Dynamic configuration file to replace route

ü  Port management，freeze time

•  Input/Output file ü  Input file need to get from outside actively

ü  Output file，pushed to the transit（e.g. cloud storage ），or service based on NS

•  Multiple process management, startup scripts ü  Multiple processes，to control their startup order

ü  Process control

•  File persistent ü  Remote log

ü  Use the cloud storage

TRANSLATED VERSION

Framework（CF2.0） �

DEA

Logging Name Service Monitoring File

Persistence

HM

gorouter（RPC，not applicable）

CC

Baidu GateWay / Front End

API Bridge

UAA

（Cluster 02）

N A T S

Container

process-1

process-2

Warden

NS Client

Container

process-1

process-2

Container

process-1

process-2 DB

TRANSLATED VERSION

Reform Summary（cf v2.0） �•  Relevant reform based on the CentOS

•  Container’s environment order

•  Buildpack’s order

•  Support RPC, single instance, multiple ports

•  Enhance the health monitor

•  Peripheral: file persistent, monitor linkage, URI Route, SDK

TRANSLATED VERSION

3. Processes and Standard

TRANSLATED VERSION

Working Process Description �

Review •  Standard •  Capacity •  SLA

Access •  Org

relationship •  Name info •  Operation info

Process approval •  Authorizatio

n apply •  Name apply •  Release opt

Release update •  PreRelea

se •  Gray

scale •  Rollback

Failure handling •  availabilit

y •  Security •  Issue

mgmt

TRANSLATED VERSION

Standard and Capacity Example �

•  Standard information collection ü  App related name, related interface people（R&D, QA, operation,

related manager, and so on）

ü  Runtime is isolated with container’s version

ü  Stateless, RPC, URI Route

ü  Dynamic and static files are isolated

ü  File persistence

•  Capacity information collection ü  PV、QPS

ü  Single instance’s CPU, memory, disk, bandwidth, restarting time

ü  Number of instances

TRANSLATED VERSION

SLA examples �•  Service object

ü  Java Application（“APP” for short in the following） ü  APP that conforms to the standard

•  Servicing time ü  24×365 all year round

•  Way to communication ü  Mail、Tel、interface people information

•  Stability related indicators ü  Core components，availability >99.99%（by month），MTTR<20mins，

MTBF>5days ü  Control services，availability >99.95%（the whole year） ü  APP’s self SLA, it won’t cause bad effect because of platform its self. ü  Notes：APP’s self problem，beyond the scope of SLA，for example,

bug, capacity forest error, external system’s failure(e.g. DB, Cache) and so on

TRANSLATED VERSION

Organization, Layer �•  Product line(Org)

• Module(Space)

• Group（APP）

•  Version (APP-*）

Product line -2

Product line-1 （Org）

Module-2

Module-1 （Space）

Group-1（A）

Group-2（B）

实例，版本-1 （APP-1-1）




Instance，v1 （A-1）

Instance，V2 （A-2）

Instance，v1 （B-1）

Instance，V2 （B-2）

It is one APP，but multiple

instances in the dashed frame.

TRANSLATED VERSION

Further encapsulation to CC �

Product line(Org) OrgName

Module(Space) OrgName_SpaceName

Module group OrgName_SpaceName_GroupTag

Module version OrgName_SpaceName_GroupTag_VersionTag

Instance（Unique id） OrgName_SpaceName_GroupTag_VersionTag_Index

TRANSLATED VERSION

GroupTag、VersionTag �• GroupTag

•  It can distinguish: configuration number、computer room、rack … from different dimension

•  Version Tag •  It can distinguish：program, data, configuration file and so on

•  Including: four version number, timestamp

•  Instance full name，for example

•  Org_Space_GroupA_1-1-1-1-438249600_1

•  Org_Space_GroupB_1-1-1-1-438249600_1

TRANSLATED VERSION

Examination, approval and release �

•  Distribute form and approve ü  APP information（program version, capacity information, related

instruction and so on）

ü  Approval（related manager, and the people who should know）

ü  Operator、Operating time

ü  Monitor information（Monitoring and controlling strategy、Interface people and so on）

•  Start to distribute operation, and add

monitor ü  Before release，related approval processes must pass

ü  Operator, program version, MD5、time information and so on，it

must keep consistent with approval

ü  It must be consistent and pass the processes，then it can

release

ü  After successful release, add the monitor

Distribute form

Approval

Release APP

Add Monitor

TRANSLATED VERSION

Pre-release, release, rollback �

app_v1 instance01 app_v1.paas.baidu.com

app_v1 instance02

app_v2 instance01

app_v2 instance02

app_v3 instance01


app.baidu.com

Generic domain name, map/unmap, multiple versions of app

Ahead, Release

Retreat, roll back

Pre-release，Offline observation in inner network

TRANSLATED VERSION

Basic grays scale release �


app_v1 instance02

app_v2 instance01

app_v2 instance02

app_v3 instance01

app_v3 instance02

app.baidu.com

1、Make one formal domain name point to multiple apps at the same time 2、Adjust the proportion of many instances’ number，then adjust the proportion of traffic.

app.baidu.com

app_v2 instance03

By adjusting the proportion of the many instance’s number, to adjust the proportion of gray scale traffic

TRANSLATED VERSION

“The path to sermon”, The platform popularization �•  The medal, who own the other half ?

ü  Support app

⁺  New service needs to follow the PaaS related standard and thought

⁺  Old service，need R&D to reform and QA to do regression test

ü  Periphery support ⁺  DB, Cache, storage, interface, security, monitor and so on

•  Clear the benefits，establish the win-win ecosystem ü  Deliver faster, save more resource, and make it more simple

ü  One-stop and all-in-all service，hand in hand to popularize

TRANSLATED VERSION

Some solutions: �•  Give users（APP developers) noble imperial

enjoyment ü  For important APP，do some specific service

ü  For important managers，it should have a set of complete, timely communication, such

as reports, etc

ü  The principle is “capitalism”, rather than “socialism”

•  Event “marketing” ü  E.g. “struts2 0day”

⁺  Actively cooperate with R&D and QA to do the issues identification, repair and

implementation

⁺  Actively report the progress and do the event managment

⁺  Late，for this to actively promote and participate the discussion and make decision,

for example, security, and architecture group

⁺  The principle is “win-win”，rather than shirking the responsibility

TRANSLATED VERSION

4. Reform Operation

TRANSLATED VERSION

Reform operation �

“NoOps” PaaS(and IaaS) overall functionality >= Traditional operation work

Storage

Servers

Networking

O/S

Middleware

Virtualization

Data

Applications

Runtime

OP(SRE)，operation

PaaS (and IaaS)

TRANSLATED VERSION

How to reform，Example �• Automatic fault recovery

ü  Add the health monitor mechanism based on the

traditional monitoring

ü  Instance automatically restart and “drift”

ü  Reduce the traditional alarm and man power

⁺  It will only alarm, when automatic recovery fail

Monitor

Whole instance name_1 ip:port

… …

Health

monitor

API … …

Real instance_1 ip:port

Instance after drifting_1

•  ”drift” is a normal phenomenon, it doesn’t alarm

•  It only need the alarm, when “drift” fail

•  It refinins to monitor instance，every time according to

name，detect and return ip:port

TRANSLATED VERSION

How to reform, Example �• More agile

ü  Make developer forget the servers, instead of resource oriented

ü  It has a complete configuration management and automatic deployment

function

ü  Release, pre-release, rollback, extremely simple, and it doesn’t need the

extra complex deployment tool

ü  Elastic extension, extremely simple

ü  Use Buildpack，implement cloud compiling and run directly

•  Experience of all in one and one-stop ü  From distribute form, release and modify the monitor，the working process is

totally automatic

ü  Integrate the third-party service, unify the management entrance

TRANSLATED VERSION

5. Future plans

TRANSLATED VERSION

Future plans �• Feedback to community

•  For private cloud function，try best to encapsulate the native components(based

CF2.0) , then make the new component open source

•  If affect the native components，try best to merge to the master branch

•  Write more document and tips, and actively to participate in communication

• Development orientation •  For large application（big instance）related

•  Intelligent scheduling related

•  Information Security

•  Further continuous integration

•  UI

TRANSLATED VERSION

We are hiring !

@Weiyu Wang(王炜煜) weibo.com/wwy1640

Thanks

TRANSLATED VERSION

baidu cloudfoundry english

Technology