openstack summit tokyo - know-how of challlenging deploy/operation ntt docomo's mail cloud...

2015/10/27 ＮＴＴ DATA Corporation

Know-how of Challlenging Deploy/Operation NTT DOCOMO's Mail

Cloud System Powered by OpenStack Swift

Abstract

Docomo mail is 24/7 cloud mail system which has accesses from over 20 million people. This mail system stores user's mail archive in OpenStack Swift with Peta Byte scale capacity deployed by NTT DATA. We have been successfully operating this service since Sep 2014 without any downtime. In this session, we'll present the actual issues and challenges we have faced and conquered.

Today’s contents and presenter

○Project Overview

Changes of Japanese mobile situation and abstraction of this project

– Project Manager : Sosuke Kakehi

○Migrate process

Process of migrating swift to existed docomo mail system

– OpenStack Swift Engineer : Masaaki Nakagawa

○Technical challenges

Swift technical challenges on this project

– OpenStack Engineer : Ryosei Kasai

○Operating session

Large scale swift operation

– OpenStack Swift Engineer : Masaaki Nakagawa

Project Overview

1 NTT Docomo's Cloud Mail System

2 Project Background

3 Customer Requirements

Cloud Mail System

NTT Docomo's Cloud Mail System - System Summary

• Docomo Mail - NTT Docomo’s Cloud Mail Service

• Over 20 million users

• Powered by OpenStack Swift

High Performance Storage

Object Storage OpenStack Swift

Later Mail

Tablet PC Smart Phone

Archived Mail

Stored to Swift

NTT Docomo's Cloud Mail System - System Scale

• Geographically Distributed Swift Cluster

• Over 6.4 Peta Byte Logical Capacity

• Over Hundreds of Servers

Proxy Node

Storage Node Region1

Project Background

Shift from “Feature phone” to “Smart phone”

Service

Smart Phone / Tablet PC

Service

Documents

Photos

Music Movie Application

E-mail Data Size was increased

Cost Cost

Cost Cost Cost

Project Background

High-end Storage

Extend the High-end Storage, extend, extend

= expensive cost, cost, cost

High-end Storage

Customer Requirements

High Availability

Low Cost

High Scalability

OSS(Software Storage) + IA Server

Disaster Recovery

Adopt OpenStack Swift

Migrate session

Overview of migration session

NTT DOCOMO has launched docomo mail service since Oct 2013, and swift was installed docomo mail system at Jan 2015. When we migrated swift to docomo mail system, docomo mail did not stop user service.

In this section, I would like to introduce overall of docomo mail system and migration process.

later older

Oct, 2013 docomo mail service in

Jan, 2015 Swift service in

May, 2014 test user start to use swift

Oct, 2015 General user start to test use Swift

swift (archived mail holder)

High speed block storage (later mail holder)

Swift migrate session System construction overview

Docomo mail frontend server (proxy of block storage and swift)

Storage Storage Storage

Internet

archived user mail

user mail user mail user mail

Swift migrate session Mail access flow

Docomo mail frontend server (proxy of block storage and swift)

Block Storage

Internet

archived user mail

access device

user mail user mail user mail

User mail will be archived/stored to swift

Swift migrate session System construction (before swift installed)

Docomo mail frontend server

Block Storage

Internet

archived user mail

user mail

Swift migrate session Migration 1st step – deploy swift and test

Block Storage

Internet

• Deploy swift • Trouble test • Tuning

archived user mail

user mail

Swift migrate session Migration 2nd step – copy test user’s archived mail

Block Storage

Internet

Copy test user’s archived mail

General user’s mail is not copied

archived user mail

user mail

Swift migrate session Migration 3rd step – copy general user’s archived mail

Block Storage

Internet

Move general user’s archived mail

keep all mail archive against swift trouble

archived user mail

user mail

Swift migrate session Migration 4th step – launch service

Block Storage

Internet

archived user mail

user mail

Conclusion of migrate session

• Firstly, docomo mail has only block storage

• We need to deploy and migrate swift with no down time

• To achieve it, we divide migrate to 4 steps

– Deploy

– Test user mail copy to swift

– General user mail copy to swift with remaining block storage

– System durability check

• We achieve no service down migration

As I said , in migrating, we achieve some technical challenges. Next session, Mr. Kasai introduce it.

Technical session

Our Technical Challenges

1 Durability assurance

2 Geographically distributed cluster

3 Quality

Challenge 1: Durability assurance

• Quality requirement in Japan

• This system needs very high quality.

• Everything should be under control

• System design for normal situation

• System design for defeat situation

Even on distributed system

• Analyze every behavior before building system

Recovery test in variety of defeat pattern

• Variety of failure pattern

(1) The point of failure • Disk, NIC, Process, Node, …

(2) The number of failures • 1, 2, 3, 4, …

(3) The range of failures • 1 node, multiple nodes/zones/regions, …

100s of test cases!!

Case #201

Zone1 Zone2

Region 1

Case #201

Zone1 Zone2

Region 1

Case #001

Case #101

Case #301

Case #501

Zone1 Zone2

Region 1

Result of recovery test

• Extreme durability and recoverability of swift

• Swift rarely loses data in it. Only accurate snipe or great disaster can causes data lost.

private network

Site 3

Storage

Site 4

Storage

Site 2

Storage

Challenge 2: Geographically distributed cluster

• Geographically distributed swift cluster to realize disaster recovery

• Important points to evaluate global distribution

1. Client request

2. Durability Site 1

Proxy 300km~ 300km~

300km~ 300km~

300km~

Pseudo-global cluster

• Pseudo-global cluster with simulated network latency

• Proxy and 3 Storage regions placed in different locations

• 10~200msec latency between locations simulated by tc

• TL msec latency for one way, 2*TL msec latency for round trip

Storage region 1

Storage region 2

Storage region 3

10~200msec latency

Client Proxy

Storage region1

TLmsec

2 points of Pseudo-global cluster testing

1. Client request

• Object PUT/GET/DELETE from client

• Error rate

• Turnaround time for 1 request

• Throughput

• Latency between proxy and storage

2. Durability

• Auto recovery by object-replicator

• Error rate

• Turnaround time of 1 sync process

• Throughput

• Latency between storages

Storage region 1

Storage region 2

Storage region 3

Storage region 1

Storage region 2

Storage region 3

Client

PUT GET

Client

Test1: Client request

Object PUT/GET/DELETE from client

• No error caused by latency

• Degradation of turnaround time

• No throughput degradation for concurrent requests

latency

limitation of network bandwidth

PUT/GET

DELETE

Latency concurrency

Throughput Turnaround time

Test2: Durability

Auto recovery by object-replicator

• No error caused by latency

• Performance degradation of one process

• No throughput degradation for concurrent process

Latency concurrency

Throughput

latency

limitation of network bandwidth

Defeat

Recovery

Performance

Challenge 3: Quality

1. Software Quality

• All processes work well ?

• Account / Container / Object

• server / replicator / updater / reaper

2. System Quality

• Our system is working well ?

• All nodes

• All APIs

Software quality

1 Add process name checking into swift-init

2 Prevent redundant commenting by drive-audit

3 Remove invalid connection checking in db_replicator

4 Add timestamp checking in AccountBroker.is_status_deleted

5 Fix error log of proxy-server when cache middleware is disabled

Source Code Analysis and Customize

• Official patch (below)

• Original patch

Strict test all processes

and more …

Our official patch

System quality

storage servers …

Tempest

proxy servers

checking tool

Test all nodes

• Automation testing tools for

1. APIs : All swift APIs, including error case

2. Nodes : All swift nodes

• Extended Tempest and checking tool

Test all APIs

Our solutions

1 Durability assurance

2 Geographically distributed cluster

3 Quality

Recovery test in variety of failure pattern

Performance test of frontend/backend with pseudo-global swift cluster

・Source Code Analysis and Customize ・Automated testing

Challenge Solutions

Operating session

Overview of operating session

Operation scheme of Docomo mail is high confidential.

We would like to introduce about NTT DATA swift solution's operation.

Docomo mail system uses NTT DATA swift solution with customizing.

Operating session Large scale system makes operation costly

Large scale Swift

scale out management repair tuning

Operating session Reduce operating work amount

Parallel access (pssh / pscp)

Automatic deploy (kickstart)

Tuning (svn / puppet)

Master repository

Operating session Reduce operation frequency

Disk failure Node down Server Process Down Backend process down ex)auditor process

Service affect

Operating session Stop monitoring which low priority

Periodic performance check

monitoring alert

Conclusion of operating session

• Swift is consisted by many nodes

• System operating costs of Swift tend to be costly

• NTT DATA has know-how to reduce swift operation cost

– Using operation parallelized tool

– Customizing for monitoring priority

– Change monitoring items to periodic check

Conclusion of this presentation

We introduce usage, challenge, and operating OpenStack swift at docomo mail service system

• System migration with no service down time

• Three technical achievement

• Reduce operating cost

Docomo mail has been service with no down time.

If you have something questions, please come to NTT booth.

○Attention All company names, product names, and service names mentioned are trademarks or registered trademarks of the respective companies

openstack summit tokyo - know-how of challlenging deploy/operation ntt docomo's mail cloud...

Technology

deploy tosca workloads in openstack via stand-alone...

openstack + vmware: deploy, upgrade, & operate a powerful...

how to deploy & operate openstack production

openstack install and deploy...

cloud foundry and openstack -...

kolla - the new way to deploy openstack · projects to...

from an openstack perspective stateless os · stateless...

cloud foundry elastic architecture and deploy based on...

nttコミュニケーションズのクラウドサービスとopenstack　クラウド...

deploy tosca network functions virtualization (nfv)...

openstack deployment and operations guide...

how to deploy openstack on tianhe-2 supercomputer · how to...

a ci/cd pipeline to deploy and maintain openstack -...

puppet: automate openstack deployment · ///// automate...

rom nttコミュニケーションズ · openstack...

things you must know before you deploy openstack

openstack lab environment · packstack •deploy of...

openstack deployment manual - bright...

andy mccrae, rackspace - using ansible to deploy and...

fits like lego: five ways to deploy guest networks over...