service provisioning and automation in the cloud · sixsq services ‣consultancy • agile...

Copyright © 2012 SixSq Sàrl. All rights reserved.

Service Provisioning and Automation in the CloudA practical usage of cloud technology to rationalise and automate software testing, certification and deployment

Marc-Elian BéginSixSq

SixSq Sàrl

‣Founded in 2007 by three partners‣Based in Geneva,

Switzerland‣Privately held, limited

liability company

‣sixsq.com

2

SixSq Services

‣ Consultancy• Agile software development processes and practices• Automation of those processes and practices• Cloud and virtualization technologies

‣ Software development• Distributed systems• Cloud-based domain specific vertical solutions

‣ Training and coaching• Agile and Scrum training• Coaching service to put theory into practice

3

SixSq Products

‣SlipStream• Automatic creation of new virtual machines• Automatic multi-machine deployment and

orchestration of services• Runs on public or private cloud infrastructure

‣StratusLab - European collaboration• Complete IaaS Cloud distribution• Open source

4

Collaborations and Partnerships

‣Building ‘Cloud Read Pack’ hardware/software private cloud solution with IBM and Darest

‣Amazon Independent Software Vendor

5

A"European"cloud"compu/ng"partnership:""big"science"teams"up"with"big"business"

Strategic)Plan)

  Establish"mul/8tenant,"mul/8provider"cloud"infrastructure"

  Iden/fy"and"adopt"policies"for"trust,"security"and"privacy"

  Create"governance"structure"

  Define"funding"schemes"

To"support"the"compu/ng"capacity"needs"for"the"ATLAS"

experiment"

SeEng"up"a"new"service"to"simplify"analysis"of"large"

genomes,"for"a"deeper"insight"into"evolu/on"

and"biodiversity"

To"create"an"Earth"Observa/on"plaGorm,"

focusing"on"earthquake"and"volcano"research"

Main Customers

‣Atos‣European Broadcast Union (Eurovision)‣ESA/ESTEC‣ESA/ESOC‣And more... I can’t talk about it yet, but it’s exciting!

7

Copyright © 2012 by SixSq Sàrl. All rights reserved.

My Cloud Definition

My Cloud Definition...

Virtualisation + Simple API (REST is best :-)

‣Yes!... this implies a IaaS foundation

‣What’s not in my definition:• Public/Private? Depends who takes care of elasticity• How you pay for the resources

9


Why we need automation?

Silo Development

11

Silo Development

11

Dev

Silo Development

11

Dev Test

Silo Development

11

Dev Test QA

Silo Development

11

Dev Test QA Ops

Silo Development

11

Dev Test QA OpsDuplication

Duplication

Duplication

Silo Development

11

Dev Test QA Ops

Silo Development

11

Dev Test QA Ops

Slow

Slow

Slow

Silo Development

11

Dev Test QA Ops

On-Demand Provisioning

12


12

Dev

TestQA

Ops


12

Dev

TestQA

Ops

My own

full system

now


12

Dev

TestQA

Ops


12

Dev

TestQA

Ops

Share

Share

Share


12

Dev

TestQA

Ops

Continuous Integration

13

<parameterized>

Agile Methods Footprint

14

User


14

User Dev


14

User Dev Ops


14

User Dev Ops


14

Scrum

User Dev Ops


14

Scrum

eXtreme Programming

User Dev Ops


14

Scrum

eXtreme Programming

DevOps


Case Study 1: Mission Control System

ESA ESOC

European Space Operations Centre, Darmstadt, Germany

16

Context and Problems

•Mission Control System deployment

•Deployment is manual•Time consuming (multiple days...)•Error prone resulting in configuration inconsistencies•Poor reproducibility

17

Relevance of this Case Study

•Complex legacy system•C++, Java, Perl, Python, Tcsh, CORBA (yes!)•Millions of lines of code, grown over several years

•Not designed for automation•Great benefit from automation, if possible!

18

Vision

21

•Automated on-demand deployments•Consistent and reliable deployments •Dramatic effort/cost reduction in deployment/testing

SLES11

Image References

22

SLES11

Store

SLES11

Image References

22

SLES11

Store

COTS

SLES11

Image References

22

SLES11

Store

COTS

COTSConfiguration

SLES11

Image References

22

Store

COTSSLES11

SLES11

Image References

22

Store

COTSSLES11COTS

SLES11

SLES11

Image References

22

StoreCOTS

SLES11

COTSSLES11

SLES11

Image References

22

StoreCOTS

SLES11

S2K S/W

COTSSLES11

SLES11

Image References

22

StoreCOTS

SLES11

S2K S/W

S2KConfiguration

COTSSLES11

SLES11

Image References

22

Store

COTSSLES11

S2KCOTS

SLES11

Deployment Scenario

23

Deployment Scenario

23

Orchestrator

Deployment Scenario

23

Orchestrator S2K Prime

Deployment Scenario

23

Orchestrator S2K Prime

S2K Backup

Deployment Scenario

23

Orchestrator

S2K Client

S2K Prime

S2K Backup

Deployment Scenario

23

S2K Client

S2K Prime

S2K Backup

Deployment Scenario

23

S2K Client

S2K Prime

Hostname

S2K Backup

Deployment Scenario

23

S2K Client

S2K Prime

Hostname

S2K Backup

Hostname

Deployment Scenario

23

S2K Client

S2K Prime

Hostname

S2K Backup

HostnameHostname

Deployment Scenario

23

S2K Client

S2K Prime

Hostname

S2K BackupReady

HostnameHostname

Solution Overview

24

Applications + deployment recipes

VM factoryDeployment engine

e.g. S2K

Cloud (IaaS)

Cloud Infrastructure

•Infrastructure as a Service (IaaS) provides:•Virtual machine life-cycle•VM image store•Storage

•StratusLab used for IaaS cloud:•License: Apache 2.0•European

25

S2K configuration in SlipStream

26

SlipStream dashboard and reports

27

7 minutes later...

28

Next steps?

•Execute Entire Verification Suite (e.g. ART)•Tighter integration with build process

•Integrate with CI solution (e.g. Hudson, Jenkins)•Generate binary packages to simplify installation

29


30

<s2k-build-version><s2k-config-profile>


30


b3427 pMSG-ESOC-sim


30


b3427 pMSG-ESOC-sim

b3434 pMSG-Eumetsat-ops


30


b3427 pMSG-ESOC-sim


b3452 pMy-Dev-deploy


30


b3427 pMSG-ESOC-sim


b3452 pMy-Dev-deploy

b3467 pMSG-auto-cert

Potential Deployments - ESOC Internal

31

ESOC

SlipStream Cloud

PackagePackagePackagePackage

Potential Deployments - Shared Dev

32

ESOC

SlipStream Cloud

Contractor A

CloudPackagePackagePackagePackage

Contractor B

CloudPackagePackagePackagePackage

StratusLabMarketplace


Case Study 2: Multi-Media News Exchange Platform

Goal

•Create news exchange platform, between most European (and others) broadcasters (> 70 sites)

•In time, replace satellites•No single point failure

34

Production Deployment

35

Paris London

GenevaFrankfurt

Geneva 2

BroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcaster

Technologies

•RESTful web services•Clojure, Noir web framework•Stateless

•Persistence: Cassandra/Datastax

36

Challenges

•Need to certify full system before large deployment•>80 sites involved all together!!•Distributed development teams (4 sites)

37

Development Deployment

38

Client All-in-oneServer

Site Deployment

39

Client

Cassandra1

RESTfulWS

Cassandra2

ClientClientClientClientClient

Production Deployment

40

Paris London

GenevaFrankfurt

Geneva 2

BroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcasterBroadcaster

Solution Overview

41

Applications + deployment recipes

VM factoryDeployment engine

e.g. TtF WS, Cass.

Cloud (IaaS)


Case Study 3: Cluster Provisioning

Helix Nebula: ESA/SSEP

43

•SSEP•Processing based on gLite based cluster (1 CE + n WN)•Large satellite image data set made available•Users can self-provision their own cluster

Helix Nebula: ESA/SSEP

44

•SSEP...•Processing based on gLite based cluster (1 CE + n WN)•Large satellite image data set made available•Users can self-provision their own cluster

Automated Deployment

45

•30 WNs deployment•124 deployed virtual cores•248 GB RAM•Provisioned in 2 minutes, from a single action!!


SlipStreamImage Factory and System Provisioning

Who’s deploying SlipStream?

47

• Academic production systems• CNRS - LAL - France• GRNET - Greece

• Commercial production systems• SixSq - Amazon EC2• Atos - Eindhoven - Holland (Helix Nebula PoC environment

Automated GPOD deployment• European Broadcast Union (Eurovision) - in progress

• Entire Transition to File server solution• Adding support for HP OpenStack Cloud Service• Certifying for IBM PureSystems

Main Features

48

•Virtual Machine Factory•Automatic creation of new virtual machines•Based on existing public images or custom built

images•Deployment

•Multi-tier machine topology modeling•Automatic deployment and orchestration of

services

•On-demand provisioning of systems

SlipStream

49

Main Features

50

•Virtual Machine Factory•Automatic creation of new virtual machines•Based on existing public images or custom built

images•Deployment

•Multi-machine deployment•Automatic deployment and orchestration of

services•On-demand provisioning of systems

SlipStream Cloud support

51


StratusLabSimple yet powerful IaaS cloud solution

Who’s deploying StratusLab?

53

•Academic production systems•CNRS (2 installations) - LAL and IBCP - France•GRNET - Greece

•Commercial production systems•Atos - Eindhoven - Holland (Helix Nebula PoC

environment with ~500 cores)•More (confidential)

•Many more private installations

Who’s behind StratusLab?

54

CNRS (FR) UCM (ES)

GRNET (GR) SIXSQ (CH)

TID (ES) TCD (IE)

Key features

• Rich contextualisation• Integration with shared/federated Marketplace• Supports copy on write and thin provisioning for fast

instantiation• Dynamic creation of disks (volumes)• Storage backend: iSCSI, Fiber Channel, Shared File System

(e.g. NFS)• Rich command-line client for automation integration• REST API (except for Open Nebula VM management with

XML-RPC)

55

Infrastructure as a Service basics

56

User

Cloud site

NetworkStorage Compute

API

Client

ClientClientClientVM

1: <init>

Network

2: <create>

3: <ssh>

User

Cloud site

NetworkStorage / Cache Compute

API

Client

ClientClientClientVM

1: <init>

Network

4: <create>

5: <ssh>

Image Marketplace0: <shop>

2: <discover>2': <retreive>

3: <validate>

IaaS Federation with StratusLab

57

Deployment Models

•Federation of Marketplaces for flexible image management

•Public cloud deployment (e.g. Atos and CNRS)•Private federated cloud deployment (e.g. EBU)•Caching of images across cloud installations

58

SixSq commitment

Maintain and further develop•StratusLab Marketplace•StratusLab Storage Service•Integration with OpenNebula

59


Conclusions

Conclusions

•Automation is paramount to succeed in building (distributed) systems

•Cloud is a key ingredient, not only virtualisation•Need a management layer and a share platform •Try... learn... have fun!!

61

Try it!!

62

slipstream.sixsq.com

slipstream.stratuslab.eu

Give us feedback...

Thanks :-)sixsq.com

[email protected]

service provisioning and automation in the cloud · sixsq services ‣consultancy • agile...

Documents