integration and automation @ mobile.de (ebay tech talk)

Integration  Systems  @  mobile.deKaren  Almog  Sr.  Web  Operations  Engineer,  Mobile  International  GmbH  kalmog@team.mobile.de

!

4

Our  Platform "

mobile.de

Over  2.7M  ads  Approx.  50k  ads  per  

day

eBay  Kleinanzeigen

Nearly  12M  ads  Over  200k  ads  per  day

+50

Developers/QA

11

Teams

1826

VMs

Our  Platform

6

What  Does  PD  Want? "

# $

%&

'Requirements

REPRODUCTION CONSISTENT  TEST    DATA

SIMPLE  SETUP ROLLBACK

A  Standard  Integra  setup  @  mobile

7

What  Does  PD  Need? (

!O/S  Installation Debian/Solaris

"

#

Server  ConfigurationTomcats,  Databases,  Search  Indexers,  DNS,  Proxy  Servers  and  more

Data  Initialization Initial  Test  Data  Sets

Application  DeploymentWAR/JAR

Approx.  60  Virtual  machines  per  environment Ñ VM  Install

$

8

What  Does  SiteOps  Want? "

[ ©

9

The  Solution:  Autoconf "

*

%

'

A  Server  Provisioning  system  and  CMDB

Automated  HW  and  VM  server  provisioning

Servers  inventoried  as  a  resource  in  a  database%

'

Reduces  provisioning  time  from  hours  to  minutes)

10

Inventory  Database  Overview "

11

Inventory  Database  Overview "

insert into server set manufacturer='VMware, Inc.', productname='VMware Virtual Platform'; select last_insert_id() into @var_server_id; insert into serverdetails set server_id=@var_server_id,detailkey='datacenter_number', detailvalue='44'; insert into serverdetails set server_id=@var_server_id,detailkey='virtual', detailvalue='vmware'; insert into serverdetails set server_id=@var_server_id,detailkey='vmware_createopts_desire_hdd', detailvalue='12'; insert into serverdetails set server_id=@var_server_id,detailkey='vmware_createopts_desire_mem', detailvalue='3072'; insert into serverdetails set server_id=@var_server_id,detailkey='vmware_createopts_desire_cpu', detailvalue='1'; insert into assignment_queue set server_id=@var_server_id, serverclass_id=477, sequence_number=48, hostsuffix=2; select * from view_assignment_queue_hostdetails where server_id=@var_server_id;

12

There’s  No  Shame  in  Using  A  Mouse! "

13

The  Solution:  Genesis "

*

%

'

A  “Master”  Autoconf  Instance  for  Integras

Based  on  Autoconf

Includes  a  Web-­‐UI  and  API%

'

DB  Resource  for  other  Tools*DNS,  Deployment,  Monitoring,  Puppet

Joined  Project  between  PD  &  SiteOps+

14

Genesis:  Web-­‐UI   "

15

Inventory  Database:  Integra  Reference  List   "

16

Genesis:  API   "

17

Genesis:  API   "

root@genesis44-219:~# vm_creator.py -i graphite

Assignment Queue entry for "graphite":

{ "serverClassId": 352, "sequenceNumber": 17, "hostSuffix": "1", "hostPrefix": "graphite", "modificationTime": null, "state": "newrequest", "serverId": 243, "id": 335 }

18

Genesis:  API   "

19

Genesis:  Data  Initialization "

*

%

'

Set  of  BASH  Scripts

Initializes  Integra’s  Database  according  to  use  case%

'

Sets  up  database  replication  &  Initialization)

Can  be  launched  from  the  Genesis  GUI*

20

Application  Deployment:  Autodeploy "

*

%

'

Continuous  Deployment  System

Web  interface  &  API

Decentralized  architecture%

'

Full  Deployment  in  20  minutes)

Instant  feedback  &  Logging*

Collaborative  effort  between  PD  &  SiteOps+

21

Autodeploy  Architecture   "

eBay Inc. confidential

Architecture Overview

7

Autodeploy UI

AutoDeploy Web App

AutoDeployDB (infradb)

Apache ZooKeeper

Server running Autodeploy

Agent

Server running Autodeploy

Agent

Server running Autodeploy

Agent

22

Autodeploy  Architecture:  Host  Agent   "

eBay Inc. confidential

AutoDeploy Agent - What happens during a deployment

8

Text

XMLDEPLOY

PLAN

Autodeploy Agent

TARGET HOST 1. Autodeploy sends deploy instructions to autodeploy agent running on target host.

2. Autodeploy agent executes local deployment script on target host.

3. Deployment script executes deployment steps (autonomous)

deployment script

• Call LBAPI• Call Nagios• Download software• Create properties• Test app• update releases table on

infradb• exit with status

4. Script output and exit code sent back to autodeploy agent

Loggingback to

AutoDeploy

1.

2.

3.4.

Script outputsent to Agent

5. Logging and status reported back to Autodeploy by agent

AutoDeploy Web Application

5.

23

Autodeploy:  GUI   "

24

Autodeploy:  GUI   "

2014

25

Autodeploy:  Stats ,

Production Pre-­‐Production  ( Integra)

Rollouts  Per  Day 8 21

Applica2ons  Changes  per  day  (avg) 29 2017

Deployable  Ar2facts  (Total) 279 279

Deployable  Hosts  (Total) 555 1621

26

Virtualization  in  Test  Environment ,

0!

450!

900!

1350!

1800!

2006! 2007! 2009! 2010! 2011! 2013! 2014!

Physical! Virtual!

27

Today ,

CURRENT  STATUS

• We  know  it  well,  it  sbll  consumes  bme  and  there  is  very  licle  led  to  automate.

• IaaS  is  becoming  ubiquitous,  PaaS  is  our  new  challenge.

• Several  IaaS  clouds  are  already  available  at  eBay.

• We  already  have  a  sort  of  IaaS  (VMWare/Autoconf)  of  our  own  and  it  works  well  enough  for  the  short  term.

28

What  Does  The  Future  Hold? ,

,Development   teams   and   processes  are  moving  towards  Micro-­‐Services

29

What  Does  The  Future  Hold? ,

THANK  YOU!  :-­‐)-Karen  Almog  Sr.  Web  Operations  Engineer,  Mobile  International  GmbH  kalmog@team.mobile.de