1  

Control Everything

Gil Clark System Architect Intuit, Inc. Payments Division

2  

Topics / Agenda

•  DevOps Automation •  Not much Build Automation (lots of other

presentations around this) •  Not Test Automation (whole other world)

3  

But first…

•  What are we supposed to control? •  Source code? – Sure, we all know that •  But there is so much more •  Because the first DevOps rule is…

4  

Automate Everything!

Even  has  its  own  T-­‐Shirt  

5  

What is “Automate”?

1)  Start  with  some  content  that  represents  “Everything”  

OS  PVM  Code  

Data  

OS  PVM  Code  

OS  PVM  Code  

Network  

2)  Get  it  to  the  Compute  Pla1orm  

3)  Configure  and  Run  the  Compute  Pla1orm  

6  

What is “Everything” OS  

(sudoers,  IPtables,  services,  

crontab,  etc.,  etc.)  

PVM  (heap  space,  treads,  class  paths,  etc.,  

etc.)  

Source  Code  (preOy  obvious)  

Code  Config  (properQes  files,  class  paths,  

libraries,  etc.,  etc.)  

Data  (schema,  seed  data,  config  data,  etc.,  

etc.)  

But  this  is  not  complete,  we  have  all  the  automaQon  “code”  as  well  –  what  is  that?  

7  

“Get it to the platform” – Phase I Deployment Packaging Automation

1.  Sync  from  P4  •  App  Code  •  App  Config  

2.  Build  libraries  è  jar,  war,  etc.  

3.  Build  packages  è  zip,  rpm,  etc.  

•  Including  App  Config  4.  Publish  packages  è file  

server,  yum,  apt  

Content  Packaging  

File  Server  HTTP  FTP  Yum  Apt  

Nexus  Jenkins  Archive  

Perforce  

Release  “Library”  

Puppet  Chef  Salt  

Rundeck  Capistrano  Fabric  

Rsync/Rdist  

Content  SynchronizaQon  

Cloud  Management  

8  

“Run the compute platform” – Phase II Environment Orchestration Automation

1.  Sync  from  P4  •  AutomaQon  Code  •  Host  Env  Config  

2.  Build  packages  è  zip,  rpm,  etc.  

•  Env  specific  App  config  •  Host  Env  config  

3.  Publish  packages  è file  server,  yum,  apt  

File  Server  HTTP  FTP  Yum  Apt  

Nexus  Jenkins  Archive  

Perforce  

Release  “Library”  

Puppet  Chef  Salt  

Rundeck  Capistrano  Fabric  

Rsync/Rdist  

Content  SynchronizaQon  and  Process  Management  

Cloud  Management  

Content  Packaging  

9  

Another View of the System

Application Code and Configuration,

Schema, Scripts

Middleware and Configuration

(Jboss, Tomcat, Apache, Mule, Oracle)

Operating System Configuration

(Linux)

Configuration

•  Sync file content

•  Install packages

•  (VM Image Management)

Orchestration

•  Push file content

•  Start, stop services CI Server - Host metadata - Automation Code - Configuration Content - Application Content  

Host  machines  are  classified  by  role  which  determines  the  content  chosen  by  the  automaQon    

ApplicaQon  Stack  

10  

So, now we have Everything OS  

(sudoers,  IPtables,  services,  

crontab,  etc.,  etc.)  

PVM  (heap  space,  treads,  class  paths,  etc.,  

etc.)  

Source  Code  (preOy  obvious)  

Code  Config  (properQes  files,  class  paths,  

libraries,  etc.,  etc.)  

Cloud  Config  (VM  templates,  ACL  config,  VPC  

config,  CloudFormaQon  scripts,  etc.,  etc.)  

Data  (schema,  seed  data,  config  data,  etc.,  

etc.)  

AutomaQon  ConfguraQon  

(job  definiQons,  recipies)  

11  

Control Everything!

Intuit  uses  Perforce  to  control  all  this  code  

ConfiguraQon  as  Code  

Infrastructure  as  Code  

12  

Example Branching Model – Overall System

DevOps  Code  (ConfiguraQon  and  OrchestraQon)  

ApplicaQon  A  

ApplicaQon  B  

Code  Build  and  Package  Config  

Code  Build  and  Package  Config  

Datacenter  A  Environment  A1  Environment  A2  Datacenter  B  

Environment  B1  Environment  B2  

13  

DevOps Specific Branching

Datacenter  A  Environment  A1  

Overall  System  OrchestraQon  Jobs  and  Scripts  ConfiguraQon  Recipes  System  properQes  config  

OrchestraQon  Jobs  and  Scripts  ConfiguraQon  Recipes  System  properQes  config  Environment  A2  

Datacenter  B  

There  are  different  techniques  for  merging  this  content  •  Treat  as  modular  and  compose  the  content  (templates,  placeholders,  etc.)  •  Treat  as  monolithic  and  use  branch  /  merge  SCM  tools  

14  

Gluing Everything Together

Datacenter  Config  

Environment  Config  

ApplicaQon  Config   Puppet  

Rundeck  

     Jenkins  

App  1  App  1  

App  1  

App  1  App  1  

App  2  

App  1  App  1  

App  3  

     Jenkins  Source  Code  

Release  Server  

(Zip,  RPM)  

ArQfact  Server  (Nexus)  

15  

Example Process Steps // Start with overall systems state; for each datacenter { select datacenter specific content, save; } for each environment { select environment specific content, save; } for each application to be deployed { select app specific content; merge content from datacenter, environment, app } move merged content to master control server for the target environment in the target datacenter; populate configuration system content (Puppet manifests and modules); populate orchestration system content (Rundeck jobs); kick configuration system (Puppet kick ala Rundeck); run orchestration jobs (Rundeck); done;

16  

Closing Contact Info: gil_clark@intuit.com