Innova'on  →  Execu'on  →  Solu'on  →  Delivered  

 

Infrastructure Automation for the Cloud

Chirag Jog, CTO Clogeny chirag@clogeny.com

Clogeny Confidential

http://www.clogeny.com © 2013 Clogeny Technologies

About Me

•  Strong Experience in DevOps, Infrastructure Automation

•  Built cutting edge products: •  Hybrid Cloud Migration Appliances •  Enterprise Java PaaS •  Multi-cloud single pane of glass Dashboard

•  Experience building highly scalable automated cloud applications

•  Leading engineering teams

Co-founder/CTO at Clogeny

http://www.clogeny.com © 2013 Clogeny Technologies

What is my talk about?

Infrastructure Automation for the Cloud

http://www.clogeny.com © 2013 Clogeny Technologies

What is my talk about?

Infrastructure Automation for the Cloud

http://www.clogeny.com © 2013 Clogeny Technologies

What is Infrastructure?

App LBs

App Servers

NoSQL

DB slaves

Cache

DB Cache

DBs

http://www.clogeny.com © 2013 Clogeny Technologies

What is Infrastructure?

DC1

DC3

DC2

http://www.clogeny.com © 2013 Clogeny Technologies

Problems in Managing Infrastructure

"   Configuration Drift

"   Changes causes Outages

"   Wall of Confusion – Dev and Ops

"   Non Functional Requirements: •  Backups, security, monitoring, synchronization,

performance, capacity planning

" Dev, Test and Prod – All Different Environments

"   Mystery Machine: On But don’t know why?

"   Code Complete != Production Ready

http://www.clogeny.com © 2013 Clogeny Technologies

Infrastructure is Hard !!

http://www.clogeny.com © 2013 Clogeny Technologies

Old Problem: Infrastructure is Hard

http://www.clogeny.com © 2013 Clogeny Technologies

Old Problem: Infrastructure is Hard

"   Proprietary Solutions

http://www.clogeny.com © 2013 Clogeny Technologies

Old Problem: Infrastructure is Hard

"   Solve very little of the problem "   Enterprise Focused "   Large Professional Services Bills "   On-premise "   Acquired by companies with large consulting

organizations "   Google, Amazon, Microsoft built their own tools

but its secret sauce

"   Proprietary Solutions

http://www.clogeny.com © 2013 Clogeny Technologies

Infrastructure is Changing !!

http://www.clogeny.com © 2013 Clogeny Technologies

Cloud Computing

http://www.clogeny.com © 2013 Clogeny Technologies

New Problem: Infrastructure is Changing

"   Easier to get .. but harder to manage

"   Demand is dynamic

"   Developers are crucial to Operations

"   Web/Cloud services are proliferating

"   Manual Configuration no longer a crutch

"   Few tools solve all the problems

"   No Shrink Wrap software

http://www.clogeny.com © 2013 Clogeny Technologies

Infrastructure is Code

"   API Driven Abstractions "   Infrastructure is now an Application

App LBs

App Servers

NoSQL

DB slaves

Cache

DB Cache

DBs

http://www.clogeny.com © 2013 Clogeny Technologies

Agile Infrastructure: The Final Frontier for Agile

http://www.clogeny.com © 2013 Clogeny Technologies

Agile Infrastructure

"   Infrastructure is Code: Apply the lessons learnt from Software Development

"   Build more Flexible systems and faster feedback

"   Beyond Continuous Integration: Continuous Delivery

"   10+ deploys per Day – Flickr, 2009

http://www.clogeny.com © 2013 Clogeny Technologies

Amazon.com

"   Mean time between deployments: 11.6s

"   Max # of deploys per hour : 1079

"   Mean # of host simultaneously receiving a

deployment: 10,000

"   Max # of hosts receiving a deployment: 30,000

http://www.clogeny.com © 2013 Clogeny Technologies

What is my talk about?

Infrastructure Automation for the Cloud

http://www.clogeny.com © 2013 Clogeny Technologies

Agile Infrastructure Techniques

u Configuration Management

u Version Control Everything

u Build from source

u One step deploy

u Monitoring

u Continuous Delivery

u Work Together – Dev and Ops

http://www.clogeny.com © 2013 Clogeny Technologies

What is Infrastructure (again) ?

• Nodes • Networking • Files • Directories • Symlinks • Mounts • DB Schema

Infrastructure is a collection of resources:

•  Routes •  Users • Groups •  Tasks •  Packages •  Software •  Services • Configurations •  Stuff

http://www.clogeny.com © 2013 Clogeny Technologies

Infrastructure as Code (really?)

package "ntp" do ! action :install !end !

service "ntpd" do ! action [:enable,:start] !end !

template "/etc/ntpd.conf" do ! source "ntpd.conf.erb" ! owner "root" ! group "root" ! mode 0644 ! action :create ! variables(:time_server => “time.example.com”) ! notifies :restart, “service[ntpd]”!end !

http://www.clogeny.com © 2013 Clogeny Technologies

Configuration Management

"   Infrastructure as Code "   Put systems into a known state "   Audit and enforce consistency "   Manage server lifecycle "   Reason about services, instead of systems "   Apply dev-test-prod cycle to infrastructure

http://www.clogeny.com © 2013 Clogeny Technologies

Configuration Management Tools

http://www.clogeny.com © 2013 Clogeny Technologies

Version Control Everything

"   Network Configurations "   System Configurations "   Application Configurations "   Application Code "   Database Schema "   Documentation "   Anything else that matters!

http://www.clogeny.com © 2013 Clogeny Technologies

Build from Source

"   Build Application and Infrastructure from source

" Dev, Test and Prod in sync

"   Test from a known state

"   Scaling

"   Disaster Recovery

http://www.clogeny.com © 2013 Clogeny Technologies

One Step Deploy

"   One automated process from version control to live services

"   Lower the time and cost to deploy

http://www.clogeny.com © 2013 Clogeny Technologies

Monitoring

"   Define ‘Normal’ Conditions "   Monitor Baselines, Charts, trends "   Monitor, Monitor and Monitor !

Ref: http://consultingblogs.emc.com/blogs/gracemollison/image_thumb_123A5775.png

http://www.clogeny.com © 2013 Clogeny Technologies

Continuous Integration

"   Check in regularly "   Create an automated and comprehensive test

suite. "   Test New builds "   Assert Services are running "   Deploy early and often "   No manual steps! "   Never go home on a broken build "   Build a pipeline!

Poll Master and

Compile

Run Unit Tests

Package

Deploy

Run Smoke/

Integration Tests

Release

http://www.clogeny.com © 2013 Clogeny Technologies

Continuous Delivery Pipeline

SCM Build CI Analyze Deploy Config

Private Clouds

Public Clouds

Repo Infrastructure

Continuous Integration

Continuous Deployment

http://www.clogeny.com © 2013 Clogeny Technologies

Work Together – Dev and Ops

" Dev and Ops see the same thing, in the same place

"   Share Metrics "   Share the Repository •  Configs in sync with Application Code

"   Minimize Surprise "   Always ship trunk/master

http://www.clogeny.com © 2013 Clogeny Technologies

How often do we see this?

http://www.clogeny.com © 2013 Clogeny Technologies

Work Together – Dev and Ops

" Devs need to understand the infrastructure "   Ops needs to understand the Application "   Manage Flows – Avoid Fires "   Build a culture – Avoid the blame game "   Break the silos and work together! "   Collaboration: not only when things go wrong "   No Rock star mentality

http://www.clogeny.com © 2013 Clogeny Technologies

What is DevOps ?

DevOps: is a software development method that stresses communication, collaboration and integration between Dev and Ops.

http://www.clogeny.com © 2013 Clogeny Technologies

What is DevOps ?

DevOps really is: "   Processes "   Tools "   Culture

http://www.clogeny.com © 2013 Clogeny Technologies

DevOps

"   Processes and Tools •  Automated infrastructure •  Shared version control •  One step build and deploy •  Shared metrics

"   Culture •  Respect •  Trust •  Healthy attitude about failure •  Avoiding Blame •  Superhero Culture

http://www.clogeny.com © 2013 Clogeny Technologies

Conclusion

ü Agile Infrastructure ü Embrace DevOps ü Business Agility ü  Improved Time to Market ü Ahead of the competition

http://www.clogeny.com © 2013 Clogeny Technologies

Questions ?

http://www.clogeny.com © 2013 Clogeny Technologies

References

"   Dev2Ops - http://dev2ops.org

"   Flickr’s 10 deploys per day - http://www.slideshare.net/jallspaw/10-deploys-per-day-dev-and-ops-cooperation-at-flickr

"   Agile Infrastructure - http://www.slideshare.net/littleidea/agile-infrastructure-agile-2009

" DevOps – The war is over - http://www.slideshare.net/jedi4ever/devops-the-war-is-over-if-you-want-it

" DevOps, the future is here – http://www.slideshare.net/KrisBuytaert/devops-the-future-is-here-its-just-not-evenly-distributed-yet

"   Jesse Robbins’s DevOps talk - http://www.slideshare.net/jesserobbins/devops-interop-las-vegas-jesse-robbins-opscode