stackstorm devops automation webinar

www.stackstorm.com!!

@Stack_Storm!!

July 2014!CONFIDENTIAL!

Vision, Common Operational Patterns, and a Little About

Our Approach!

Vision à Specific Patterns!

2!!

7/21/14!© 2014 StackStorm, Inc.!Confidential!

We’ve been chatting with you – what have we learned?!•  Let’s talk operational patterns. !•  A little on monitoring. !•  A lot on operations automation.!!We want to learn more today! !!What are we building?!!!!

Market!

“So$ware is ea,ng everything.”

{ developed and operated in the DevOps way }

!

StackStorm does DevOps opera,ons automa,on

!

3!!


StackStorm Presenters!Patrick Hoolboom, Unicorn Stormer!•  7+ years building and running high pace DevOps

environment (2 years at Cloudmark)!•  Puppet, Chef, Nagios, NewRelic, Logstash and more!•  “Open source your process” – lets share operational

templates!

!!

Evan Powell, Co-Founder & CEO!•  15+ years in infrastructure software!•  Founding CEO of Clarus Systems, acquired by OPNT!•  Founding CEO of Nexenta Systems!

•  Defined and led OpenStorage and SDS space!•  Led Nexenta to 5k+ customers, 280+ employees, $350 million+

of partner sales, $75 million in funding !

!

!!

4!!


130+ Discussions!

PIX | SYSTEM

5!!5!!


Source CA: h4p://www.slideshare.net/CAinc/dev-‐ops-‐research-‐cust-‐deck-‐mann-‐march-‐31

6!!


CI/CD, ChatOps, DevOps!

7!!

CI CD ChatOps “DevOps” CD*

Every single soul

SREs

Love it!

Stupid!


8!!

7/21/14 © 2014 StackStorm, Inc. ConfidenRal

Day 1: Pre4y well sorted Day 2: Pre4y much chaos •  But if infrastructure is immutable

then maybe we never have Day 2?

Patterns: Monitoring!Monitoring as a service:!• Check in your checks and monitoring is free!• Transparency => shame => compliance!Ongoing challenges:!• Pager fatigue!• Automate all the thresholds !• Dependencies => deduplicating alerts!• Drive => Host => Service => Application =>

User!

!

10!!


Patterns: Monitoring!

11!!

Nagios – Sensu – APM Zabbix

Graphite StatsD

Humans via ChatOps

PAGER DUTY

Auto correcRons: Nagios or other

Declare event, ask for help

-‐>>Event pipeline -‐>>

-‐>>Event pipeline -‐>>

Rieman / StackTach

Patterns: Monitoring!Infer the threshold!• Anomaly detection!• Pattern recognition!Start with a check!And/or update the checks and thresholds as you learn more!

Monitoring as a service " Remediation as a service!

!

12!!


Remediation as a Service #1!

13!!

Facebook’s FBAR: “AutomaRng the work of hundreds” h4ps://www.facebook.com/notes/facebook-‐engineering/making-‐facebook-‐self-‐healing/10150275248698920


Remediation as a Service #2!

14!!

Microsoh AutoPilot: States: •  Healthy, Failed,

ProbaRon RemediaRons (“Rs”): •  Restart, Reboot,

Reimage, RMA h4p://research.microsoh.com/pubs/64604/osr2007.pdf

“Machine learning algorithms to analyze these data in order to understand how to improve the policy selngs, with the ulRmate goal of automaRng many of the current manual policies. “

Patterns: Tools – Monitoring?!The #1 tool we have seen is NewRelic!• Roughly 75% of the discussions!• Amazing because it does not fit all the

requirements!The #2 tool we have seen is Splunk!• Still seeing them more than LogStash,

ElasticSearch although it is close!

!15!!


Patterns: Tools – Configuration!Puppet vs. Chef!• Ops vs. developers?!Ansible and Salt growing in importance!• Points of differentiation and also overlap!Best practices include:!• Separate source of truth !• Code reviews (duh?)!

!16!!


“State in the Repo” vs. Single Source of Truth!

17!!

•  Provides visibility into the system!

•  Fights configuration drift!•  Reduces convergence

time of changes!•  Tied in to monitoring

and configuration systems!


18!!

Lack of trust

Hard to test Simple script + simple script + simple script +

Zounds

What’s Wrong with Day 2 AutomaCon?

Monitoring

My script vs. your script IntegraCo

n

sprawl

Automation: Don’t Forget #1!Simplicity scales.

19!!

Best.


BEST PRACTICES •  Have X states (three seems logical) for a monitored

system •  Have Y remediaRons •  Document both •  Automate the known mappings

•  Atomic acRons •  Workflows or acRons of acRons to Re them together

Lessons learned from many operators large and small

Automation: Don’t Forget #2:!Failure detectors must be able to dis7nguish between the symptoms of failure and overloading, otherwise overloaded computers may be marked as failed and removed from service, amplifying the problem and triggering a cascade of failures that disables the enRre applicaRon.

20!!

h4p://research.microsoh.com/pubs/64604/osr2007.pdf

BEST PRACTICE Check a threshold with each step so that run away automaRons cannot occur.


Automation: Don’t Forget #3!Humans are good at pa4ern matching – if they perceive the pa4erns. •  Always provide context. When a threshold is violated that requires that humans get back involved you are at risk of pulng the human in a difficult place.

21!!

BEST PRACTICE When the system needs a human, provide the human with context. •  Automate the delivery of that context for the sake of consistency •  Don’t allow the automa,on to just be addi,onal complexity on top

of complexity


Automation: Don’t Forget #4!AutomaRon frees up humans to a4ack technical debt, refactor for growth and so forth. However, the implicit knowledge formed by eliminated manual acRon itself may be lost.

22!!

BEST PRACTICE Refer to the prior Don’t forget points including simplicity of state analysis and simplicity of allowed remediaRons. •  Also – allow manual invoking of automaRons •  Make automaRons human readable •  And otherwise transparent •  Give the automaRon an edgy personality (huh?)


ChatOps is Brilliant!Add some code to IRC or other chat to do stuff, and you have bot executed code. Add to that an enRre automaRon library – plus a personality – and the requirement that ALL changes happen through chat and DevOps happens.

23!!

BEST PRACTICES Get to know Hubot. And stay tuned for StackStorm’s take on this approach. ChatOps benefits: •  Dev/Ops interface and Newby / Old pro interface •  Context for humans •  Trust in automaRon

h4p://puppetlabs.com/blog/really-‐building-‐data-‐driven-‐infrastructure


Integrations & Relationships!

Auto Scaling


Integrations & Relationships!

Auto Scaling

AutomaRon as a Service


Before!

26!!

Events

AcCons

Events AcCons

Scripts

Scripts

AcCons

Scripts

Events

#BadAuto!•  Who did what to what and how did it go?!•  Fragments of automation and configuration management!•  No opportunity for learning!


StackStorm!

27!!

Events

AcCons

#ShareAndLearn!•  Scripts -> automations!•  Workflow – stitch stactions!•  Close the loop – map events to

automations!•  Total audit & transparency!•  Still DevOps friendly !•  All automations are code in repositories!


Technical Details!• Centralized view (federated) into automations!•  Audit trail and RBAC access controls!

•  Infrastructure as code!•  Full APIs, configurations as code!

• Collaboration inherent!•  Abstraction of automations for sharing!•  Two way integration into collaboration including ChatOps!

•  Bi directional close relationship with monitoring!•  StackStorm helps users determine where to put

problem analysis – StackTach and others promising!•  Facility for self learning!•  Simple today headed towards controller synthesis

approaches!

28!!


Example Use Cases!• OpenStack and hybrid management!

• Facilitated troubleshooting!• Over time – automated resolution!

• Multi-stage deployments!• Develop, stage, deploy for example!

• Response to security events!•  Post-hoc as opposed to real time!

29!!


Business Model!OpenSource!• Apache licenses!Free!• Free forever edition (community)!Enterprise!• On premise or hosted – all deployments now

are on premise!!

30!!


Who Wins?!

31!!


What Can You Do?!Register as a beta participant!• Are you doing some OpenStack?!• Can you provide feedback at least 2x per month?!• Automating something already?!•  Interested in ChatOps? Not required….!!Once we GA – grab free version!• Contribute to community w/ integrations and

automations!• OpenSource and share your operations patterns!

32!!


Questions!To respond, please unmute your line by pressing *6!!CI, CD, Day 2 operations?!• Where are you in the continuum?!

ChatOps – do you use it?!!Have you had automations run amok?!!DevOps team? Title? Silo?!!

33!!


Summary!World is changing rapidly!!We’ve learned a lot!!Safe, composable automations with circuit breakers and other controls should help!!

34!!


Private and confidential!

Thank YouFollow us on Twitter @Stack_Storm!

stackstorm devops automation webinar

Technology

stackstorm presenters

tools monitoring

monitoring myscriptvs

service remediation

common operational patterns

vision specic patterns

conguration systems

ceo of nexenta systems