incident management with workflows

Patrick Hoolboom September 22, 2016

Incident Managementwith Workflows

© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY

What is a Workflow?


What Is a Workflow?

• A sequence of processes through which a piece of work passes frominitiation to completion• Process as Code• Living Documentation

– Document your process in an easily human readable, executable format

© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. INTERNAL USE ONLY 3

Event Driven Automation 2.0

© 2016 BROCADE COMMUNICATIONS SYSTEMS, INC. 4

FBAR (saving 1532 hours/day)

Naoru

Nurse

Winston (powered by StackStorm)

Azure Automation

Mistral workflow service

StackStorm automation platform

ACT

OBSERVE

ORIENT

DECIDE

When to use a workflow


When to use a workflow

• Clearly defined process• When multiple systems or services need to be touched• Frequently performed tasks


Why Use Workflows?


Why Use Workflows?

• Consistency

– Trust that your automations will perform the same tasks every timefor a given event

• Speed– Reduce time to resolution for an incident

Audit– Creates a clear audit trail of what was done when

• Connect Disparate Systems


Tools…


What Can Be Automated?

• Security checks– On malware detection in a VM, isolate

network port on a switch• Blue-green app deployment

– On Jenkins tests passed, bring new vmclaster, deploy and configure app, setloadbalancer to send % of traffic to new app,monitor, roll forward, or back out

• Networking– On BGP peer goes down: collect

troubleshooting data, post on slack & createJIRA ticket

– On Link aggregation member error, checkload, if capacity of rest of LAG bundleenough, disable link with error

• Restart a down service– On monitoring event, bounce a service

• OpenStack orphan VM clean-up– On orphans detected, shut down, email owner,

keep for few days, delete• NFV:

– Nokia, AT&T, with Mistral and OpenStack• OpenStack VM evacuation on

hardware failures– On host RAID failure, get list of impacted VMs,

email VM owners, evacuate VMs, create JIRAticket for hardware replacement.

• Cassandra “node down”recovery

– Replace a node on alert

• Clean up disk space– On monitoring event, clean up disk space


StackStorm


Architecture

12

Web GUI CLI Chatops

Sensor Containers Action Runners

Sensor Plugins(inbound integrations)

Master Content Repo

to Audit…

Action Plugins(outbound integrations)

PLATFORM

CLIEN

TSPLU

GINS

AMQP message busAMQP message bus

Workflow Engine

REST API

{*}

RulesEngine

IFTTT.yml

KV Store

k[v]

● Diagnostic Workflows

● Remediation Workflows

Workflow DesignPatterns


Workflow Design PatternsDiagnostic Workflows

• Troubleshooting and data gathering steps• No remediations or changes to the system• Good way to “get your feet wet” with workflows


Workflow Design PatternsRemediation Workflows

• Fix the issue!• Should be triggered after diagnostic workflows if applicable•


● Facilitated Troubleshooting

● Auto-Remediation

Workflow Use CasesDuring an Incident


Workflow Use CasesFacilitated Troubleshooting

• Useful if you don’t quite trust the automation– Gain confidence in your workflows

• Faster Time to Resolution

• Consistent Data Collection• Diagnostic workflow with notifications

– Send data to user via

• Email

• Chat

• Ticketing System


Workflow Use CasesAuto-Remediation

• Trusted Automation– Will make automated changes to the system

• Much Faster Time to Resolution• Consistent Solutions• Less Pager Fatigue


● Low Disk Space Event

Example


Automation Example


Automation

EngineerService

Monitoring IncidentManagement

Event: “low diskon web301”

Web301 is“low disk”

Resolve knowncases, fast. Is it

/var/log? Clean up!

Unknownproblem, need a

human

Wake up, buddy.Something real

is going on…

● Email: [email protected]

● Twitter: @DoriftoShoes

Thank You!


incident management with workflows

Software