the service score gamifying operational …...gamifying operational excellence basically, if we can...

159
Gamifying Operational Excellence The Service Score Card

Upload: others

Post on 06-Jul-2020

94 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Gamifying Operational Excellence

TheService Score Card

Page 2: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

1 The Problem

3 A Solution tour

4 The results

5 Take aways & lessons Learnt & Questions

2 A Solution idea

Agenda

Page 3: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

“If it's not broken, I’ll fix it.”

From Australia, on loan as

Staff SRE @ linkedIn

jobs, companies, recruiter

& Finder of encoding bugs

about meDanny ☃ Lawrence

Page 4: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

“If it's not broken, I’ll fix it.”

From Australia, on loan as

Staff SRE @ linkedIn

jobs, companies, recruiter

& Finder of encoding bugs

about meDanny ☃ Lawrence

Page 5: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

“If it's not broken, I’ll fix it.”

From Australia, on loan as

Staff SRE @ linkedIn

jobs, companies, recruiter

& Finder of encoding bugs

about meDanny ☃ Lawrence

Page 6: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

“If it's not broken, I’ll fix it.”

From Australia, on loan as

Staff SRE @ linkedIn

jobs, companies, recruiter

& Finder of encoding bugs

about meDanny ☃ Lawrence

Page 7: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

“If it's not broken, I’ll fix it.”

From Australia, on loan as

Staff SRE @ linkedIn

jobs, companies, recruiter

& Finder of encoding bugs

about meDanny ☃ Lawrence

Page 8: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Good news SRECON.

You passed the ☃ test.

about meDanny ☃ Lawrence

Page 9: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Some terms(before we really get started)

Page 10: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Operational Excellenceeffective and efficient delivery of information, technology, and services required by end users

that add measurable value.

10

Gamifying Operational Excellence

Page 11: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Operational ExcellenceDoing everything required to make sure

all of your services are as fast and as reliable as possible.

11

Gamifying Operational Excellence

Page 12: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Gamificationapplication of game-design elements and game

principles in non-game contexts.

12

Gamifying Operational Excellence

Page 13: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Some background(LinkedIn SRE crash course)

Page 14: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Mostly JavaMultitudes of services

Doing lots of thingsService-oriented architectureEverything talks to everything

My direct team looks after 80+ servicesWe have 200+ SREs

14

LinkedIn SRE Crash Course

Page 15: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

The Problem(What started this whole thing)

Page 16: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Problem 1:The GOOD

& The BAD

16

Gamifying Operational Excellence

Page 17: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

BAD serviceswake me up

17

Gamifying Operational Excellence

Page 18: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

GOOD serviceslet me sleep

18

Gamifying Operational Excellence

Page 19: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

What makes a GOOD service at LinkedIn is a moving target.

19

Gamifying Operational Excellence

Page 20: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Technologies and dependencies change

over time.

20

Gamifying Operational Excellence

Page 21: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Upgrading dependencies & libraries Java / Jetty / Play / Tomcat

Correct usage of TLSSwitching databases / caches

Migrate from SVN to GITReduce application startup time

Setup error budgetingTrue up the number of metrics

21

Some examples

Page 22: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

A GOOD service can turn into a BAD service.

If you are not checking it

22

Gamifying Operational Excellence

Page 23: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

UnfortunatelyBAD services

do not magically turn into

GOOD services23

Gamifying Operational Excellence

Page 24: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Problem 2:Knowing what is BAD

24

Gamifying Operational Excellence

Page 25: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Problem 3:Knowing why it’s BAD

25

Gamifying Operational Excellence

Page 26: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Problem 4:Tribal knowledge

about how to get to GOOD

26

Gamifying Operational Excellence

Page 27: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

The only thing we appear to hate more than not having documentation,

...Is writing documentation.

27

Gamifying Operational Excellence

Page 28: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

The Problemsummary

Page 29: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

BAD services wake me upTime will cause GOOD to turn BAD

Hard to know what is BADHard to know why is BAD

Not sure how to fix the BAD

29

Gamifying Operational Excellence

Page 30: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

The Service ScoreCard(A solution)

Page 31: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

In order determine the healthof the services we support,

we define a list of production requirements.

31

Gamifying Operational Excellence

Page 32: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Apply a weight to each requirement

32

Gamifying Operational Excellence

Page 33: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Codify each requirement into a check.

33

Gamifying Operational Excellence

Page 34: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Execute these checksfor each service

34

Service Scorecard

Page 35: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Tally up the results for service.

35

Gamifying Operational Excellence

Page 36: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Grade the service from“F” to “A+”

36

Gamifying Operational Excellence

Page 37: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Add all the services into a highscore system

37

Gamifying Operational Excellence

Page 38: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Then

38

Gamifying Operational Excellence

Page 39: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Publish those scores to the company

39

Gamifying Operational Excellence

Page 40: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

This is great, but how do I improve the score?

How can I add X check into the system.

40

Gamifying Operational Excellence

Page 41: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

What makes a check?

Page 42: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

checks are one type of plugin.

fetch plugins gather datacheck plugins check the data.

42

Gamifying Operational Excellence

Page 43: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

We use the fetch plugin to gather remote data from:

SVN, GIT, Configuration DBs,host databases, monitoring systems, build systems, deployment systems.

43

Gamifying Operational Excellence

Page 44: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Basically,if we can fetch it,

then we do so.

44

Gamifying Operational Excellence

Page 45: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

We build a giant context object.

45

Gamifying Operational Excellence

Page 46: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

The check plugin will look at our context object.

46

Gamifying Operational Excellence

Page 47: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

All plugins are small python scripts,where small is 10~30 LOC

47

Gamifying Operational Excellence

Page 48: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Simply return 2 or 3 things.

state*: True, False, None or 0.0 - 1.0message*: short stringdata: python dict of interesting things.

48

Gamifying Operational Excellence

Page 49: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Example fetch plugin

Page 50: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.tags(“ownership”)def fetch_ownership(service_name): “Fetch all the ownership data of a service”

o = r.get(“http://owners/” + service_name)

return True, “gathered data”, o.json()

50

Page 51: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.tags(“ownership”)def fetch_ownership(service_name): “Fetch all the ownership data of a service”

o = r.get(“http://owners/” + service_name)

return True, “gathered data”, o.json()

51

Page 52: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.tags(“ownership”)def fetch_ownership(service_name): “Fetch all the ownership data of a service”

o = r.get(“http://owners/” + service_name)

return True, “gathered data”, o.json()

52

Page 53: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.tags(“ownership”)def fetch_ownership(service_name): “Fetch all the ownership data of a service”

o = r.get(“http://owners/” + service_name)

return True, “gathered data”, o.json()

53

Page 54: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.tags(“ownership”)def fetch_ownership(service_name): “Fetch all the ownership data of a service”

o = r.get(“http://owners/” + service_name)

return True, “gathered owner data”, o.json()

54

Page 55: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Example check plugin

Page 56: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.weight(5)@ssc.tags(‘ownership’)@ssc.wiki(‘http://wiki/ssc_eng_owner’)def check_eng_team(ctx): “ensure ENG ownership of a service”

if ctx.ownership.eng_team: return True, ctx.ownership.eng_team return False, “missing eng_team”

56

Page 57: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.weight(5)@ssc.tags(‘ownership’)@ssc.wiki(‘http://wiki/ssc_eng_owner’)def check_eng_team(ctx): “ensure ENG ownership of a service”

if ctx.ownership.eng_team: return True, ctx.ownership.eng_team return False, “missing eng_team”

57

Page 58: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.weight(5)@ssc.tags(‘ownership’)@ssc.wiki(‘http://wiki/ssc_eng_owner’)def check_eng_team(ctx): “ensure ENG ownership of a service”

if ctx.ownership.eng_team: return True, ctx.ownership.eng_team return False, “missing eng_team”

58

Page 59: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.weight(5)@ssc.tags(‘ownership’)@ssc.wiki(‘http://wiki/ssc_eng_owner’)def check_eng_team(ctx): “ensure ENG ownership of a service”

if ctx.ownership.eng_team: return True, ctx.ownership.eng_team return False, “missing eng_team”

59

Page 60: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.weight(5)@ssc.tags(‘ownership’)@ssc.wiki(‘http://wiki/ssc_eng_owner’)def check_eng_team(ctx): “ensure ENG ownership of a service”

if ctx.ownership.eng_team: return True, ctx.ownership.eng_team return False, “missing eng_team”

60

Page 61: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.weight(5)@ssc.tags(‘ownership’)@ssc.wiki(‘http://wiki/ssc_eng_owner’)def check_eng_team(ctx): “ensure ENG ownership of a service”

if ctx.ownership.eng_team: return True, ctx.ownership.eng_team return False, “missing eng_team”

61

Page 62: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

@ssc.weight(5)@ssc.tags(‘ownership’)@ssc.wiki(‘http://wiki/ssc_eng_owner’)def check_eng_team(ctx): “ensure ENG ownership of a service”

if ctx.ownership.eng_team: return True, ctx.ownership.eng_team return False, “missing eng_team”

62

Page 63: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Putting it all together

Page 64: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Problems

Understanding what is BADKnowing why it is BAD

Not sure how to fix the BAD

64

Gamifying Operational Excellence

Page 65: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Problems

Understanding what is BAD

65

Gamifying Operational Excellence

Page 66: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

66

Service Scorecard

Page 67: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

67

Service Scorecard

Page 68: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

68

Service Scorecard

Page 69: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

69

Service Scorecard

Page 70: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

70

Service Scorecard

Page 71: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

71

Service Scorecard

Page 72: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

72

Service Scorecard

Page 73: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

73

Service Scorecard

Page 74: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

74

Service Scorecard

Page 75: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

75

Service Scorecard

Page 76: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

76

Service Scorecard

Page 77: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

77

Service Scorecard

Page 78: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

78

Service Scorecard

Page 79: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

79

Service Scorecard

Page 80: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Problems

Understanding what is BADKnowing why it is BAD

80

Gamifying Operational Excellence

Page 81: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

81

Service Scorecard

Page 82: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

82

Service Scorecard

Page 83: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

83

Page 84: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

84

Page 85: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

85

Page 86: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

86

Page 87: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

87

Page 88: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

88

Page 89: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

89

Page 90: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

90

Page 91: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

91

Page 92: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

92

Page 93: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

93

Page 94: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Problems

Understanding what is BADKnowing why it is BAD

Not sure how to fix the BAD

94

Gamifying Operational Excellence

Page 95: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

95

Page 96: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

96

Page 97: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

97

Page 98: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

98

Page 99: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

99

Page 100: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

What is the check?Why is it important?

How long it will take to fix?How will it be fixed?

100

Gamifying Operational Excellence

Page 101: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

101

Page 102: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

102

AngularJSimage: CC BY 4.0 https://angular.io/presskit.html (2017)

Page 103: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

103

{{service_name}}becomes

jobs-server

Page 104: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

104

Page 105: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

105

{{context.ownership.eng_owner}}becomesjobs-team

Page 106: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Using our fetched data in the wiki

Page 107: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

107

{{service_name}}

Page 108: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

108

{html}

<script src=”https://cdn/angularjs.js”/ >

{html}

Page 109: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

109

var query = $location.search();

var service_name = query[‘service_name’];

var url = ‘http://ssc/api/’ + service_name;

$http.get().success(

function(ctx) {

$scope.ctx = ctx;

}

);

Page 110: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

110

var query = $location.search();

var service_name = query[‘service_name’];

var url = ‘http://ssc/api/’ + service_name;

$http.get().success(

function(ctx) {

$scope.ctx = ctx;

}

);

Page 111: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

111

var query = $location.search();

var service_name = query[‘service_name’];

var url = ‘http://ssc/api/’ + service_name;

$http.get().success(

function(ctx) {

$scope.ctx = ctx;

}

);

Page 112: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

112

var query = $location.search();

var service_name = query[‘service_name’];

var url = ‘http://ssc/api/’ + service_name;

$http.get().success(

function(ctx) {

$scope.ctx = ctx;

}

);

Page 113: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

113

var query = $location.search();

var service_name = query[‘service_name’];

var url = ‘http://ssc/api/’ + service_name;

$http.get().success(

function(ctx) {

$scope.ctx = data;

}

);

Page 114: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

114

var query = $location.search();

var service_name = query[‘service_name’];

var url = ‘http://ssc/api/’ + service_name;

$http.get().success(

function(ctx) {

$scope.ctx = ctx;

}

);

Page 115: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

115

{{ctx.ownership.owner_eng}}

Page 116: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

116

{{ctx.ownership.owner_eng}}

{{ctx.number_of_hosts}}

{{ctx.product.lib.jetty.version}}

{{ctx.hosts.hostnames}}

{{ctx.is_deployed_in_prod}}

{{ctx.commits.last_commit}}

Page 117: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Problems

Understanding what is BADKnowing why it is BAD

Not sure how to fix the BAD

117

Gamifying Operational Excellence

Page 118: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Now

Reports show what is BADChecks validate why it is BADWiki shows how to fix the BAD

118

Gamifying Operational Excellence

Page 119: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

No more of these emails

“If you use a lib-core, then upgrade it, we found a bug”

119

Gamifying Operational Excellence

Page 120: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

How many of my 80 services use this lib?How do I check?

How do I upgrade?

120

Gamifying Operational Excellence

Page 121: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

121

Page 122: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

122

Page 123: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

123

Page 124: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Where does this tool fit?

Page 125: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

125

Gamifying Operational Excellence

pre-commit Build Deployment Monitoring

Page 126: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

126

Gamifying Operational Excellence

pre-commit Build Deployment Monitoring

Service Scorecard

Page 127: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

127

Gamifying Operational Excellence

pre-commit Build Deployment Monitoring

Service Scorecard

API

Page 128: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

128

Gamifying Operational Excellence

Service Scorecard

API

hack-days Reporting Deployment Monitoring

Page 129: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Results &

Outcomes

Page 130: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

What we do with the scores?

130

Gamifying Operational Excellence

Page 131: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Priority #1:Getting the grades better

131

Gamifying Operational Excellence

Page 132: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

132

When we started Now

Average grade for my team 40% 80%

Average score across SRE 35% 60%

Checks in 24 hours 15,560 89,859

Number of checks per service 15 31

Center the source to page, and align to bottom of page number. Do not increase in size, and keep on one line.

Gamifying Operational Excellence

Page 133: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

We can now explore news ways to use the scores

133

Gamifying Operational Excellence

Page 134: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Carrot&

Stick

134

Gamifying Operational Excellence

Page 135: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Carrot / GOOD

Stick / BAD

135

Gamifying Operational Excellence

Page 136: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

No SRE supportfor

F Gradeservices.

136

Gamifying Operational Excellence

Page 137: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

F Grade services generally cause the

most problems.

137

Gamifying Operational Excellence

Page 138: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

No deploy moratorium for

A+ services

138

Gamifying Operational Excellence

Page 139: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

A+ services generally cause the

least problems.

139

Gamifying Operational Excellence

Page 140: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

A servicesare allowed to deploy 24/7

140

Gamifying Operational Excellence

Page 141: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Premium SRE support for A+ services

141

Gamifying Operational Excellence

Page 142: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Priority build queuesfor

GOODServices.

142

Gamifying Operational Excellence

Page 143: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Tiger teams to raise the scores on

F Grade services

143

Gamifying Operational Excellence

Page 144: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Hack Days

144

Gamifying Operational Excellence

Page 145: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

FREE BEER

145

Gamifying Operational Excellence

Page 146: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Basically any problem can be solve with

FREE BEER

146

Gamifying Operational Excellence

Page 147: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

OR T-Shirts

147

Gamifying Operational Excellence

Page 148: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

/

148

Page 149: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Influence where we allocate open headcount

149

Gamifying Operational Excellence

Page 150: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Simple way to get things done

150

Gamifying Operational Excellence

Page 151: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Take aways&

Lessons Learnt

Page 152: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Everyone cares about Reliability.

152

Gamifying Operational Excellence

Page 153: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Everyone cares about Reliability,

Everyone is a Site Reliability Engineer.

153

Gamifying Operational Excellence

Page 154: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Everyone cares about Reliability,

You just need to empower them.

154

Gamifying Operational Excellence

Page 155: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Hack Days are important,

This POC was built in an afternoon.

155

Gamifying Operational Excellence

Page 156: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Getting the data was easy,

Finding interesting ways to use it is hard.

156

Gamifying Operational Excellence

Page 157: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Make it as easy as possible to do the right thing.

157

Gamifying Operational Excellence

Page 158: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Cheers !

Page 159: The Service Score Gamifying Operational …...Gamifying Operational Excellence Basically, if we can fetch it, then we do so. 44 Gamifying Operational Excellence We build a giant context

Q & A