Reliably Complex ContextAndy DomeierSPS Commerce

@AndyJD_

Digital Transformation

• Technology is a competitive advantage more today than it has ever been

• Velocity Wins• Ship Fast, Ship Often• Critical to reduce the friction from getting an idea to production

• Performance Wins• Critical to reduce risk and outperform your competitor's performance and

availability (or your customer will fire you quickly)

@AndyJD_

Complexity is Increasing to Enable Velocity & Improve Performance

Change rates are higher than ever

SaaS tools are business critical (GitHub, Monitoring, Alerting, etc)

Micro Services are great for product agility

Products & services depend highly on 3rd

parties such as a cloud, data center, ISP, etc..

Meanwhile legacy systems & tech debt still exist and are likely generating a lot of revenue……………

@AndyJD_

Complexity Increases our Need for Context• Define Context: • Google:

• The circumstances that form the setting for an event, statement, or idea, and in terms of which it can be fully understood and assessed.

• Continuous Cognitive Demand• Intuition• Process• Continuous Partial Attention

Available Context

Change Management

DockerKube

Service Mesh

Incident Management

Monitoring

MetricsTracing

Logs

Observabilit

yDocumentation

ArchitectureMicro Services

Monolith

3rd Party Services

The Cloud

Org StructureRate of Change

Humans

Technical Debt

A/V J

Operational Readiness

Performance Reporting

How effective your organizationis with context correlates to how good your organizationwill be at delivering services.

Most Important

The People!

Predefined Metrics & Thresholds

The People!

Monitoring

Discoverable Information

The People!

Monitoring

Observability

Obscure & Hard to Find

The People!

Monitoring

Observability

Difficult & Time Consuming

People

PeopleWe are many things!• The Processes we Make• The Documentation we

Write• Principals & Practice we

Adopt• Diversity we all Bring• Things in our Head

People - Processes

• Change Management• Does your company have an extensive decision tree process?

• Do you have it memorized?• How often do you do it?

• Do you have visibility to changes that matter to you?

• Incident Management• Is this a known issue with known resolution steps?• Who needs to know?• Who can make decisions?

• Do you Agile?• How exactly do you Agile?

People – Culture & Org Structure

• Org & Culture will significantly influence the importance of context sharing mechanisms• Large & Rigid – Documentation scales• Small & Fluid – Stop on over!• Distributed – Documentation, Chat, Video, Screenshare

• Understanding Org can sometimes require significant context• Who owns what? What does the org incent them to deliver? How do we

effectively succeed together?

Operational Readiness

• A base set of consistent values for service delivery give you context about the app and enables you to make decisions• Operations Information

• Who?• Where?

• Performance• KPI’s• Cost

• Agility• Change Effectiveness• Architecture & Resiliency

• Security• Sounds cool!

Operational Confidence:If it’s important at Readiness, It’s

important over time

@AndyJD_

People – Local Disk = Brains

• How many things are you able to do through out an average day just because you “know”?• Deploy? Recover Incidents? Process? Is it Healthy? Batteries? Bus Schedule?

• How many of you know something about a service, that you know less than 3 other people at your company know?• Why document or automate it, it’s happened once in 3 years?• If I document it, no one will find it (or try to find it) anyways?• This just comes to me anyways• We perceive documentation like cold storage, it’s there but it’s slow

People!• We are the most complex part of our own systems

• Effort Investments• Be explicit in what we should store in our brains• Write down decisions you make• Have a clear path for information discovery in your documentation approach• Be a good teammate and take time to share valuable context, keep time and place in

mind

• Doing the people-ing well gives everyone in your org cognitive freedom to focus on solving what’s most valuable to your business & customers

Monitoring

Monitoring - Approach

• Consistency in Approach• Service Health

• Availability• Key Performance Indicators (KPIs)• Key Dependencies

• Dashboards & Alerting• Document or automate an opinion• Consistency

@AndyJD_

Monitoring –Perspectives• Monitor for Customer

Experience• Customer Experience often

comprises of many micro metrics

• Monitor for Delivery Team• Isolated to a subset of

services• Monitor for Specialists• Infra Team• App Team• DB Team

Macro Monitoring

Data Visualization

• Edward Tufte - Visual Literacy• The ability to interpret, negotiate, and make meaning

from information presented in the form of an image@AndyJD_

Micro Monitoring

Observability

Cindy Sridharan - @copyconstructhttps://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e?__s=1wtnvg3z1cq5rhuqprym

Visualization Shares Context but it’s Predefined

@AndyJD_

Observability - Discovery

• Where can you Seek Answers?• Monitoring & Observability Tools• Documentation & the Internet• People!

• What Context is Easiest to Ask Questions of?• Start with your local disk!• Where is the most information? (logs?)• Where is the easiest information to consume? (metrics? traces?)• Organized documentation• Ask a neighbor

Complexity Brings… Complexity

Twitter: @Chrismunns“I Love APIs” 2015

Observability - Expect Complexity Everywhere

Is it Still Observability if we Automate it?

Push Context or Ease Discovery

• Trigger Event:• Incident or Service Alert

• Helpful Context:• Who owns this Service?• Has something changed

recently?• Are the dependencies healthy?• Who owns the dependencies?• Has this happened before?• When & who fixed it last

time?• Is this impacting customers?

Push Context or Ease Discovery

• Trigger Event:• Change or Deployment

• Helpful Context:• Who cares about this Change?

(Dependencies)• Does this Change Impact

Customers?• What’s our Confidence in this

Change?• How big is this Change?

With the value of Context top of mind, we

can do more

You and your org have your own context….

Your complexity is unique to your environment…

How can you help your organization be most effective with your context?

Thanks!!!

@AndyJD_

Relevant Context…. When it Matters…

Monitoring & Alerting

Router LambdaSNS

Wiki Search

Alerts Commenter

DocumentationLookupThe

Wiki

API Gateway

Relevant Context…. When it Matters…

Incident Management

Router Lambda

SNS

Incident CommunicationChat

Recent Changes

Search Jira

Change Management

Jira Commenter

API Gateway

Stakeholders