Debugging 101Neelima Mukiri

Case Studies (Please refer to the blog

post)

Define the Problem

Who?

What?

Where? When?

How?

Why?

Define the Problem

Understand the System• Product ecosystem

• Programming Language

• Frameworks

• Operating System

• Networking

• Storage

• Hardware

Collect More Information

• Software, hardware configurations

• Work loads

• State of the system at panic

• Compare with a working system

• User and privilege information

Recreate the Failure

http://www.youtube.com/watch?v=gIdcvIqkSBA&t=0m55s

Identify Potential Causes

• Brainstorm

• Clues from data

• Experience and Intuition

• Potential Cause and effect analysis

Divide and Conquer

• Software Instrumentation

• Hardware Dumps

• Hardware assisted memory virtualization

• Software/Hardware Versions Involved

Track the Changes

• Document EVERYTHING

• It is easy to lose track of experiments done, results and side effects

• Recreate the failure after every change

• Absence of failure means nothing…almost

Cause and effect analysis

• Zoom in on probable cause

• Does it explain all the symptoms

• Does anything in the system state contradict expected behavior for this cause

• Test fixes

DON’T PANIC

Method to the madness• Define the problem precisely

• Understand the system in depth

• Collect data

• Catch the failure

• Identify potential causes

• Divide and conquer

• Track changes and effects

• Document everything

• Test the fixes

• Diversity Rocks

Collaborate

Brainstorm

Discuss, explain, get a fresh look

Design for Debugging• Bugs are inevitable

• Logs, monitoring, configuration collection

• Replication Tools

• Stress Options

• Debug tools are Products

• Can you Ping your system?

• Workarounds and Feature Switches

Happy Debugging!