smart log analysis

SMART LOG ANALYSIS

A General Framework and SMB Prototype

Windows ServiceabilityTim Burke, Kishore Chintalapati (manager)

Mike Tiberio (coach), Apurva Sharma, Samarth Shetty Badilaguthu

TALK OVERVIEW

Problem Space Current Approaches Design Objectives My Project: Smart Log Analysis and SMB Prototype Benefits Future Plans Demo

PROBLEM SPACE

Multiple Data Sources Multiple Tools (Netmon, Perfmon, Notepad,

…) Difficulty in correlating different source

Information Overload Manual Analysis Knowledge Loss

CURRENT APPROACHES

Open Notepad Open NetMon Repeat The Nuclear Option

Perl Grep

Credit: Eric Roode

\b(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b

http://www.regular-expressions.info/examples.html

THE RADIANT FUTURE

Network Captures ETW Traces Custom Logs

Smart Analysis Framework

Viewer Automatic Analysis

DESIGN OBJECTIVES

A unified way of viewing, searching, and analyzing data

Easily track and highlight relationships among data.

Group data into high-level operations Extensibility and Flexibility

DESIGN CONSIDERATIONS

Data is data, independent of the source Data consists of sets of named values Modular Easy rule creation Performance and Scalability Developer focused

MY PROJECT

Framework Viewer Prototype Text Rule Editor

From Logs From Source

Extensible Component Agnostic Scalable Embeddable

THE FRAMEWORK

Storage Plugins

Provider RulesFile Format Plugins

Log Viewer

Query Engine

SQL Server

Parsed Data

Log Parser

ETW Parser

Windows Events

Etc.

RDR

SRV

Log FIles

Config Files

Custom Storage

Parsed Data

Storage Manager

Format Engine

CLR Adapter

Formatting Rules

Saved Queries

LOG VIEWER

Boolean expression filters Filter based on any tag or value Similar to Netmon filters

Procedural queries Data correlation Complex scenarios

Custom formatting

TEXT LOG RULE EDITOR

Easy creation of parsing rules From text logs From source code

Preview rule effects

BENEFITS

Allows quicker, easier debugging Automates common analysis tasks Merges data sources to allow cross-

source analysis.

FUTURE PLANS

Complete the prototypes Implement more log parsers (Netmon,

…) Have component experts create rule

sets Implement automatic analyses on top

of the framework Integrate with other tools for capturing

data like MSDT

DEMO

QUESTIONS?