DEV233DEV233Team Foundation:Team Foundation:Using Metrics to Manage Using Metrics to Manage and Troubleshoot Your and Troubleshoot Your ProjectsProjects

DEV233DEV233Team Foundation:Team Foundation:Using Metrics to Manage Using Metrics to Manage and Troubleshoot Your and Troubleshoot Your ProjectsProjects

Sam GuckenheimerSam GuckenheimerGroup Product PlannerGroup Product PlannerMicrosoft CorporationMicrosoft Corporation

Managing 20Managing 20thth Century Projects Century Projects

Functionality

Time

Resources

Quality

““The Iron Triangle”The Iron Triangle”(err… tetrahedron)(err… tetrahedron)

Image copyright Tetra Pak

2121stst Century Mantra Century Mantra

Do more with less!Do more with less!But if your only variables are:But if your only variables are:

FunctionalityFunctionality

Resources Resources

TimeTime

Quality Quality

……then how are you going to do that? then how are you going to do that?

An Older TruthAn Older Truth

Happy families are all alike; Happy families are all alike; every unhappy family is every unhappy family is unhappy in its own way.unhappy in its own way.

Tolstoy, Tolstoy, Anna KareninaAnna Karenina

Все счастливые семьи похожи друг на друга, каждая несчастливая семья несчастлива по-своему.

Two Process ParadigmsTwo Process Paradigms

Traditionally, we decompose tasks and measure completion This is the cost accounting tradition

Alternatively, we account for customer value incrementally delivered This is the Lean Manufacturing and Theory of Constraints model

Two Process ParadigmsTwo Process ParadigmsWork-down Value-up

Planning and change process

Get planning and design right up front

Change happens, embrace it

Primary measurement

Task completion Only deliverables that the customer count

Definition of quality Conformance to specification

Value to the customer

Acceptance of variance

Tasks can be identified and estimated deterministically

Variance is part of all process flows

Intermediate work products

Documents, models, and other artifacts are necessary

Just enough to minimize the uncertainty

Troubleshooting approach

Time, resource, functionality and quality

Find and remove the bottlenecks

Approach to Trust Monitor and measure; incent performance relative to plan

Pride of workmanship and teamwork

Prescriptive vs. Descriptive MetricsPrescriptive vs. Descriptive Metrics

Robert D. Austin, Measuring and Managing Performance in Organizations

Metrics lose their meaning when made prescriptive

Standard Reporting with Standard Reporting with Team FoundationTeam FoundationStandard Reporting with Standard Reporting with Team FoundationTeam Foundation

How Far Can We Get in The Available How Far Can We Get in The Available Time?Time?

Work planned

Work completed

How Effectively Is Our (Outsourced) Team How Effectively Is Our (Outsourced) Team Delivering?Delivering?

Test rates (pass, inconclusive, fail)

shown in bars

Against code coverage, …

code churn, …

and active bugs

What Requirements Haven’t Been What Requirements Haven’t Been Tested? Tested?

Tracks progression of requirements’ states from Untested to Passed by

successive Build

Are We Getting Better?Are We Getting Better?

Custom Reporting with Team FoundationCustom Reporting with Team Foundation

Signs of UnhappinessSigns of Unhappiness

UnderestimatingUnderestimating

Uneven Task DecompositionUneven Task Decomposition

Scope CreepScope Creep

Inadequate Bug AllotmentInadequate Bug Allotment

Resource LeaksResource Leaks

Development Practices Too LooseDevelopment Practices Too Loose

Build FailuresBuild Failures

Inadequate Unit TestingInadequate Unit Testing

Tests Passing; Solution Doesn’t WorkTests Passing; Solution Doesn’t Work

Solution Stuck in TestingSolution Stuck in Testing

Too Little TestingToo Little Testing

UnderestimatingUnderestimating

Steady rates of progress, but slope

too shallow

Slow progress leading to cuts in

planned work, but not enough cuts

Uneven Task DecompositionUneven Task Decomposition

High variance in task completion/day indicates

inconsistency

Scope CreepScope Creep

“Dark matter” emerging during

iteration

Planned work is squeezed out

Inadequate Bug AllotmentInadequate Bug Allotment

New work not planned at iteration

start

Resource LeaksResource Leaks

Sudden drop in velocity indicates other drain on

resources

Development Practices Too Development Practices Too LooseLoose

Growing “Fault Feedback Ratio” – Bugs requiring

multiple handling

Build FailuresBuild Failures

Inadequate Unit TestingInadequate Unit Testing

Rising Code Churn

Falling Code CoverageFewer Passing

and More Inconclusive

Tests

Tests Passing; Solution Doesn’t Tests Passing; Solution Doesn’t WorkWork

Flat Big Rate

Flat & Low Code Coverage

Tests Passing; Solution Doesn’t Tests Passing; Solution Doesn’t WorkWork

Find > Fix leads to growing bug backlog

above

Tests Passing; Solution Doesn’t Tests Passing; Solution Doesn’t WorkWork

Tests aren’t catching or regressing bugs

Solution Stuck in TestingSolution Stuck in Testing

Bulge in resolved →Insufficient resources or

inadequate quality from dev

Too Little TestingToo Little Testing

Testing Rates Stuck Low

Drilldown with Microsoft Drilldown with Microsoft Office ExcelOffice ExcelDrilldown with Microsoft Drilldown with Microsoft Office ExcelOffice Excel

Friction-Free Metrics Enable Friction-Free Metrics Enable TransparencyTransparency

Visual Studio Team SystemVisual Studio Team SystemVisual Studio Team SuiteVisual Studio Team Suite

MS

F P

rocess a

nd

Gu

idan

ce

MS

F P

rocess a

nd

Gu

idan

ce

Visual Studio Team Foundation ServerVisual Studio Team Foundation Server

VisualStudio

IndustryPartners

SoftwareSoftwareArchitectsArchitects

SoftwareSoftwareDevelopersDevelopers

SoftwareSoftwareTestersTesters

DatabaseDatabaseProfessionalsProfessionalsVisual

StudioTeam

ExplorerApplication Application ModelingModeling

Infrastructure Infrastructure and Deployment and Deployment

ModelingModeling

Code AnalysisCode Analysis

Performance Performance TuningTuning

Security AnalysisSecurity Analysis DatabaseDatabase Deployment Deployment

Database Database Change Change

ManagementManagement

Database Database TestingTesting

Performance Performance TestingTesting

Manual TestingManual Testing

Test Case Test Case ManagementManagement

Visual Studio Professional Visual Studio Professional EditionEdition

Change ManagementChange Management

Work Item TrackingWork Item Tracking

ReportingReporting

Project SiteProject Site

Integration ServicesIntegration Services

Project ManagementProject Management

Load Test AgentLoad Test Agent

Visio and UML Visio and UML ModelingModeling

Class Class ModelingModeling

Unit TestingUnit Testing

Code Code CoverageCoverage

New!

Naked Self-PromotionNaked Self-PromotionFor More Information…For More Information…

ResourcesResourcesTechnical Chats and Webcastshttp://www.microsoft.com/communities/chats/default.mspx http://www.microsoft.com/usa/webcasts/default.asp

Microsoft Learning and Certificationhttp://www.microsoft.com/learning/default.mspx

MSDN & TechNet http://microsoft.com/msdnhttp://microsoft.com/technet

Virtual Labshttp://www.microsoft.com/technet/traincert/virtuallab/rms.mspx

Newsgroupshttp://communities2.microsoft.com/communities/newsgroups/en-us/default.aspx

Technical Community Siteshttp://www.microsoft.com/communities/default.mspx

User Groupshttp://www.microsoft.com/communities/usergroups/default.mspx

Fill out a session Fill out a session evaluation on evaluation on

CommNet andCommNet and Win an XBOX Win an XBOX

360!360!

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.