as simple as possible, but no simpler sam guckenheimer [email protected]

64
As Simple As As Simple As Possible, Possible, But But No No Simpler Simpler Sam Guckenheimer Sam Guckenheimer http://lab.msdn.microsoft.com/vs2005 http://lab.msdn.microsoft.com/vs2005 /teamsystem/ /teamsystem/ [email protected] [email protected]

Upload: corey-townsend

Post on 13-Jan-2016

217 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

As Simple As Possible,As Simple As Possible,But But NoNo Simpler Simpler

Sam GuckenheimerSam Guckenheimerhttp://lab.msdn.microsoft.com/vs2005/teamsystem/http://lab.msdn.microsoft.com/vs2005/teamsystem/

[email protected]@microsoft.com

Page 2: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Simple Project ManagementSimple Project Management

Functionality

Quality

Resources

Time

““The Iron Triangle”The Iron Triangle”

(err… tetrahedron)(err… tetrahedron)

Page 3: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

2121stst Century Mantra Century Mantra

Do more with less!Do more with less! But if your only variables are:But if your only variables are:

FunctionalityFunctionality QualityQuality Resources Resources Time Time

……then how are you going to do that? then how are you going to do that?

Page 4: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

An Older TruthAn Older Truth

Happy families are all alike; every Happy families are all alike; every unhappy family is unhappy in its unhappy family is unhappy in its own way.own way.

Tolstoy, Tolstoy, Anna KareninaAnna Karenina

Все счастливые семьи похожи друг на друга, каждая несчастливая семья несчастлива по-своему.

Page 5: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 6: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 7: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

CodeCode Some why-notsSome why-nots

Use managed codeUse managed code Use modern Use modern

frameworksframeworks Use service-Use service-

oriented oriented architecturearchitecture

Use available toolsUse available tools

TransparencyTransparency Responsible Responsible

costingcosting Visible resultsVisible results

Available toolsAvailable tools Unit testsUnit tests Code coverageCode coverage Static analysisStatic analysis Profiling Profiling

performanceperformance Source controlSource control Work item trackingWork item tracking Build automationBuild automation

Page 8: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Unit Tests and Code CoverageUnit Tests and Code Coverage

Unit Test Results

Code Under Test not covered during the test run

Page 9: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Code AnalysisCode Analysis

http://blogs.msdn.com/jason_anderson/archive/2004/09/05/225798.aspx

Code Analysis recommendations as build warnings

Direct jump to code from the warning

Page 10: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 11: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Product DefinitionProduct Definition

Personas and ScenariosPersonas and Scenarios Qualities of ServiceQualities of Service Capture implicit requirementsCapture implicit requirements

Kano analysisKano analysis

Stack rankingStack ranking

Continually challenge your assumptions!

Page 12: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Personas and Scenarios Personas and Scenarios

PM Starts New Portfolio Project

PM Enumerates Requirements in Excel

PM Schedules Work in MS Project

PM Monitors Project Status

PM Reviews Project Status

PM Promotes For Deployment

CEO Signs Contract

Architect Updates Design

Architect Adds Tasks & Checks In

Dev Writes Code

Dev Writes & Runts Unit Tests

Dev Reviews Work

Dev Runs Code Analysis

Dev Writes Load Tests

Dev Checks In Work

Dev Diagnoses & Fixes

Dev Checks In Work

Tester Checks Build Status

Tester Runs Load Test

Tester Reports Bug

Jacqui AckermanJacqui AckermanProject ManagerProject Manager

Art BensonArt BensonArchitectArchitect

Martin GainesMartin GainesDeveloperDeveloper

Renee DavisRenee Davis TesterTester

Page 13: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Qualities of ServiceQualities of Service PerformancePerformance

ResponsivenessResponsiveness ConcurrencyConcurrency Efficiency Efficiency Fault toleranceFault tolerance ScalabilityScalability

TrustworthinessTrustworthiness SecuritySecurity PrivacyPrivacy Conformance to Conformance to

standardsstandards InteroperabilityInteroperability

UsabilityUsability AccessibilityAccessibility AttractivenessAttractiveness CompatibilityCompatibility DiscoverabilityDiscoverability Ease of useEase of use LocalizabilityLocalizability

ManageabilityManageability AvailabilityAvailability ReliabilityReliability Installability and Installability and

uninstallabilityuninstallability MaintainabilityMaintainability MonitorabilityMonitorability RecoverabilityRecoverability TestabilityTestability SupportabilitySupportability

Page 14: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Kano AnalysisKano Analysis

Hinshitsu (Quality), The Journal of the Japanese Society for Quality Control , XIV:2, pp.39-48, April 1984

Page 15: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Challenging AssumptionsChallenging Assumptions

Customer in usability lab

Customer’s desktop

Page 16: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 17: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

ArchitectureArchitecture

Service-Oriented ArchitectureService-Oriented Architecture Infrastructure ArchitectureInfrastructure Architecture LegacyLegacy

Page 18: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Service OrientationService OrientationBuild Build systemssystems using using autonomous autonomous servicesservices that adhere to the four that adhere to the four tenets of Service Orientation:tenets of Service Orientation:

1.1. Boundaries are explicitBoundaries are explicit

2.2. Services are autonomousServices are autonomous

3.3. Services share schema and Services share schema and contract, not classcontract, not class

4.4. Service compatibility is determined Service compatibility is determined based on policybased on policy

http://msdn.microsoft.com/msdnmag/issues/04/01/Indigo/default.aspx

Page 19: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Application DesignerApplication Designer

Service-OrientedArchitecture modelService-Oriented

Architecture model

Port Details editorPort Details editor

Page 20: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Infrastructure ArchitectureInfrastructure Architecture

Points of FailurePoints of Failure Points of ObservationPoints of Observation Points of AttackPoints of Attack ManageabilityManageability

Page 21: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Logical Infrastructure Logical Infrastructure DesignerDesigner

Services assignedto logical infrastructure

Services assignedto logical infrastructure

Architecture validatedagainst operational

settings and constraints

Architecture validatedagainst operational

settings and constraints

Page 22: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 23: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Build AutomationBuild Automation Nightly buildNightly build

Project heartbeatProject heartbeat PrePre check-in tests check-in tests

Validation of code prior against current Validation of code prior against current base prior to check-inbase prior to check-in

Variant is continuous integrationVariant is continuous integration Build verification testsBuild verification tests

Functional tests (from unit tests)Functional tests (from unit tests) Component integration testsComponent integration tests

Build reportingBuild reporting Against backlog, by check-in/changesetAgainst backlog, by check-in/changeset

Page 24: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Build ReportingBuild Reporting

Page 25: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 26: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

VersionsVersions

Track versions for each ofTrack versions for each of SourceSource TestsTests Executables and other runtimes you Executables and other runtimes you

createcreate XML, HTML, images, docs & databasesXML, HTML, images, docs & databases

Environmental/deployment componentsEnvironmental/deployment components BugsBugs

Report them together & relate themReport them together & relate them

Page 27: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 28: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

EnvironmentEnvironment

Production environmentProduction environment Test environmentTest environment Capturing environmentCapturing environment ToolsTools

Microsoft Virtual PCMicrosoft Virtual PC Microsoft Virtual Server Microsoft Virtual Server

Maintain lab imagesMaintain lab images

Page 29: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 30: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

SecuritySecurity

The core problemThe core problem Threat modelingThreat modeling Code analysisCode analysis Security testingSecurity testing

Michael Howard, Writing Secure Code, 2003J.D. Meier et al., Improving Web Application Security, 2003

Page 31: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Security: Core ProblemSecurity: Core Problem

Odds of securing a single level is 1 / Odds of securing a single level is 1 / ∞∞ Bad guy has to find only one vulnerability Bad guy has to find only one vulnerability Infinite time Infinite time

Microsoft as exampleMicrosoft as example 100’s of different IT environments100’s of different IT environments 2,500 unique attacks per day2,500 unique attacks per day 125,000 incoming virus-infected e-mails per month125,000 incoming virus-infected e-mails per month

Need to secure at every levelNeed to secure at every level DesignDesign DefaultDefault DeploymentDeployment

Multiple layers of defense neededMultiple layers of defense needed

Page 32: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Threat ModelingThreat Modeling

Analyze the design for vulnerabilityAnalyze the design for vulnerability Model data flowsModel data flows

SS - Spoofing Identity- Spoofing Identity TT - Tampering with Data- Tampering with Data RR - Repudiation- Repudiation II - Information Disclosure- Information Disclosure DD - Denial of Service- Denial of Service EE - Elevation of Privilege- Elevation of Privilege

Page 33: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 34: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

PerformancePerformance Deployment configurationDeployment configuration

Model performance as part of product Model performance as part of product definitiondefinition

Replicate environment in labReplicate environment in lab Test it as part of developmentTest it as part of development Fix it where it hurtsFix it where it hurts

Three-tiered problemThree-tiered problem SystemSystem ComponentsComponents CodeCode

Page 35: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

System and ComponentSystem and Component

Performance measures of test and Systems Under Test

Alerts and warnings on Systems Under Test

Page 36: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Code PerformanceCode Performance

Timeline of memory consumption

Suspect functions, drillable to code

Page 37: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 38: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

ManageabilityManageability

Operations documented and current Operations documented and current for every service or applicationfor every service or application

Service level agreement in placeService level agreement in place Security scanning in placeSecurity scanning in place Proactively monitor and fixProactively monitor and fix Reactive and proactive problem Reactive and proactive problem

managementmanagement

Page 39: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 40: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Testing Mission & ApproachTesting Mission & Approach Marick’s FrameworkMarick’s Framework

Different missions and approaches Different missions and approaches apply for each quadrantapply for each quadrant

Technology Technology FacingFacing

Business Business FacingFacing

Support Support ProgrammingProgrammingCritique Critique ProductProduct

http://www.testing.com/cgi-bin/blog/2003/08/21#agile-testing-project-1

Page 41: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Let the punishment fit the Let the punishment fit the crime!crime!

A good test approach is:A good test approach is: DiversifiedDiversified Risk-focusedRisk-focused Product-specificProduct-specific PracticalPractical DefensibleDefensible

Fit the technique and its data to its Fit the technique and its data to its purpose in the quadrantpurpose in the quadrant

Gilbert & Sullivan, The Mikado

Kaner, Bach & Pettichord, Lessons Learned in Software Testing, 2002

Page 42: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Testing Mission & ApproachTesting Mission & ApproachRepresentative techniquesRepresentative techniques

Technology Technology FacingFacing

Business Business FacingFacing

Support Support ProgrammingProgramming

Unit testing, code Unit testing, code coverage, code analysiscoverage, code analysis

Test-Driven DevelopmentTest-Driven Development

Granularity matches Granularity matches code code

Discrete scenariosDiscrete scenarios

Example-driven dataExample-driven data

Realistic 80% casesRealistic 80% cases

Prioritized regression Prioritized regression testingtesting

Critique Critique ProductProduct

Specialize by QoSSpecialize by QoS

Model-driven testsModel-driven tests

Generated dataGenerated data

Exploratory testingExploratory testing

Soap operasSoap operas

Page 43: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Test CoverageTest Coverage

Identify the Scenario, QoS or Code Identify the Scenario, QoS or Code that the test teststhat the test tests If they’re newly discovered, capture themIf they’re newly discovered, capture them If you can’t name them, question the If you can’t name them, question the

value of the testvalue of the test

Measure coverage against these Measure coverage against these dimensionsdimensions

Page 44: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Test Automation and Its Test Automation and Its DiscontentsDiscontents

Technology FacingTechnology Facing Business FacingBusiness Facing

Support Support ProgrammingProgramming Lowest costLowest cost

Critique ProductCritique Product Highest costHighest cost

ROI= ΣΣtt (Value of Information) - (Value of Information) - ΣΣtt (Cost to Maintain) (Cost to Maintain) ΣΣtt (Cost to Implement) (Cost to Implement)

(adjusted for net present value and risk)

Page 45: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Test Automation and Its Test Automation and Its DiscontentsDiscontents

Value depends on contextValue depends on context Automation is a programming exerciseAutomation is a programming exercise Opportunity cost high due to resource Opportunity cost high due to resource

constraintsconstraints Options theory problemOptions theory problem

Very sensitive to volatilityVery sensitive to volatility Often incalculableOften incalculable

ROI= ΣΣtt (Value of Information) - (Value of Information) - ΣΣtt (Cost to Maintain) (Cost to Maintain) ΣΣtt (Cost to Implement) (Cost to Implement)

Page 46: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Testing Web ApplicationsTesting Web Applications

View of content as rendered

Content validation

http request & response

Performance breakdown

Data substitution

Page 47: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 48: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

MetricsMetrics Consider many dimensions at onceConsider many dimensions at once

Single metrics easily misleadSingle metrics easily mislead Test resultsTest results Bug ratesBug rates Code churnCode churn Code coverageCode coverage Requirements coverageRequirements coverage

Never use metrics for reward or Never use metrics for reward or punishmentpunishment

Flow of value, not completion of tasksFlow of value, not completion of tasks Planned Planned andand unplanned work unplanned work

Robert Austin, Measuring and Managing Performance In Organizations, 1996

Page 49: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Which Component is Healthiest? Which Component is Healthiest?

Contrast two views of project dataContrast two views of project data

Fewest bugs Highest test pass rate

Page 50: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Which Component is Healthiest?Which Component is Healthiest? Conclusions:Conclusions:

Tests are staleTests are stale Highest risk hereHighest risk here

Lowest code coverage

Highest code churn

Page 51: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Focus on Flow of Focus on Flow of ValueValue

David J. Anderson, Managing with Cumulative Flow, 2004www.agilemanagement.net/Articles/Papers/BorConManagingwithCumulat.html

Cumulative Flow

020406080

100120140160180200220240

10-F

eb

17-F

eb

24-F

eb

2-Mar

9-Mar

16-M

ar

23-M

ar

30-M

ar

Time

Fea

ture

s

Backlog Started Designed Tested Complete

Control height of work in progress

Value measured on completion

Page 52: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 53: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Processes Differ for Good Processes Differ for Good Reasons…Reasons… EconomicsEconomics

RegulationRegulation LiabilityLiability

Plan-Driven vs. AdaptivePlan-Driven vs. Adaptive Iteration lengthIteration length Documentation requiredDocumentation required Sign-off gatesSign-off gates Time tracking requirementsTime tracking requirements

Page 54: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

InfrastructureInfrastructureArchitectArchitect

Project Project ManagerManager DeveloperDeveloper

TesterTesterSolutionSolutionArchitectArchitect

Business Business StakeholderStakeholder

……and for Bad Reasonsand for Bad Reasons……and for Bad Reasonsand for Bad Reasons

Page 55: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Solution is TransparencySolution is Transparency

InfrastructureInfrastructureArchitectArchitect

SolutionSolutionArchitectArchitect

Project ManagerProject Manager

DeveloperDeveloper

TesterTester

End UserEnd User

Page 56: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

TransparencyTransparency

Single product backlogSingle product backlog Task-aware versioningTask-aware versioning Project portalsProject portals Process handbookProcess handbook

Page 57: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Single Product BacklogSingle Product BacklogSingle backlog of all Work Items (Reqts, Tasks, Bugs, etc.)

Queries to filter, view, report

Details for each entry

Complete change history

Page 58: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Task-aware VersioningTask-aware Versioning

Source files to check in …

…with Work Items done…

…and Check-in Notes and Policy Status

Page 59: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Project PortalProject Portal

Page 60: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Process HandbookProcess Handbook

http://workspaces.gotdotnet.com/msfv4

Page 61: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 62: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

CultureCulture Productivity and predictabilityProductivity and predictability Responsibility over assignmentResponsibility over assignment Team and individualTeam and individual Product mentalityProduct mentality

Page 63: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

13 Symptoms of Unhappiness13 Symptoms of Unhappiness It’s the code, stupid!It’s the code, stupid! Actually it’s the requirements!Actually it’s the requirements! No, the problem is that you neglected the architecture!No, the problem is that you neglected the architecture! Architecture, schmarchitecture. I just want a working build.Architecture, schmarchitecture. I just want a working build. What good is that the way we mix up versions?!What good is that the way we mix up versions?! Not code versions, but the environments, don’t you get it?Not code versions, but the environments, don’t you get it? Ever heard of security?!Ever heard of security?! Yeah, but you ignored performance, duh!Yeah, but you ignored performance, duh! So what if it worked in the lab -- it’s still unmanageable!So what if it worked in the lab -- it’s still unmanageable! Oh, and did we mention testing?Oh, and did we mention testing? Since you’re not measuring it, you can’t manage it anyway!Since you’re not measuring it, you can’t manage it anyway! With a process like that, what do you expect?With a process like that, what do you expect? It’s our culture – you’ll never change that.It’s our culture – you’ll never change that.

Page 64: As Simple As Possible, But No Simpler Sam Guckenheimer  samgu@microsoft.com

Sam GuckenheimerSam Guckenheimerhttp://lab.msdn.microsoft.com/vs2005/teamsystem/http://lab.msdn.microsoft.com/vs2005/teamsystem/

[email protected]@microsoft.com