value flow science - fitter lifecycles from lean balanced scorecards (2011)

Value Flow Science - Fitter lifecycles from lean balanced scorecards

Value Flow Science fitter lifecycles from lean balanced scorecards Neil ThompsonThompson information Systems Consulting Ltd(material co-developed with Mike Smith, Learntesting / Testing Solutions Group)Thompson information Systems Consulting Ltd1Func Spec Reviewers

Sys Test Analysts

Tech Design Reviewers

Int Test Analysts

v1.2

UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Quality is value to some person(s)2

Summit image from www.topnews.in

Quality is value to meYes, but...Jerry Weinberg, Quality Software Management 1992Quality is value to meQuality is value to meQuality is value to meQuality is value to meQuality is value to meQuality is value to me

Thompson information Systems Consulting Ltd

UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 To balance quality: the Value Flow ScoreCard3Objectives

Measures

Targets

Initiatives

FinancialSupplierCustomerImprovement &InfrastructureProcessProductWHY wedo things

WHAT(will constitute success)

HOW todo thingswell

SIX VIEWPOINTS of what stakeholders want

Its a simple table which we can use to help control our work: do things well enough for an appropriate balance of stakeholders in this presentation, test policy/strategy, test coverage, process improvement and process definition (but arguably can apply it to anything!)


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 The software lifecycle as a flow of value4Working systems have value; documents in themselves do not; so this is thequickestroute!Programming

abcDemonstrations &acceptance testsStatedrequirementsRAW MATERIALS FINISHEDPRODUCTSDLCs are necessary, but introduce impediments to value flow: misunderstandings, disagreementsdocuments are like inventory/stock, or wasteProgramming

aabbcd

DocumentedrequirementsImplicitrequirementsMeeting / escalation to agreeII

Acceptance tests

??

Intermediate documentation!


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 To improve value flow: agile methods following principles of lean manufacturing5

LEVELS OF DOCUMENTATION,pushed by specifiers

WORKINGSOFTWAREAcceptedSystem-testedIntegratedUnit / Component-tested

FLOW OF FULLY-WORKING SOFTWARE, pulled bycustomer demandRequirements

+ FuncSpec+ TechnicalDesign+ Unit / Componentspecifications+ Test Specifications


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 But can apply principles to any lifecycle...6This material has been jointly developed with Mike Smith, who authored:white papers on test process (1999 & 2002)keynote presentation to Ericsson on measurement in testing (2007)Neil Thompsons earlier works:Organisation before automation (EuroSTAR 1993, multidimensionality of test coverage)Goldratts Theory of Constraints & Systems Thinking in process definition (STAREast 2003), SDLC (EuroSP3 2004), and process improvement (EuroSTAR 2006) SIGiST 2008: Value Flow ScoreCards for betterstrategies, coverage and processesHolistic Test Analysis & Design, STARWest 2007a flexible tabular format used for test coveragerelating this to Balanced ScoreCards (Kaplan & Norton, business strategy etc)

Separating what from how, ICST 2008:Test Conditions as the keystone test entity

Both of us: participation in the Software Testing Retreat Test entities and Appropriate Testing (ApT)

Next Generation Testing 2009: Balancing SDLC Value Flow


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 (a) Taking Balanced ScoreCard beyond strategy7

www.balancedscorecard.org Paul Arveson 1998version afterKaplan & Norton

Product

Risks

- Faults- Failures(Financial)valueEfficiencyProductivityOn-time, in budget

Customer(User)BenefitsAcceptanceSatisfaction- ComplaintsImprovementeg TPI/TMMPredictabilityLearningInnovationProcess(Manufactg)Complianceeg ISO9000Repeatability

- MistakesSoftware Quality versionpublished by Isabel Evanswww.testing-solutions.com,adapted here by Neil Thompson

WHYWHATHOW

ObjectivesMeasuresTargetsInitiativesWe can apply these(complementary)views of qualityto testingAlso acknowledging Philip B. Crosby, David A. Garvin, Robert M. Pirsig...


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 (b) Treating each part of lifecycle as a link of value chain88

FinancialEfficiencyProductivityOn-time, in budget- Cost of quality

SupplierUpwardmanagement

InformationgatheringValue chain Supply chain:in the IS SDLC, each participant should try to manage their supplierfor example, development supplies testing(in trad lifecycles, at least!)we add supplier viewpoint to the other 5, giving a 6th view of qualityNow each step in the value chain can manage its inputs, outputs and other stakeholders

CustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- ComplaintsImprove-menteg TPI/TMMPredictabilityLearningInnovationProcessComplianceeg ISO9000Repeatability

- Mistakes

ProductVERIFICATIONRisksTest coverage

- Faults- Failures

Six-Sigma Business ScoreCard published by Praveen Gupta (2nd ed. McGraw Hill 2007), butthis slide shows Neil Thompsons version


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 (Intermediate test levels)A sample link in value chain: test analysis for Acceptance Testing9Programming

DocumentedrequirementsImplicitrequirementsAcceptance testing

(Intermediate documentation levels)RAW MATERIALS FINISHEDPRODUCT

TESTDESIGNTESTANALYSISSTATICTESTING

TESTEXECUTION

Requir-ements reviews

Financial

Customer

Supplier

ImprovtProcess

Product

(Main Supplier isstatic testing)(Main Customer istest design)


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Scorecards may be cascaded through the lifecycle, so they flow four ways10

Financial

Customer

Supplier

ImprovtProcess

ProductTESTANALYSISi

iiOrganisation & Project Objectives from TEST POLICY, STRATEGY, PROJECT TEST PLANCoverage Objectivesfrom REQUIREMENTS,FUNCTIONAL SPEC, TECHNICAL DESIGN,MODULE SPECS andreviews of them

iiiTest Analystsown Objectives

ivFeedback objectivesfor Business Analysts,Architects & Developers

Financial

Customer

Supplier

ImprovtProcess

ProductTESTDESIGN

Test Designersown Objectives Initiativesfor nextstage oftest processInitiativesfor nextlevel down theSDLC

(etc)(etc)


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Potential number of ScoreCards depends on how your SDLC is handled by different roles11

Pieces of a jig-saw!

This example is a full-ish set:higher-level tests are scripted other staffmay then executeBut you dont necessarily need all theseBusiness Analysts

Requirements Reviewers

Architects

Acceptance Test Analysts

Func Spec Reviewers

Designers

Acceptance Testers

Sys Testers

Sys Test Analysts

Tech Design Reviewers

Developers

Int Test Analysts

via pair programming?Component Test Analysts, Designers & Executers?

AT Designers & Scripters

ST Designers & Scripters

IT Designers, Scripters & Executers


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 The Value Flow ScoreCard in action12

Financial

Customer

Supplier

ImprovtProcess

Product

Financial

Customer

Supplier

Improvt

Process

ProductYes its just a table! Into which we can put useful thingsWe start with repositionablepaper notes, then canput in spreadsheet(s)


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Use #1: alignment of testing with organisation13Organisations objectives

Organisations Test Policy

Project Test StrategyHOWHOWWHATHOWWHATHOWWHAT

WHAT

A top-down view:down the business / organisation

WHYWHYWHY(etc)(etc)


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Use #1, eg Test Policy14

FinancialEfficiencyProductivityOn-time, in budget

- Cost of qualitySupplierUpwardmanagement

InformationgatheringCustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- ComplaintsImprovement & Infrastructure

eg TPI/TMMPredictabilityLearningInnovationProcessComplianceeg ISO9000Repeatability

- MistakesProductVERIFICATIONRisksTest coverage

- Faults- FailuresOrganisationsGoals & ObjectivesObjectives

Measures

Targets

InitiativesOrganisations ScoreCards

TMM levels TMM level 2 at least, now

TMM level 3 within 2 years

Constant improvement of development & test processes

(for Test Strategy / Strategies)

and fordevelopment?


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Example 1, Test Policy: All views included? Why-What-How (G-Q-M) thought through?15



InformationgatheringCustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- ComplaintsImprovement & Infrastructure

eg TPI/TMMPredictabilityLearningInnovationProcessComplianceeg ISO9000Repeatability


- Faults- FailuresOrganisationsGoals & ObjectivesObjectives

Measures

Targets

InitiativesOrganisations ScoreCards

IS actively supports employees TMM levels TMM level 2 at least, now Both static & dynamic Planning, preparation & evaluation Software & related work products Products to satisfy specified requirements Products to be fit for purpose Detect defects early Indep- endence increases with test type Testing prioritised & managed Product risks Importance of reqts

Automate regr tests as much as possibleSource: summarised from an example in TestGrip by Marselis, van Royen, Schotanus & Pinkster (CMG, 2007) Use TestFrame for test analysis & execution Defect Detection Percentage Freq of process adjustments heeding metrics Twice per year Proj Mgr is responsible for quality Bus Mgt is responsible for enforcing Test Policy

TMM level 3 within 2 years Staff must be certified Defect source analysis

(comprehensive scope)

ISTQB Advisors Expert Managers Advanced Analysts Foundation Constant improvt of dev & test processes

GOAL

QUESTION

METRICWhy

What

How


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Use #2: testing as measurement within the lifecycle16A left-to-right view (across a W-model)WHYHOWWHATHOWWHAT

TEST BASIS (EGSYSTEM SPEC)TESTDESIGNTESTEXECUTIONTESTANALYSIS

WHYSystem development starts withthe logical (what) before specifyingthe physical (how), solets do this for testing also!(etc)(etc)

NB its not only for waterfall or V-model SDLCs, rather iterative / incremental go down & up through layers of stakeholders, specifications & system integrations


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Use #2, eg test coverage:Test conditions as measures & targets (not test cases!)17Test Items(level of integration)Features to betestedTest BasisReferencesProductRisksAreas wecould coverObjectives

Measures

Targets

Initiatives


- Cost of qualitySupplierUpwardmanagementInfo from otherlevels of Treble-V modelCustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- ComplaintsImprovement &Infrastructureeg TPI/TMMPredictabilityLearningInnovationProcessComplianceeg ISO9000Repeatability


- Faults- Failures(from LEVEL TEST PLAN

andTEST BASES)(to test design & execution)

Test Conditionswe intend to cover

Productbenefits

Features to betestedProductRisks

ConstraintsProductRisksProductRisksProductRisks

Objectives forTest Cases

Agreedwithstakeholders

(to next levelof sys design)


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Use #3: process improvement, eg using Goldratts Theory of Constraints:Swimlane symptoms, causes & proposed remedies18Note: this is similar to Kaplan & Nortons Strategy Maps (Harvard Business School Press 2004)



InformationgethringCustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- ComplaintsImprovement &Infrastructureeg TPI/TMMPredictabilityLearningInnovationProcessComplianceeg ISO9000Repeatability


- Faults- FailuresObjectives

Measures

Targets

Initiatives

PRE-REQUISITESTRANSITIONCONFLICTRESOLUTIONCURRENT ILLSFUTURE REMEDIES


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Use #4A: context-driven testing, egGoldratt conflict resolution on process areas with choices19Objectives

Measures

Targets

Initiatives



InformationgatheringCustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- ComplaintsImprovement &Infrastructureeg TPI/TMMPredictabilityLearningInnovationProcessComplianceeg ISO9000Repeatability


- Faults- Failures

FromContext / CircumstancesResources: money ( skills, environments) timeApplication characteristicsProcess constraints, eg: quality mgmt configuration mgmtTechnology

Technical risksBusiness risks

Legal: regulation standards

CURRENT SITUATION Test specifications Handover & acceptancecriteria

informalformal

informalformal

Appropriate Testing in this context / circumstancesCONFLICTRESOLUTION

etc (about 30categories)

DESIRED SITUATION

Where in the range(specific aspects)

Where in the range(specific aspects)

SectorCulture

Job type & sizeMoral: safety

CHOICE AREAS...


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Use #4B: lifecycle methodology selection / design, Value Flow ScoreCard as unifying framework20



InformationgatheringCustomerVALIDATIONRisksBenefitsAcceptanceSatisfaction- ComplaintsImprovement &Infrastructureeg TPI/TMMPredictabilityLearningInnovationProcessComplianceeg ISO9000Repeatability


- Faults- FailuresObjectives

Measures

Targets

Initiatives

Conflicts & balances

One handThe other

RisksRisksRisksRisksRisksRisks

Game Theory.................................................................................................................................................(any other approaches?)...........................................

Appropriate lifecycle methodology in this context / circumstances

METHODOLOGY PER PROJECTBALANCE


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Discussion (1)21 Examples from attenders experience & current situations?


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Discussion (2)22 How the concept extends to Value Flow Science... further than might initially be obvious!(SOME SPECULATIVE SLIDES FOLLOW)


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Testing has a hierarchy, eg...23

AcceptanceTestingSystemTestingIntegrationTesting

Levels of specificationIndividual units may malfunction (so seek bugs of type x)Units may not interact properly (so seek bugs of type y)Users maybe unhappy (sogenerate confidence)System may contain bugs not found bylower levels (so seek bugs of type z)RequirementsFunctional &NF specifica-tionsTechnicalspec, Hi-leveldesignDetaileddesigns

...spec may not be adhered to

Risks &testingresponsibs

Remember: not only for waterfall or V-model SDLCs, rather iterative / incremental go down & up through layers of stakeholders, specifications & system integrations

UnitTestingLevels of system &service integrationLevels of stakeholdersBusiness, Users,Business Analysts,Acceptance TestersArchitects,independent testersDesigners,integration testers

+ BusinessprocessesDevelopers,unit testers


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 ...Quality and Science can also be seen as hierarchies, which testing can parallel24

Levels of system &service integrationLevels of stakeholdersBusiness, Users,Business Analysts,Acceptance TestersArchitects,independent testersDesigners,integration testers

+ BusinessprocessesDevelopers,unit testersLayers of quality

Layers of science

Static values: InorganicDynamic values?

...............Scale...............

Biological Social IntellectualPhysicsChemistry: InorganicChemistry: OrganicBiology (& systemsthinking)Social sciencesPhilosophy


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Value flows down through, then up through, these layersLevels of system &service integrationLevels of stakeholdersBusiness, Users,Business Analysts,Acceptance TestersArchitects,independent testersDesigners,integration testers

+ BusinessprocessesDevelopers,unit testers

Understanding of problem...

Understanding of solution...

Desired qualityTested (known) quality


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Test appropriately to your scale26(Ouroboros: Greek or , from "tail-devouring snake)

Physics (gravityend)Physics(quantumend)Systemsthinking)Understanding of solutionSocial sciences

Chemistry: Inorganic

OrganicBiology

Image from http://www.aaas.org/spp/dser/03_Areas/cosmos/perspectives/Essay_Primack_SNAKE.GIF


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Value flow must be maintained through iterations27

2nd law of thermodynamics!EvolutionUnderstanding of solution


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Discussion (3)28 How this might help bridge the (arguably) growing divide between traditional & agile software practitioners, eg: - waterfall, V-model, W-model, iterative, incremental... - schools of software testing, eg Analytic, Standard, Quality, Context-Driven, Agile... Factory, Oblivious...? - scripted (or at least pre-designed) & exploratory testing


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Four, five, six... schools of software testing?29

(Updated version) March 2007Copyright 2003-2007 Bret Pettichord. Permission to reproduce granted with attributionEmphasis onadapting to the circumstances under which the product is developed and usedAnnotations by Neil Thompson after the Bret Pettichord ppt (blue text), the list in Cem Kaners blog December 2006 (black text), and other sources! (red text)Emphasis on analytical methods for assessing the quality of the software, including improvement of testability byimproved precision of specifications and many types of modeling(Control):Emphasis on standards and processes that enforce or rely heavily on standards(Test-Driven):emphasis on code-focused testing by programmersOblivious /Groucho?Holistic?(like C-D)Emphasis on policing developers and acting as gatekeeper

Axiomatic?Factory: Emphasis on reduction of testing tasks to routines that can be automated or delegated to cheap labour


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Appendix A: Amendment history30The version presented at Test Management Summit (11:45, Wed 26 Jan 2011, Trafalgar 2 room, Institute of Directors, 116 Pall Mall, London, UK) was v1.1 this version 1.2 contains amendments since: slide 26 attribution added slide 29 key clarified appendices added: this Amendment history, plus the References which follow


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 Appendix B: References & acknowledgements3131

ScoreCards:Kaplan & Norton; Bournes; Evans; GuptaLean & agile:Toyota; Poppendiecks; Alistair Cockburn; David Anderson

Goldratt: Jens Pas; William Dettmer ; , Greg DaichSystems Thinking: Jerry Weinberg; Dennis Sherwood

Appropriate Testing (ApT) & test entities:the Software Testing Retreat Balance.........................

Colour-codedsources oftesting ideas:- Rob Sabourin

Good Enough Quality (+BSC):- James Bach, + Kaner & Pettichord

Factors affectingchoices:- Matt Archer


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 References (further details)32Kaplan, RS & DP Norton: The Balanced Scorecard Translating Strategy into Action (Harvard Business School Press 1996)...Also Strategy Maps (Harvard Business School Press 2004)Bourne, M & P Bourne: Balanced Scorecard in a Week (Chartered Management Insititute via Hodder Arnold 2002)Kaner, C, J Bach & B Pettichord: Lessons Learned in Software Testing a Context-Driven Approach (Wiley 2002)Evans, I: Achieving Software Quality through Teamwork (Artech House 2004) Gupta, P: Six Sigma Business Scorecard Creating a Comprehensive Corporate Performance Measurement System (2nd ed, McGraw Hill 2007)Liker, JK: The Toyota Way 14 Management Principles from the company that invented Lean Production (McGraw Hill 2004)Poppendieck, M & T Poppendieck: Lean Software Development an Agile Toolkit (Addison Wesley 2003)(also same authors, Implementing Lean Software Development from Concept to Cash (Addison Wesley 2007)Cockburn, A: Agile Software Development (Addison Wesley 2002)Anderson, DJ: Agile Management for Software Engineering Applying the Theory of Constraints for Business Results (Prentice Hall 2004)Boehm, B & R Turner: Balancing Agility and Discipline (Addison Wesley 2005)


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 References (further details contd)3333Dettmer, HW: Goldratts Theory of Constraints a Systems Approach to Continuous Improvement (ASQ 1997)Daich, G: Software Documentation Superstitions (STAREast 2002, http://www.stickyminds.com/s.asp?F=S5857_CP_2 )Weinberg, GM: Quality Software Management Vol 1 Systems Thinking (Dorset House 1992)Sherwood, D: Seeing the Forest for the Trees a Managers Guide to Applying Systems Thinking (Nicholas Brealey, 2002)Guckenheimer, S. with Perez, JJ: Software engineering with Microsoft Visual Studio Team System (Addison-Wesley Pearson Education 2006)Thompson, N & M Smith: Holistic Test Analysis and Design (STARWest 2007, http://www.stickyminds.com/s.asp?F=S13265_CP_2 )Smith, M & N Thompson: The Keystone to Support a Generic Test Process Separating the What from the How (ICST 2008, http://www2.computer.org/portal/web/csdl/doi/10.1109/ICSTW.2008.46 )Thompson, N & M Smith: Value Flow ScoreCards for better strategies, coverage & processes (BCS SIGiST 2008, www.bcs.org/upload/pdf/value-flow-scorecards.pdf Sabourin, R: Just in Time Testing (course, eg via http://www.testing-solutions.com/library/downloads/e/Enabling%20Test%20Maximisation%20-%20Rob%20Sabourin.pdf )Archer, M: A Thinking Framework for Context-Driven Test Documentation (SQC conference London 2008, http://www.ivarjacobson.com/resources/module/publications_action-download/file_id-190/


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 References: science additions34

Cosmic Ouroboros:Glashow, SL: Physics Newest Frontiers (interview sketch reproduced by Ferris, T: New York Times Magazine, 1982)Rees, M: Just Six Numbers the Deep Forces that Shape the Universe (Weidenfeld & Nicolson 1999 - also see lecture http://www.youtube.com/watch?v=NLc6NEuec3E )Primack, JR & Abrams, NE: The View From the Center of the Universe: Discovering our Extraordinary Place in the Cosmos (Riverhead 2006)Evolution:Dennett, DC: Darwins Dangerous Idea Evolution & the Meanings of Life (Simon & Schuster 1995)Smolin, L: The Life of the Cosmos (Oxford University Press 1997)

Individual sciences (the quote-marks are deliberately tongue-in-cheek):Mathematical foundations Penrose, R: The Road to Reality a Complete Guide to the Laws of the Universe (Jonathan Cape 2004)Physics Smolin, L: The Trouble with Physics the Rise of String Theory, the Fall of a Science and What Comes Next (Houghton Mifflin 2006)Chemistry [vacant, candidates sought]Biology Dawkins, R: The Extended Phenotype the Long Reach of the Gene (Oxford University Press 1982)(jumping to software for now) Kaner, C: Software Testing as a Social Science(STEP 2008, www.kaner.com/pdfs/KanerSocialScienceSTEP.pdf )


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011 References: value additions35Pirsig, RM:Zen and the Art of Motorcycle Maintenance an Inquiry into Values (Bodley Head 1974, also see http://en.wikipedia.org/wiki/Zen_and_the_Art_of_Motorcycle_Maintenance )

Lila an Inquiry into Morals yes, I know about Sam Harris, Ill get to that in due course (Bantam 1991, also see http://en.wikipedia.org/wiki/Lila:_An_Inquiry_into_Morals )


UKFIFTHTEST MANAGEMENT SUMMIT 26 Jan 2011