copyright © 2003 software quality research laboratory software production essentials seeing past...

Copyright © 2003 Software Quality Research LaboratoryCopyright © 2003 Software Quality Research Laboratory

Software Production Software Production EssentialsEssentials

Seeing Past the Buzz WordsSeeing Past the Buzz Words


Why Process?Why Process? QualityQuality

Maximize Customer SatisfactionMaximize Customer Satisfaction Minimize Rework and RepairMinimize Rework and Repair

ProductivityProductivity Optimize Production CostOptimize Production Cost Shorten Time to MarketShorten Time to Market

This is not a tradeoff. Quality is This is not a tradeoff. Quality is Free.Free.


Cost of Correcting an ErrorCost of Correcting an Error

1X in Requirements Analysis1X in Requirements Analysis 5X in Specification and Design5X in Specification and Design

10X in Coding10X in Coding 30X in Integration30X in Integration

70X in Acceptance Test70X in Acceptance Test 300X in Field300X in Field

Typical Range: $50 - $14,000Typical Range: $50 - $14,000


Basic Process ElementsBasic Process Elements

Requirements DefinitionRequirements Definition

CodingCoding

ReleaseRelease

Everybody does it, but the order may vary.


How Do We Fill the Gaps?How Do We Fill the Gaps?

Requirements DefinitionRequirements Definition Specification?/Design?/Verification?/?Specification?/Design?/Verification?/? CodingCoding Test?/ Inspection?/?Test?/ Inspection?/? ReleaseRelease

No shortage of lifecycle/process candidates.


Software Production is Software Production is IncrementalIncremental

An efficient, repeatable process is necessary for expanding, changing requirements.

?

?

?


Software Production KeysSoftware Production Keys

Architecture

ArchitectureDefinition

Requirements

Code

Implementation

Specification

Test Planning

Test Cases

Certification

ReleaseDecision

Code Inspection

Peer Review

Peer Review

Peer Review

(1) A seamless, traceable transformation from requirements to code specification.

(2) Mapping specified code to a robust, maintainable architecture.

(4) Quantitative reliability assessment via model-based statistical testing.

(3) Peer review of key work products including 100% code inspection.


Specification Process OverviewSpecification Process Overview

SpecificationFunctions

CanonicalSequences

Black BoxSpec

State DataTables

State BoxSpec

StimulusGathering

ResponseGeneration

State DataState

Transitions

ResponsesStimuli

Enumeration

RequirementsInitial Requirements

Capture

Systematic BehaviorDiscovery byEnumeration

Black Box and StateBox Transformations

Code Definition


Rigorous Specification ProcessRigorous Specification Process

Establish Establish system boundarysystem boundary.. Define human/software/hardware Define human/software/hardware interfacesinterfaces.. Itemize Itemize stimulistimuli.. Itemize Itemize responsesresponses.. Perform Perform enumerationenumeration.. Perform Perform canonical sequence analysiscanonical sequence analysis.. Generate Generate state machine specificationstate machine specification.. Transform to Transform to formal notationformal notation (e.g., ACL2). (e.g., ACL2).


Map Specified Code to Map Specified Code to ArchitectureArchitecture

Architecture

+Method1()+Method2()+Method3()

-data1-data2-data3

Exec

+set1()+set2()

-Resp1-Resp2

Display

+set1()+set2()

-Resp1-Resp2

Control

+set1()+set2()

-Resp1-Resp2

Console

+get1()+get2()

-Stim1-Stim2

Sensor

-End1

1

-End2

1

-End3

1..*

-End4

1

-End5

1

-End6

1-End7

1 -End8

1..*

StimulusGathering

ResponseGeneration

State MachineData/Transitions


Independent ReviewIndependent Review Domain Expert ReviewDomain Expert Review

Initial RequirementsInitial Requirements Black Box SpecificationBlack Box Specification Usage ModelUsage Model

Development Team Peer ReviewDevelopment Team Peer Review Architecture SpecificationArchitecture Specification Test PlanTest Plan

Code InspectionCode Inspection Automated Enforcement of Coding StandardsAutomated Enforcement of Coding Standards Manual Verification of Functional CorrectnessManual Verification of Functional Correctness


Testing is Always SamplingTesting is Always Sampling

Population(All Uses)

Sample(Tests)

What to test: a statistically appropriate

sample

How much to test: to an acceptable

reliability estimate


Statistical Usage TestingStatistical Usage Testing

Statistical Usage TestingStatistical Usage Testing involvesinvolves Modeling the population of usesModeling the population of uses Selecting a sample for testingSelecting a sample for testing Estimating field performance from Estimating field performance from

test sample resultstest sample results Making a release decisionMaking a release decision


Model-Based Statistical TestingModel-Based Statistical TestingUsage Modeling

andTest Planning

Test Case Generationand Automation

Support

Results Managementand Quantitative

Analysis

Build Model

Analyze/ReviseModel

Generate TestCases

Analyze/ReviseTest Suite

Execute Tests

Release Decision

ReliabilityAnalysis

Retest

Release

TML Files

HTMLReport

Test Cases/Scripts

HTMLReport

TestResults

ReleaseCriteria

HTMLReport


Usage ModelUsage Model: All Possible : All Possible Uses and their LikelihoodUses and their Likelihood

Begin

Beta

EndStart

A

B

Quit

Quit

STATE MACHINENodes are states-of-useArcs are possible stimuliProbabilities define expected usageTest case is path from initial to terminal state

AB

B

A

p=1p=.9

p=.1

p=.1p=.1

p=.01

p=.05

p=.85

Idle

Alpha

p=.89


A Usage Model is A Usage Model is a Finite-State a Finite-State Markov ChainMarkov Chain

Well-understood formalismWell-understood formalism Rich body of analytical resultsRich body of analytical results Engineering basis for testingEngineering basis for testing Objectivity in test planning and Objectivity in test planning and

managementmanagement Simulates “use” of product and not Simulates “use” of product and not

the product itselfthe product itself


Statistical Testing based on a Statistical Testing based on a Usage Model affords optionsUsage Model affords options

Coverage testing…all states and arcsCoverage testing…all states and arcs Importance testing…most probable pathsImportance testing…most probable paths Random testing…representative usageRandom testing…representative usage Testing to reliability goal…0.99, 0.999, …Testing to reliability goal…0.99, 0.999, … Testing to level of confidence…95%, 99%…Testing to level of confidence…95%, 99%… Testing to budget with measured reliabilityTesting to budget with measured reliability


Test AutomationTest Automation

Partition andSample Input Space

UsageModel

GenerateTest

Cases

TestExecution

CompareResults to

Oracle

Test Record

Export TestScripts

Record Pass/Fail

AnalyzeResults

Result Checkingand Comparison

Parametric Data Generation forAbstract Stimuli

Test Case Generation

Release?


Results Analysis / Release DecisionResults Analysis / Release Decision

Record all failures by test case number Record all failures by test case number and transition where failure occurs, then:and transition where failure occurs, then:

Compute Compute estimated reliabilityestimated reliability based on based on testing experiencetesting experience

Evaluate Evaluate stopping criteriastopping criteria ReleaseRelease or or fix and retestfix and retest based on based on

semi-quantitative risk assessment semi-quantitative risk assessment


Benefits of Benefits of Model Based TestingModel Based Testing

Better ProductBetter Product Clearer requirements, improved specificationClearer requirements, improved specification

Better Use of ResourcesBetter Use of Resources Quantitative optimization of testing strategyQuantitative optimization of testing strategy Reusable assets: model, test framework, oracleReusable assets: model, test framework, oracle

Shorter Life CycleShorter Life Cycle Test planning in parallel with developmentTest planning in parallel with development Easier test automationEasier test automation

Better ManagementBetter Management Quantitative support for management decisionsQuantitative support for management decisions Quantification of expected field reliabilityQuantification of expected field reliability Clear path to continuous process improvementClear path to continuous process improvement


Essential Process ElementsEssential Process Elements Requirements DefinitionRequirements Definition Robust Architecture DefinitionRobust Architecture Definition Sequence-Based SpecificationSequence-Based Specification Domain Expert and Peer ReviewDomain Expert and Peer Review CodingCoding 100% Code Inspection100% Code Inspection Model-Based Testing/CertificationModel-Based Testing/Certification IncrementalIncremental Release Release

Fill the gaps with rigorous engineering Fill the gaps with rigorous engineering practices.practices.

copyright © 2003 software quality research laboratory software production essentials seeing past...

Documents

quality quality

code specification

architecture slide

coding coding test

requirements analysis

state machine specification

production cost

acceptance test