advanced software engineering: software...

Web: http://www.megadatasys.com Assisted with templates

Advanced Software Engineering: Software Testing

COMP 3705(L6)

Sada Narayanappa

Anneliese Andrews Thomas Thelin

Carina Andersson

Advanced Software Engineering 2

LectureChapter 9

– MetricsChapter 16 (& Appendix III)

– Process improvement and assessment– TMM, TIM

Presentation technique


Test management questionsWhat is effective? What is efficient?When to stop testing?


Why measure?Why is metrics important for the development and test process? Are metrics useful for motivation purposes?Can metrics improve quality?


Measurement basicsBasic data:

Time (calendar and staff hours)Failures / FaultsSize

Basic rule:Feedback to originUse data or don’t measure


Purpose of metricsProject monitoring – check the statusProject controlling – corrective actionsPlan new projectsMeasure and analyze results

– The profit of testing– The cost of testing– The quality of testing– The quality of the test process– Basis of improvement, not only for the test process


Monitoring testingStatus

– Coverage metrics– Test case metrics

Failure / Fault metrics– How much have we accomplished?– What is the quality status of the software?

Productivity metrics– How productive and efficient are the testers?

Efficiency / Cost metrics– How much time have we spent?– How much money have we spent?

Effectiveness metrics– How effective is the testing techniques in detecting defects?

Metrics

Estimation

Cost

Stop?


Test metrics: Size/complexity/lengthWhat?

Size – LOC, Function PointsComplexity – McCabeLength – Halstead

Why?Estimate test effort


Test metrics: EfficiencyWhat?

# faults/hour# faults/test case

Why?Evaluate efficiency of V&V activities


Test metrics: EffectivenessWhat?

% found faults per phase% missed faults

Why?Evaluate effectiveness of V&V activities


Test metrics: CoverageWhat?

% statements covered% branches covered% data flow

% requirements% equivalence classes

Why?Track completeness of test


Test metrics: Test execution statusWhat?

# faults/hour# executed testsRequirements coverage

Why?Track progress of test projectDecide stopping criteria


Test metrics: Trouble reportsWhat?

# faults/sizerepair timeroot cause

Why?Monitor qualityMonitor efficiencyImprove


Selecting the right metricsWhat is the purpose of the collected data?

– What kinds of questions can they answer?– Who will use the data?– How is the data used?

When and who needs the data?– Which forms and tools are used to collect the data?– Who will collect them?– Who will analyse the data?– Who have access to the data?


Goals– What is the organization trying to achieve? – The objective of process improvement is to satisfy these goals

Questions– Questions about areas of uncertainty related to the goals – You need process knowledge to derive the questions

Metrics– Measurements to be collected to answer the questions

Goal-Question-Metric Paradigm (GQM)


GQM guidelinesAnalyze <object (s) of study>

– the detection of design faults using inspection and testingfor the purpose of <purpose>

– evaluationWith respect to their <quality focus>

– effectiveness and efficiencyfrom the point of view of the <perspective>

– managers in the context of <context>

– developers, and in a real application domain


Test estimationGuessingAuthorized deadlinesPrevious test effort

– In-project experience through incremental planningFormula-based

– COCOMO– Test point analysis

Consensus of experts (Delphi)Detailed work breakdown of testing tasksHistorical data helps estimating regardless of the method


Test Point Analysis

Dynamic test points

Static test points

Total number of test points

Primary test hours

Environmental factors Skill factors

Planning and control

factorsTotal number of test hours

TPf=FPf*Df*Qd

PT=TP*S*E


Cost of testingHow much does testing cost?

As much as resources we have!


Costs of testingPre-run costsplanning, specification, environment setup

Execution costsexecution time (staff and equipment), regression tests

Post-run costsoutcome analysis, documentation


Minimizing costsTest performance

– Automate execution and evaluation

Test maintenance– Add new, review effectiveness

Testware development– Consider as part of the product


When to stop testing?

All coverage goals met (requirements, code, ...)Detection of specific number of failuresRates of failure detection fallen below a specified levelFault seeding ratios are favourableReliability above a certain valueNumber of failures left below the limitCost has reach the limit


Most common stop criteriaTime has reach the limit (milestone, release date, ...)All planned tests executed and passed


Example

Number of failures per day

Number ofexecutedtest cases

Number ofdetectedfailures

Interpretation?


Example

Interpretation?




Process models

Purpose– Improvement– Assessment

Models– (Integrated) Capability maturity model (CMM, CMMI)– Software process improvement and capability determination (SPICE)– ISO 9001, Bootstrap– Test maturity model (TMM)– Test process improvement model (TPI)– Test improvement model (TIM)


CMMP ro ces s chan ge m an ag em en tTechn ol o g y ch ang e m anag em en tDefec t p revent io n

S o ftware q ua l it y m an ag em en tQu ant it a t ive p ro ces s m anag em en t

P eer rev iewsInt e rg ro up coo rdin at io nS o ftware p ro du ct en gi neeri ngInt egra t ed so ftw are m anag em entTra i ni ng p rog ramm eOrg ani za ti on p roces s d e fin it io nOrg ani za ti on p roces s fo cus

S o ftw are co nfi gu ra t io n m an ag em entS o ftw are q ua l it y as su ranceS o ftw are s ub con trac t m anag em en tS o ftw are p ro jec t tracki ng an d o versigh tS o ftw are p ro jec t pl ann in gR equ irem en ts m an agem en t

I n iti al

Rep eatab l e

Defi n ed

Ma n ag ed

O p ti m i zi n g


Test Maturity Model (TMM)LevelsMaturity goals and sub-goals

– Scope, boundaries, accomplishments– Activities, tasks, responsibilities

Assessment model– Maturity goals– Assessment guidelines– Assessment procedure


Test Matu

rity

Model


Change model

VisionStrategiesProcesses

PlansOrganization

Quality system

Models

Ability

Application

Create

Utilize

Create

Utilize

ArchitectureDesign

SoftwareHardware

Product


Test improvement model – TIMKey areas

– Organization– Planning and monitoring– Test cases– Testware– Review / Inspection

Maturity levels– Initial – Basic– Cost effective– Risk thinking– Optimizing


Maturity profile (example)

0

1

2

3

4

Organization

Planning and monitoring

Test cases

Testware

Review / Inspection


Phases of assessment / improvement

TMM– Plan (Preparation)– Do (conducting, reporting)– Check (analyzing, action

planning)– Act (implementing

improvement)

TIM1. Evaluation – mirror of the test

organization2. Present, discuss and reflect –

understand the problems3. Develop improvement strategy 4. Implement the improvement

strategy 5. Continually improvement


EvaluationInterview based on a questionnaire Covers the key areas and maturity levels“Magic” questions:

– What would you like to change? – What would make your situation better?


Present, discuss and reflectWrite a report and presentAll questions are discussedGoal: create a common understanding


Develop improvement strategyPrioritized areas are identifiedImprovement puzzleSupport from the organization is very important


Implement the improvement strategySeminars and workshopsPilot project – internal mentorsRole play


General adviceIdentify the real problems before starting an improvement program“What the customer wants is not always what it needs”Implement “easy changes” firstInvolve peopleChanges take time!

advanced software engineering: software...

Documents