advanced software engineering: software testing · advanced software engineering 16 history of...

Web: http://www.megadatasys.com Assisted with templates

Advanced Software Engineering: Software Testing

COMP 3705(L5)

Sada Narayanappa

Anneliese Andrews Thomas Thelin

Carina Andersson

Advanced Software Engineering 2

News/Project/PapersQuestions:

Advanced Software Engineering 3

LectureChapter 8

– OrganizationChapter 14

– Tools– Test automation

Advanced Software Engineering 4

Organization• Support for decision making• Enhance teamwork• Independency• Balance testing – quality• Assist test management• Ownership of test technology• Resources utilization• Career path

Advanced Software Engineering 5

Test OrganisationDegrees of freedom

– Tall or flat

– Market or product

– Centralized or decentralized

– Hierarchical or diffused

– Line or staff

– Functional or project

Advanced Software Engineering 6

7 approaches to test organisation1. Each person’s responsibility2. Each unit’s responsibility3. Dedicated resource4. Test organisation in QA5. Test organisation in development6. Centralized test organisation7. Test technology centre

ExplanationBenefitsDrawbacks( Example )( Change )

Advanced Software Engineering 7

1. Each person’s responsibility

Product developers

M

P P P P P

PT

Advanced Software Engineering 8

2. Each unit’s responsibility

Product developers

M

P P P P P

PT

PT

Advanced Software Engineering 9

3a. Dedicated resource

Product developers

M

P P P P P

P T

Advanced Software Engineering 10

3b. Dedicated resource on a large scale

Product developers

M

P P P P P

Product developers

M

P P P P P

Test developers

TM

T T T T T

Advanced Software Engineering 11

4. Test organisation in QA

Test Development

QAO

Group

Product Development

PDO

Group

Advanced Software Engineering 12

5. Test organisation in development

Test Development

Group

Product Development

PDO

Group

Advanced Software Engineering 13

6. Centralized test organisation

Test Development

Group

Product Development

VP

Group

Advanced Software Engineering 14

7. Test technology centre

Test Development

Group

Product Development

VP

Group

Test Technology

Group

SE

Advanced Software Engineering 15

What criteria to consider?

Which organisation should we choose?

Size

Maturity

Focus

Advanced Software Engineering 16

History of software testing

Defect prevention & Test

process1950 1960 1970 1980 1990 2000

Fix bugs

Exhaustive

testing

Prove it worksProve it does

not work

Early test designInternet (Agile)

Test automation

tools

Advanced test

automation

Automated business

optimization

Advanced Software Engineering 17

ToolsGood at repeating tasks

Good at organising data

Requires training

Introduced incrementally

No “silver bullet”

Ease of usePowerRobustnessFunctionalityEase of insertionQuality of supportCostCompany policies and goals

Advanced Software Engineering 18

Test tools

Test management tools

Testexecution and comparison tools

Dynamic analysis tools

Debuggingtools

Coverage toolsStatic analysis tools

Test design tools

Architectural design

Detailed design

Code

Requirement specification

Unit test

Integration test

Performancesimulator tools

System test

Acceptance test

Advanced Software Engineering 19

Tools categories by activityReviews and inspections

Test planning

Test design and development

Test execution and evaluation

Testing support

Test DesignTools

Static Analysis

Tools

Execution:

•Unit•Integration•System•Acceptance

Advanced Software Engineering 20

Reviews and inspectionsComplexity analysis

– Identify problem areas

Code comprehension– Show different views of the

artefact

Syntax and semantic analysis– Check and warn

Test DesignTools

Static Analysis

Tools

Execution:

•Unit•Integration•System•Acceptance

Advanced Software Engineering 21

Test planningTemplates for test plan documentation

Test schedule and staffing estimates

Complexity analyser

To large extent general project management tools

Advanced Software Engineering 22

Test design and developmentTest data generator

Requirements-based test design tool

Capture/replay

Coverage analysis

Often integrated with test execution tools

Advanced Software Engineering 23

Test execution and evaluationCapture/replay

Coverage analysis

Memory testing

Test case management

Simulators and performance

Advanced Software Engineering 24

Capture/replay

Static c/r tools (without scripting language)

Static c/r tools (with scripting language)

Variable c/r tools

Advanced Software Engineering 25

Coverage analysisMonitoring execution

Pre-processing code

Problem: intrusive, in particular for real-time systems if-then-else

while-loop

case-of

Advanced Software Engineering 26

Memory testingLeaks (allocate but not free memory)

Array bounds

Initialization of variables

Easy to use, low cost

Advanced Software Engineering 27

Simulators and performanceHardware emulators

Environment simulators

Load generators

PC target HWHW emul

SWSW

target HWSWsimulator simulator

target HWSWgenerator

Advanced Software Engineering 28

Test case managementKeep track of test ware

Manage test execution

Seamless integration with other tools

Automated test reporting

Not yet available :-(

Advanced Software Engineering 29

Testing support toolsTrouble reporting

– Report, Dispatch, Follow-up

Configuration management– Manage, control and coordinate changes

Advanced Software Engineering 30

Test automationEffectiveness – independent of automationEfficiency – can be improved by automation

Automation takes 2-10(-30) times the time for first run!

Advanced Software Engineering 31

Different levels of automation

Select/identify test cases to runSet up test environment

Create test environmentLoad test dataRepeat for each test caseSet up test prerequisitesExecuteCompare resultsLog resultsAnalyze test failuresReport defectsClear up after test case

Clear up test environmentDelete unwanted test dataSave important data

Summarize results

Select/identify test cases to runSet up test environment

Create test environmentLoad test data

Repeat for each test caseSet up test prerequisitesExecuteCompare resultsLog resultsClear up after test case

Clear up test environmentDelete unwanted test dataSave important dataSummarize resultsAnalyze test failuresReport defects

Advanced Software Engineering 32

Test automation levels [Bob Binder, ISSRE04]

10 000 test/hFull test automation*5

1000 tests/hModel-based testing4

100 tests/hAutomated test scripting3

10 tests/hManual testing2

1 test/hTest by poking around1

* Evaluation is the bottle-neck of automated testing (Oracle)

Advanced Software Engineering 33

Test automation promises

1. Efficient regression test2. Run tests more often3. Perform difficult tests (e.g. load, outcome check)4. Better use of resources5. Consistency and repeatability6. Reuse of tests7. Earlier time to market8. Increased confidence

Advanced Software Engineering 34

Common problems1. Unrealistic expectations2. Poor testing practice ”Automatic chaos just gives faster chaos”3. Expected effectiveness4. False sense of security5. Maintenance of automatic tests6. Technical problems (e.g. Interoperability)7. Organizational issues

Advanced Software Engineering 35

What can be automated?

IntellectualPerformed once

RepeatedClerical

1. Identify

2. Design

3. Build

4. Execute

5. Check

Advanced Software Engineering 36

Cost distributionManual testing

Same tests automated

More mature automation

Edit Set up Execute Analyze Clear up

Advanced Software Engineering 37

Limitations of automating testingDoes not replace manual testingManual tests find more defects than automated tests

– Does not improve effectivenessGreater reliance on quality of tests

– Oracle problemTest automation may limit the software development

– Costs of maintaining automated tests

Advanced Software Engineering 38

Limitations of automating testing

Advanced Software Engineering 39

What to automate first?

Most important testsA set of breadth tests (sample each system area overall)Test for the most important functionsTests that are easiest to automateTests that will give the quickest paybackTest that are run the most often

Advanced Software Engineering 40

Tests...... to Automate

Tests that are run often– Regression tests– Important but trivial

Expensive to perform manually– Multi-user tests– Endurance/reliability tests

Difficult to perform manually– Timing critical– Complex tests– Difficult comparisons

... not to automateTests that are run rarelyTests that are not important

– will not find severe faultsHard to recognize defects

– Usability tests– Do the colours look nice?

Advanced Software Engineering 41

There is no silver bullet!The Reckless assumptions of automation

“Automated tests execute a sequence of actions without human intervention. This approach helps eliminate human error, and provides faster results. Since most products require tests to be run many times, automated testing generally leads to significant labour cost savings over time. Typically a company will pass the break-even point for labour costs after just two or three runs of an automated test.”

Advanced Software Engineering 42

Reckless assumptions of automationTesting is a sequence of actions that we can automate

– Manual testing is a process that adapts easily to change and can cope with complexity

– Humans are able to detect and evaluate problem patterns– How do you automatically interpret test results?

Advanced Software Engineering 43

Reckless assumptions of automation

Testing means repeating the same actions over and over

– Regression testing, no new defects found– Stepping in someone else’s footprints minimizes the chance of

being blown up by a land mine

Advanced Software Engineering 44

Reckless assumptions of automationAn automated test is faster, because it needs no human intervention

– It can be surprisingly hard to create a test suite that runs without human intervention

– E.g. interpreting results, fixing/maintaining tests, problems and bugs in the test tools, changes to software under test, ...

Advanced Software Engineering 45

Reckless assumptions of automationAutomation reduces human errors

– Some errors are amplified – any error that goes unnoticed when the test scripts and result files are created will go systematically unnoticed over and over again

Advanced Software Engineering 46

Reckless assumptions of automationWe can quantify the costs and benefits of manual vs. automated testing

– Two different processes that reveal different failures – a fare comparison is difficult

– Test automation is one part of a test strategy – not a substitute for manual testing

Advanced Software Engineering 47

Reckless assumptions of automationAutomation will lead to significant labour cost savings

– Developing automationWriting test scripts is software developmentTesters are seldom competent codersThe quality of test code must be good

– Operating automated tests– Maintaining automated test suites– Probably does not significantly reduce the need for manual testing

Advanced Software Engineering 48

Reckless assumptions of automationAutomation will not harm the test process

– It’s dangerous to automate something we don’t understand-(GIGO – Garbage in Garbage out) - this only creates garbasefaster

– The result will be a large mass of test code that nobody understands