advanced software engineering: software testing · advanced software engineering 16 history of...

Web: http://www.megadatasys.com Assisted with templates

Advanced Software Engineering: Software Testing

COMP 3705(L5)

Sada Narayanappa

Anneliese Andrews Thomas Thelin

Carina Andersson

Advanced Software Engineering 2

News/Project/PapersQuestions:


LectureChapter 8

– OrganizationChapter 14

– Tools– Test automation


Organization• Support for decision making• Enhance teamwork• Independency• Balance testing – quality• Assist test management• Ownership of test technology• Resources utilization• Career path


Test OrganisationDegrees of freedom

– Tall or flat

– Market or product

– Centralized or decentralized

– Hierarchical or diffused

– Line or staff

– Functional or project


7 approaches to test organisation1. Each person’s responsibility2. Each unit’s responsibility3. Dedicated resource4. Test organisation in QA5. Test organisation in development6. Centralized test organisation7. Test technology centre

ExplanationBenefitsDrawbacks( Example )( Change )


1. Each person’s responsibility

Product developers

M

P P P P P

PT


2. Each unit’s responsibility

Product developers

M

P P P P P

PT

PT


3a. Dedicated resource

Product developers

M

P P P P P

P T


3b. Dedicated resource on a large scale

Product developers

M

P P P P P

Product developers

M

P P P P P

Test developers

TM

T T T T T


4. Test organisation in QA

Test Development

QAO

Group

Product Development

PDO

Group


5. Test organisation in development

Test Development

Group

Product Development

PDO

Group


6. Centralized test organisation

Test Development

Group

Product Development

VP

Group


7. Test technology centre

Test Development

Group

Product Development

VP

Group

Test Technology

Group

SE


What criteria to consider?

Which organisation should we choose?

Size

Maturity

Focus


History of software testing

Defect prevention & Test

process1950 1960 1970 1980 1990 2000

Fix bugs

Exhaustive

testing

Prove it worksProve it does

not work

Early test designInternet (Agile)

Test automation

tools

Advanced test

automation

Automated business

optimization


ToolsGood at repeating tasks

Good at organising data

Requires training

Introduced incrementally

No “silver bullet”

Ease of usePowerRobustnessFunctionalityEase of insertionQuality of supportCostCompany policies and goals


Test tools

Test management tools

Testexecution and comparison tools

Dynamic analysis tools

Debuggingtools

Coverage toolsStatic analysis tools

Test design tools

Architectural design

Detailed design

Code

Requirement specification

Unit test

Integration test

Performancesimulator tools

System test

Acceptance test


Tools categories by activityReviews and inspections

Test planning

Test design and development

Test execution and evaluation

Testing support

Test DesignTools

Static Analysis

Tools

Execution:

•Unit•Integration•System•Acceptance


Reviews and inspectionsComplexity analysis

– Identify problem areas

Code comprehension– Show different views of the

artefact

Syntax and semantic analysis– Check and warn

Test DesignTools

Static Analysis

Tools

Execution:

•Unit•Integration•System•Acceptance


Test planningTemplates for test plan documentation

Test schedule and staffing estimates

Complexity analyser

To large extent general project management tools


Test design and developmentTest data generator

Requirements-based test design tool

Capture/replay

Coverage analysis

Often integrated with test execution tools


Test execution and evaluationCapture/replay

Coverage analysis

Memory testing

Test case management

Simulators and performance


Capture/replay

Static c/r tools (without scripting language)

Static c/r tools (with scripting language)

Variable c/r tools


Coverage analysisMonitoring execution

Pre-processing code

Problem: intrusive, in particular for real-time systems if-then-else

while-loop

case-of


Memory testingLeaks (allocate but not free memory)

Array bounds

Initialization of variables

Easy to use, low cost


Simulators and performanceHardware emulators

Environment simulators

Load generators

PC target HWHW emul

SWSW

target HWSWsimulator simulator

target HWSWgenerator


Test case managementKeep track of test ware

Manage test execution

Seamless integration with other tools

Automated test reporting

Not yet available :-(


Testing support toolsTrouble reporting

– Report, Dispatch, Follow-up

Configuration management– Manage, control and coordinate changes


Test automationEffectiveness – independent of automationEfficiency – can be improved by automation

Automation takes 2-10(-30) times the time for first run!


Different levels of automation

Select/identify test cases to runSet up test environment

Create test environmentLoad test dataRepeat for each test caseSet up test prerequisitesExecuteCompare resultsLog resultsAnalyze test failuresReport defectsClear up after test case

Clear up test environmentDelete unwanted test dataSave important data

Summarize results

Select/identify test cases to runSet up test environment

Create test environmentLoad test data

Repeat for each test caseSet up test prerequisitesExecuteCompare resultsLog resultsClear up after test case

Clear up test environmentDelete unwanted test dataSave important dataSummarize resultsAnalyze test failuresReport defects


Test automation levels [Bob Binder, ISSRE04]

10 000 test/hFull test automation*5

1000 tests/hModel-based testing4

100 tests/hAutomated test scripting3

10 tests/hManual testing2

1 test/hTest by poking around1

* Evaluation is the bottle-neck of automated testing (Oracle)


Test automation promises

1. Efficient regression test2. Run tests more often3. Perform difficult tests (e.g. load, outcome check)4. Better use of resources5. Consistency and repeatability6. Reuse of tests7. Earlier time to market8. Increased confidence


Common problems1. Unrealistic expectations2. Poor testing practice ”Automatic chaos just gives faster chaos”3. Expected effectiveness4. False sense of security5. Maintenance of automatic tests6. Technical problems (e.g. Interoperability)7. Organizational issues


What can be automated?

IntellectualPerformed once

RepeatedClerical

1. Identify

2. Design

3. Build

4. Execute

5. Check


Cost distributionManual testing

Same tests automated

More mature automation

Edit Set up Execute Analyze Clear up


Limitations of automating testingDoes not replace manual testingManual tests find more defects than automated tests

– Does not improve effectivenessGreater reliance on quality of tests

– Oracle problemTest automation may limit the software development

– Costs of maintaining automated tests


Limitations of automating testing


What to automate first?

Most important testsA set of breadth tests (sample each system area overall)Test for the most important functionsTests that are easiest to automateTests that will give the quickest paybackTest that are run the most often


Tests...... to Automate

Tests that are run often– Regression tests– Important but trivial

Expensive to perform manually– Multi-user tests– Endurance/reliability tests

Difficult to perform manually– Timing critical– Complex tests– Difficult comparisons

... not to automateTests that are run rarelyTests that are not important

– will not find severe faultsHard to recognize defects

– Usability tests– Do the colours look nice?


There is no silver bullet!The Reckless assumptions of automation

“Automated tests execute a sequence of actions without human intervention. This approach helps eliminate human error, and provides faster results. Since most products require tests to be run many times, automated testing generally leads to significant labour cost savings over time. Typically a company will pass the break-even point for labour costs after just two or three runs of an automated test.”


Reckless assumptions of automationTesting is a sequence of actions that we can automate

– Manual testing is a process that adapts easily to change and can cope with complexity

– Humans are able to detect and evaluate problem patterns– How do you automatically interpret test results?


Reckless assumptions of automation

Testing means repeating the same actions over and over

– Regression testing, no new defects found– Stepping in someone else’s footprints minimizes the chance of

being blown up by a land mine


Reckless assumptions of automationAn automated test is faster, because it needs no human intervention

– It can be surprisingly hard to create a test suite that runs without human intervention

– E.g. interpreting results, fixing/maintaining tests, problems and bugs in the test tools, changes to software under test, ...


Reckless assumptions of automationAutomation reduces human errors

– Some errors are amplified – any error that goes unnoticed when the test scripts and result files are created will go systematically unnoticed over and over again


Reckless assumptions of automationWe can quantify the costs and benefits of manual vs. automated testing

– Two different processes that reveal different failures – a fare comparison is difficult

– Test automation is one part of a test strategy – not a substitute for manual testing


Reckless assumptions of automationAutomation will lead to significant labour cost savings

– Developing automationWriting test scripts is software developmentTesters are seldom competent codersThe quality of test code must be good

– Operating automated tests– Maintaining automated test suites– Probably does not significantly reduce the need for manual testing


Reckless assumptions of automationAutomation will not harm the test process

– It’s dangerous to automate something we don’t understand-(GIGO – Garbage in Garbage out) - this only creates garbasefaster

– The result will be a large mass of test code that nobody understands

advanced software engineering: software testing · advanced software engineering 16 history of...

Documents