integration and testing - cs.rit.eduhpb/scia/bridge_2005/intro_se... · software engineering i –...

Integration and Testing

Uses slides from Lethbridge & Laganiere, 2001

Software Engineering I – SE361

Testing phases: V model

Requirements

Specifications

Design

Detailed Design

Implementation

Unit Testing

Integration Testing

System Testing

Acceptance Testing

A lifecycle view that shows relationshipsbetween development and test phases


Concurrent Test Development

Reqs

Design Impl

Test Design

TestImpl

Unit Testing

IntegTesting

SystemTesting

Testing is done after development, but test design & impl.often happens concurrently with development


Testing Phases Unit Testing

Developer tests individual modules Usually glass box

Integration testing Put modules together, try to get them working together Integration testing is complete when the different pieces

are able to work together System testing

Black-box testing of entire deliverable against specs Acceptance testing

Testing against user needs, often by the user


Coverage

Testing should “try out all possibilities” Exhaustive testing is impossible for non-trivial software Objective is to cover possibilities as well as possible

Unit testing / glass-box testing Function coverage: Each function/method executed by

at least one test case Statement coverage: Each line of code covered by at

least one test case (need more test cases than above) Path coverage: Every possible path through code

covered by at least one test case (need lots of test cases)


…Coverage Black-box testing

Requirements coverage: Each requirement / use case /user story covered by at least one test case

Equivalence class coverage: Each equivalence class foreach input covered by at least one test case

Combination of equivalence classes coverage: Eachlegal combination of equivalence classes for differentinputs covered – may require huge number of test cases

There are tools that will automatically analyze variousforms of coverage and generate statistics

Also tools to help generate test cases


System Testing

Thorough black-box testing of entire software beforedelivery, based on requirements specification

Final phase of testing before delivery

Requirements coverage is a must, equivalence classcoverage preferred

For most commercial software, reasonably good coverageof combinations of equivalence classes is the norm

Can automate system testing by creating “test harnesses”that feed test inputs and check outputs


Integration Testing

Integration: Putting together the code created by different developersand trying to run an end-to-end test case

Can be a major challenge, because every small misunderstanding/miscommunication between developers results in mismatches thatneed to be found and removed Each time it doesn’t work, need to trace which code is the source

of the problem Not uncommon to need to resolve dozens of bugs before that first

test case will run through! Integration testing becomes easier if

Unit testing is thorough Interfaces between modules are well-defined


Alternative Integration Strategies

Big bang integration

Put all the code together in one go, and try to get it allworking together

Works only for relatively small projects

Incremental integration Test each individual module in isolation

Start with one module, add one module at a time, work outproblems among them before adding next

More manageable, but more work

“Horizontal” integration if there are multiple subsystems

“Vertical” integration if there are multiple layers


Top down integration

Start by testing just the user interface. The underlying functionality are simulated by stubs.

Pieces of code that have the same interface as the lower levelfunctionality.

Do not perform any real computations or manipulate any real data.

Then you work downwards, integrating lower and lowerlayers.

Advantage: Easy to see the behavior at each stage The big drawback to top down testing is the cost of writing

the stubs.


Bottom-up integration

Start by testing the very lowest levels of the software.

You needs drivers to test these lower layers of software. Drivers are simple programs designed specifically for testing that

make calls to these lower layers.

Drivers in bottom-up testing have a similar role to stubs intop-down testing, and are time-consuming to write.

Advantage: Each piece being tested is complete in itself

Problem: Need to write lots of drivers, functionality maynot be easy to visualize


Sandwich testing

Sandwich testing is a hybrid between bottom-up and topdown testing.

Test the user interface in isolation, using stubs.

Test the very lowest level functions, using drivers.

When the complete system is integrated, only the middlelayer remains on which to perform the final set of tests.


Example of different integration strategies

stub stub stub

UI Layer

stub stub stub

UI Layer

Functional layer

UI Layer

Functional layer

Databaselayer

Networklayer

Functional layer

Databaselayer

Networklayer

driver driverdriver

Databaselayer

Networklayer

driver driverdriver

stub stub stub

UI Layer

Top-down testing Bottom-up testing Sandwich testing

Databaselayer

Networklayer

driver driverdriver

Fullyintegrated

system


The test-fix-test cycleWhen a failure occurs during testing:

Each failure report is entered into a failure tracking system. It is then screened and assigned a priority. Low-priority failures might be put on a known bugs list

that is included with the software’s release notes. Some failure reports might be merged if they appear to

result from the same defects. Somebody is assigned to investigate a failure. That person tracks down the defect and fixes it. Finally a new version of the system is created, ready to be

tested again.


The ripple effect

There is a high probability that the efforts to remove thedefects may have actually added new defects

The maintainer tries to fix problems without fullyunderstanding the ramifications of the changes

The maintainer makes ordinary human errors

The system regresses into a more and more failure-pronestate


Regression testing

It tends to be far too expensive to re-run every single testcase every time a change is made to software.

Hence only a subset of the previously-successful test casesis actually re-run.

This process is called regression testing. The tests that are re-run are called regression tests.

Regression test cases are carefully selected to cover asmuch of the system as possible.

The “law of conservation of bugs”: The number of bugs remaining in a large system is

proportional to the number of bugs already fixed


Deciding when to stop testing

All of the level 1 (“critical”) test cases must have beensuccessfully executed.

Certain pre-defined percentages of level 2 and level 3 testcases must have been executed successfully.

The targets must have been achieved and are maintainedfor at least two cycles of ‘builds’.

A build involves compiling and integrating all the components. Failure rates can fluctuate from build to build as:

Different sets of regression tests are run. New defects are introduced.


The roles of people involved intesting

The first pass of unit and integration testing is calleddeveloper testing.

Preliminary testing performed by the software developers who dothe design.

Independent testing may be performed by separate group. They do not have a vested interest in seeing as many test cases

pass as possible.

They develop specific expertise in how to do good testing, andhow to use testing tools.


Test planning

Decide on overall test strategy

What type of integration

Whether to automate system tests

Whether there is an independent test team

Decide on the coverage strategy for system tests

Compute the number of test cases needed

Identify the test cases and implement them

The set of test cases constitutes a “test suite”

May categorize into critical, important, optional tests (level 1, 2, 3)

Identify a subset of the tests as regression tests


Testing performed by users andclients

Alpha testing Performed by the user or client, but under the supervision of the

software development team.

Beta testing Performed by the user or client in a normal work environment. Recruited from the potential user population. An open beta release is the release of low-quality software to the

general population.

Acceptance testing Performed by users and customers. However, the customers do it on their own initiative.


Inspecting compared to testing

Both testing and inspection rely on different aspects ofhuman intelligence.

Testing can find defects whose consequences areobvious but which are buried in complex code.

Inspecting can find defects that relate to maintainabilityor efficiency.

The chances of mistakes are reduced if both activitiesare performed.


Testing or inspecting, which comesfirst?

It is important to inspect software before extensivelytesting it.

The reason for this is that inspecting allows you to quicklyget rid of many defects.

If you test first, and inspectors recommend that redesign isneeded, the testing work has been wasted.

There is a growing consensus that it is most efficient to inspectsoftware before any testing is done.

Even before developer testing


Packaging for Delivery

Software we deliver to the user must include

Executable in a convenient format e.g. EXE, JAR file

Release notes

User documentation: instructions on usage Tutorials, user manuals, “getting started” instructions

Installation instructions

May create “installables”

Compressed packages e.g. zip files, tar files

Scripts that automate installation procedures

integration and testing - cs.rit.eduhpb/scia/bridge_2005/intro_se... · software engineering i –...

Documents