presentation paper bio return to main menu t9validator retrieval updates ... cantata, and junit....

P R E S E N T A T I O N

International Conference On

Software Testing, Analysis & ReviewDEC 4-8, 2000 • COPENHAGEN, DENMARK

Presentation

Bio

Return to Main Menu T9

Thursday, Dec 7, 2000

A Case Study of Testing a

Harry M. Sneed & Siegfried Goeschl

Paper

Distributed Internet Application

Software Daten Service 2000

A Case Study of Testinga Distributed Internet System

Siegfried [email protected]

&Harry M. [email protected]

Software Daten ServiceA-1200 Vienna, AUSTRIA

www.sds.at

2 Software Daten Service 2000

! Introduction of the testing process in a distributedenvironment

! Case study! Testing tools! Defect classification! Conclusion

Topics


Web-Page

Web-Page

Web-Page

Web-Page

Web-Page

Web-Page

Web-Page

Web-Page

Application-Server

Web-Server

Web-Server

Data-Server

CORBA-Interface

Application-Server

Web-Server

Web-Server

Applets

XML HTML XML HTML

local data local data

Servlets

Corelets

global data

Internet System Architecture


Testing the System Architecture

TransactionTest

Middleware

Web-Server

Application-Server

Database-Server

PerformanceTest

Tran

sact

ion

Mon

itorin

g

Mem

ory

Man

agem

ent

Data bases

DatabaseIntegrity

TestMemory Mgt.

Test

DatabaseAccess

Test

TransmissionTest

Web work stationsWeb Page

Test

UsabilityTest


Processing Layer

Access Layer

Presentation Layer

Control Layer

Before Images DB After Images DB

Arguments

Arguments

Arguments

Results

Results

Results

Web-Page

Inputs Outputs

Internet Application Architecture


Applet Servlet Servlet

Application-Server

Application-Server

Business Objects

Functionality

Distribution

Business

Rules

Querry Updates

Web Page Edditing Process Control

Edit RuleTesting

WorkflowControlTesting

BusinessLogic TestingIntegrity Rules

Data IntegrityTesting

Search PathTestingWeb-Page-Usage

Usage ProfileTesting

Web-Page-Presentation

UpdateThreadTesting

Testing the Application


Testing Levels

ClassesunderTest

Class Flattener

Test Driver

Test

Stu

bs

Test

Met

hods

Integration Testing

Specified Arguments

Message Generator

Input Messages

Componentsunder Test

Return Messages

Message Validator

Unit Testing

ExpectedReturn Values

Grey Box

Replay Capture

Web Pages

Web Client

Web Server

Data

System Testing

Black Box

DataGenerator

DataValidator

Retrieval Updates

White Box


Reengineering of a financial document retrievalsystem :

– Presentation and control layer in PERL– Processing layer written in C or FORTE®

– SYBASE® database

Reengineering tasks :

– Rewriting the presentation and control layer (Datastream,UK)

– Rewriting the processing layer using C++/CORBA (ourdevelopment group at Datastream, UK)

– Changing the database layout (Disclosure, USA)

The Case Study Example


Architecture of the Case Study Example

IntranetInternet

HTTP

SGML

IIOP

SGML

SGML

SGML

Webserver

SGML-Server

FORTE-Server

C-Server

CORBA-Server

C++/SOLARISCORBA Server

DB Access Layer

SOLARIS/NTFAST-CGI or ASP

C++/SOLARISSGML-to-CORBA Converter

BusinessPartners

SYBASEDatabase

Customers

New New

Work In ProgressWork In Progress


The Project Vision

! New server system must be fully backwardcompatible" No “Big Bang” integration feasible

! Better average response time" Performance problems due to growing customer base

! Higher reliability of the new system" FORTE server unstable" Regular server restarts due to memory leaks

! Better maintainability" Low quality production code of original system

! Flexible test environment" Multiple development/deployment platforms (SOLARIS/NT)" Test database updated daily (production database replica)


! Assertion Package– “Just In Time” debugging

! LOGVIEW 2– Monitoring multiple logfiles

! APITEST– C++ unit test framework– Multiplatform support– PURIFY integration

! DROPTEST– PERL regression test framework– Integration and system testing

The Test Tools


The Testing Strategy

Server API

DB Access Layer

Off-The-ShelfCode

ApplicationFramework

ORB Runtime

DB CompatibilityLayer

SGML Parser

ApplicationFramework

ORB Runtime

IIOPScripts

Webserver Runtime

SGML

DROPTEST

Correctness

APITEST

Robustness

ReusedCode

ApplicationCode

Sybase Runtime

APITEST

Robustness

APITEST

Performance

Webserver SGML-Server CORBA Server


! Displays the updates ofmultiple logfiles distributedover the network

! No file locking! Filters of large logfiles by

– text– time range

! Prints logfiles! Saves the current logfile! Turns automatic updating

on/off

LOGVIEW 2


APITEST

! C++ unit test framework for multiple platforms– UNIX– WINDOWS 98/NT

! Stress testing support– Built-in multithreading– Repetition of tests

! Test Output– Screen or file– File for every worker thread

! No hard coded test data through configuration filesupport

! PURIFY® integration for memory leak detection


APITEST PURIFY Integration


APITEST Command Line Options

-C file Test data file-F file Script file with tests to execute-I Ignore failed tests-M threads Number of executing threads-O filename Name of output file-R Random shuffle test sequence for each thread-S testsuite Testsuite to execute-T testcase Testcase to execute-V Enable verbose output-W ms Time to wait between two test cases-X repeats Number of repetitions


! All tests are located in testtest! Optional pre-testing step is

stored in prepre! Optional post-testing step

is stored in postpost! Reference output is found

in referencereference! Output of test run is stored

in safesafe! Additional PERL scripts are

stored in bin bin and common common– Replace utility– Filter utility– Timestamp removal– Common test setup

DROPTEST Structure


Defect Classification I

0

20

40

60

80

100

120

140

160

minor 26 9 4 2 0 41severe 27 20 5 10 4 66fatal 11 14 13 6 0 44all 64 43 22 18 4 151

droptest apitest build manual review all


Defect Classification II

0

20

40

60

80

100

120

140

memory 9 20 3 32business logic 19 23 15 57sgml 36 0 0 36all 64 43 18 125

droptest apitest manual all


Conclusion I

! Assertion package with “Just In Time” debugging facility isessential during development and testing– debugging a violated assertion interactively is convenient

! APITEST was invaluable for detecting memory related defects(63% of all memory related defects) such as

– exception handling code generated by IDL compiler– a memory leak in the SYBASE libraries– a memory leak and a multithread-safety problem in the CORBA naming

service– a memory leak in the SNMP library

! DROPTEST was essential for integration testing :– 23% of all defects were SGML server related– ease of generating the reference data (daily database updates)– ease of integrating real life requests

! LOGVIEW2 simplified the task of monitoring multiple logfiles onmultiple machines


Conclusion II

The Effort :– 2 developers and 1 tester– 5 months development time– 4.700 statements of newly developed code (including test stubs)– 7.400 statements of reused in-house code (application framework)– 151 defects were discovered– 40% of the development effort went into testing

The Result :– Testing proved to be a technical challenge– The development group bet 100 US$ that the independent QA

group (USA) wouldn’t find a defect in their code. They won.– The system was considered high quality and error-free


References

DROPTEST! Fred Wild, “Design for Testability”, Dr. Dobb’s Journal, 2/1997

Software Documentation Tool! Object Outline, http://www.bbeesoft.com

Famous bugs! Nancy Leveson, “Safeware: System Safety and Computers”, Addision-Wesley,

1995! Ariane-5: http://www.esrin.esa.it/htdocs/tidc/Press/Press96/ariane5rep.html! http://infotech.fanshawec.on.ca/gsantor/Computing/FamousBugs.htm! http://academic.emporia.edu/pheattch/cs552s98/bugs.htm

A Case Study of Testing a Distributed Internet-

System

by

Siegfried GOESCHL Software Daten Service, A-1200 Vienna, AUSTRIA

Email: [email protected] &

Harry M. SNEED Software Daten Service, A-1200 Vienna, AUSTRIA

Email: [email protected]

Abstract This paper describes the testing strategy, methods and tools used for testing a distributed financial services system on the internet with CORBA interfaces for handling a high volume of queries to a relational database. The new system architecture was derived by reengineering a previous more limited system for handling local requests. The paper starts with an outline of the software architecture, the testing requirements and the testing strategy. It then goes on to describe the tools used in the project and the results achieved. For the project a generic C++ integration test framework was developed especially for testing distributed components. This tool is described in detail.

Keywords: Distributed Systems, Internet, CORBA, Unit Testing, API Testing, Interface Test Drivers, Message Generation, Interface Validation, Regression Testing

2

Table of Contents

1 INTRODUCTION ......................................................... 3

2 TEST PLANNING ........................................................ 5

2.1 DATABASE LAYER TEST.................................................... 5 2.2 CORBA SERVER TEST .................................................... 5 2.3 SGML SERVER TEST ..................................................... 5

3 TEST TOOLBOX ......................................................... 6

3.1. ASSERTION PACKAGE ...................................................... 6 3.1.1. Motivation ...................................................... 6 3.1.2. Just In Time Debugging .......................................... 6 3.1.3. Customisable Behaviour .......................................... 6

3.2. LOGFILE VIEWER ......................................................... 7 3.2.1. Motivation ...................................................... 7

3.3. C++ UNIT TEST FRAMEWORK................................................. 8 3.3.1. Motivation ...................................................... 8 3.3.2. Basic Concepts .................................................. 9 3.3.3. Detection of Memory Leaks ...................................... 10 3.3.4. Performance Measurements ....................................... 11 3.3.5. Stress Testing ................................................. 11 3.3.6. Test of Multithread Safety ..................................... 11 3.3.7. Simple Modification of Test Data ............................... 11 3.3.8. Integration of Regression Testing .............................. 12 3.3.9. Simple Scripting .............................................. 12 3.3.10. Command Line Options .......................................... 12

3.4. PERL REGRESSION TEST FRAMEWORK........................................... 12 3.4.1. Motivation ..................................................... 12 3.4.2. Basic Concepts ................................................. 13 3.1.1 3.4.3. SGML Server Regession Testing .......................... 13

4 CONCLUSION .......................................................... 14

5 REFERENCES .......................................................... 16

1 Introduction

On one hand, there is a general lack of adequate tools for testing distributed systems. On the other hand, there is a great need for such tools, especially in light of the effort required for testing such systems. It has been noted that testing takes at least one half of the effort required to develop object-oriented, distributed systems. [Binder94a]

The special problems involved in verifying and validating distributed software have been addressed in the appropriate literature. [Bourne97] There are not only multiple nodes of a network with an exponential number of interconnections, but also multiple architectural layers to be tested. This raises test costs to an ever higher level. In addition, there are four distinct levels of testing to be done:

• Module or class testing, often referred to as White-Box testing,

• Integration testing, often referred to as Grey-Box testing,

• System testing, often referred to as Black-Box testing, and

• Acceptance testing, often referred to as Red-Box testing. [Hetzel88]

The testing of individual classes can be handled with the help of existing unit test tools such as TestBed, Cantata, and JUnit. [BeGa98] System testing can also be done using conventional capture/replay and database editing tools. [Beizer95] Acceptance testing represents a greater challenge in the case of distributed systems, but it is not critical if the other tests have been performed well. The greatest challenge is that of integration testing. Distributed systems consist of a large number of interacting components executing in diverse environments with complex interfaces. These interactions and their interfaces have to be verified and validated by integration testing since neither unit nor system testing suffices for this purpose. The only way to accomplish this in an effective and efficient manner is via automated tests. As pointed out by D.Graham and M.Fewster in their work on test automation, test tools may not replace human intelligence in testing, but without them testing complex systems at a reasonable cost will never be possible. [FewGra99]

The tool described in this paper grew out of an internet project in Great Britain in the years 1998/99. The project combined web servers with distributed CORBA servers and relational databases. The goal of the project was to provide clients in different parts of the world with rapid access to financial data. The standard queries needed to be processed within seconds on relational tables with several million tuples.

The project in question was actually a reengineering project since the basic application had already existed in the form of two different server systems providing access to a Sybase database. One of these server systems was programmed in standard C, the other in the 4th Generation language FORTE©. The results of the queries were delivered as SGML formats. However, this original solution had reached it limits. It was becoming increasingly slow and unreliable under the increasing load of the clients. Therefore, as often happens with quick-and-dirty solutions, management decided to reengineer the system, reimplementing the critical transactions in C++ with a new CORBA server for faster message distribution and a separate SGML front-end to transform the data into SGML. The new architecture is depicted in the following diagram.

4

Figure 1-1 Architecture Overview

The webserver receives HTML requests, converts them to the SGML-based request and routes them to the appropriate server depending on the request type. The SGML server accepts SGML requests from the web server and invokes one or more IDL functions on the CORBA Server. The CORBA server delegates the calls to a database layer which encapsulates the business logic and shields the server from changes to the database layout.

The reengineering project was subject to the following constraints:

• the project team had little experience using a SYBASE© database

• the functionality to be reengineered was either written in C or FORTE© and consisted of approximately 10.000 lines of legacy code

• the original database also had to be reengineered due to performance problems

• the existing monolithic, single-entry API had to replaced by a modular IDL interface to enable reuse

• developing a CORBA server in C++ is error-prone since the developer bears responsibility for the memory management

• the resulting product had to support multiple platforms (SOLARIS and NT)

In view of these constraints, testing was assigned a particularly high value from the start. It was decided to commit at least half of the project resources to testing, to plan the test carefully, to make a test plan in accordance with prevailing standards, and to develop appropriate test tools for executing the tests. The project should be test driven in accordance with the recommendations of Hetzel, Beizer, Binder and others. [Binder99]. It was mainly due to this emphasis on testing, that the project came to a successful conclusion. Without the aid of automated test tools embedded in a well defined test process, the project could never have been completed on time.[KoomenPol99]

5

2 Test Planning

In planning the test, the following requirements had to be considered:

• ensure that the SGML server is fully backward compatible to the previous server

• ensure that the new system has a better average response time than the old one

• ensure that the new system is reliable

• ensure that the tests can be run on multiple platforms

Altogether, there were three layers of software to be tested first individually and then in union with one another.

• SGML Server

• CORBA Server

• Database Server.

Using a bottom-up test strategy, the architectural layers were to be tested in reverse order, starting with the database access layer and working up to the client.

2.1 Database Layer Test To test the database layer it was decided to create a regression test suite written in C++ to compensate for the lack of SYBASE experience and to test the changes in the database schema. This regression test also provided a way to benchmark various queries against the database. Since there were multiple ways of accessing the same data, this was important for recording and comparing results.

2.2 CORBA Server Test To test the CORBA server, it was decided to implement a proxy client application to generate requests to the server. This proxy client should simulate the behaviour of a later real client in generating all kinds of requests in any order. The focus for this test was to prove the absence of memory leaks and to test the reliability of the ORB (Object Request Broker)

2.3 SGML Server Test To test the SGML Server it was decided to create a regression test suite and run it with previously recorded requests in accordance with the API specification as well as with hundreds of real-life requests from the production system. The requests extracted from production served to represent the existing operational profile. The requests derived from the specification served to represent the new requirements.

6

3 Test Toolbox

To facilitate testing, a set of tools was used consisting of:

• an Assertion Package

• Logfile Viewer

• C++ Unit Test Framework

• Perl Regression Test Framework

3.1. Assertion Package

3.1.1. Motivation

Meyers concept of Design by Contract [Mey97] is an important concept for software developers and can reduce the time spent during debugging significantly. It requires the programmers to formulate pre conditions on the states their inputs should have when they start and post conditions on the states their outputs should have when they are finished. Besides that, invariant assertions can be postulated on the state of objects which should not change.

The standard assertion function in the C Runtime Library is not very useful in a production environment because it prints an error message and terminates the program which is exactly not the behaviour you would like to have in a client or server test. Imagine a GUI application where the assertion function prints an error message to an invisible error console or a server which suddenly terminates. The standard assertion function is only intended for local module testing.

For this reason, it was decided to implement an assertion package with the following functionality:

• Just In Time Debugging Support on Windows NT

• customisable behaviour

• selective activation of the assertions.

3.1.2. Just In Time Debugging

The idea of just in time debugging is borrowed from the MFC assertion package which starts a debugger whenever a failed assertion is encountered. It has turned out that this feature is extremely valuable for debugging and testing. It is important to know whether one’s program terminated because of some addressing error or because of a violated invariant. In the latter case, one can investigate the call stack to pinpoint the reason why the invariant was violated.

3.1.3. Customisable Behaviour

An industrial strength assertion package has to be flexible to meet the following requirements:

7

• log an error message

• send an SNMP (Simple Network Management Protocol) alert to a management console

• send a mail to the system administrator

• throw an exception instead of terminating the program.

This flexibility could be achieved by implementing a Chain of Command stack [GoF95] where the system under test can register various assertion handlers. Each assertion handler is responsible for providing a certain level of functionality before it delegates the assertion to the next handler.

3.2. Logfile Viewer

3.2.1. Motivation

Under Windows NT the editors used to view a logfile have several shortcomings:

• some programs try to lock the logfile exclusively

• hardly any program updates the view of the logfile automatically

• one needs a filter to view large logfiles

• one may need to view multiple logfiles simultaneously

Therefore, it was necessary to develop a customised logfile viewer explicitly for testing client/server type applications which would not have such limitations.

8

Figure 2 - Monitoring multiple logfiles

3.3. C++ Unit Test Framework

3.3.1. Motivation

Experience has proven that it pays off to create and maintain formal unit tests as part of the software development cycle [Wie96]. Usually a software developer writes a test driver for his/her unit of work, but that test driver only ensures the software doesn’t crash during the first execution. Typically it consists of a “main” routine and a few statements to print the results. Later the test driver is neglected so that it no longer fits to the unit under test and might not even compile. At this stage everybody has abandoned unit testing and it becomes costly to maintain the software because there is no cheap way to reproduce individual unit tests.

This dilemma is well known and the question always arises why software developers don’t create a proper unit test. Usually there are several answers:

• “it’s too difficult to test this unit separately ...”

• “there is not enough code to justify a unit test ...”

• “I don’t have enough time ...”

• “I don’t know how to write a unit test ...”

9

The point to be stressed here is that all of these objections are valid to a certain degree since most software developers are reasonable persons. The answers are no longer reasonable if a unit test framework fulfils the following requirements:

• there is a unit test framework available

• it takes as much time to create a proper unit test as to “hack” a throw-away test driver

• one gets more “bang for the bucks” as a software developer and therefore making it more difficult to neglect unit testing

The advantages of using a reusable unit test framework instead of a throw-away test driver are obvious. The test framework supports:

• detection of memory leaks

• performance measurements

• stress testing

• test of multi-thread safety

• simple modification of test data

• integration of regression testing

• simple scripting

For these reasons and others, the project decided to implement its own unit test framework for C++ to satisfy the special requirements of a distributed test. The result was the tool APITEST. In the end, it was used in several projects both on SOLARIS© and Windows NT©.

3.3.2. Basic Concepts

Every unit test can be divided into one or more test suites with a unique name. Every test suite can contain one or more test cases. Every test case is uniquely identified through a number within a test suite.

Figure 3 - Relationships between Unit Test, Test Suites and Test Cases

An invocation of a particular test case can lead to one of the following results:

•

• passed

• failed

• crashed

• undefined

10

A test case fails if it does not fulfil all required post conditions of the test. A test case crashes if an exception is propagated out of the test case implementation. A test case is undefined if it is not used any longer. [Sneed98]

The command line interfaces allows one to run a unit test in multiple ways:

• run all test suites

• run one particular test suite

• run on test case within a test suite

After execution of a unit test an execution summary is displayed as follows: Execution Summary Test started : Tue Oct 05 13:39:26 1999 Test finished : Tue Oct 05 13:39:26 1999 Test Executed : 6 Test Crashed : 0 Test Passed : 4 Test Failed : 1 Test Not Defined : 1 TEST 3 FAILED CPU time (sec) : 1.0220000

3.3.3. Detection of Memory Leaks

One of the most difficult aspects of the programming language C/C++ when making large-scale software development is the manual memory management. Therefore it is essential to simplify the detection of memory leaks.

Figure 4 - Integrating APITEST with Purify

APITEST provides a close integration with INSURE++© from Pure Atria, a commonly used tool for detecting memory leaks and related problems during development. INSURE++ tracks all memory and checks allocations of memory during execution, but it turned out that the

11

results are difficult to interpret due to the way resources are allocated by initialisation routines. It became necessary to execute test cases repeatedly in order to obtain sensible results. APITEST prints status messages on the INSURE++ console before executing a test case and reports the allocated resources after executing the test case.

In the absence of INSURE++ it is possible to track resource consumption manually, but it requires more time to detect small memory leaks. Therefore, it turned out that the combination of APITEST and INSURE++ was the best solution.

3.3.4. Performance Measurements

The execution summary provides information about the time spent executing the test cases. For meaningful results it is necessary to execute tests more than once, otherwise results are distorted by the initialisation time.

3.3.5. Stress Testing

For the server development it is also essential to test the system under full load and/or gather performance data with an increasing load. Assuming that there is a unit test acting as a client it is convenient to reuse the unit test and test the server with multiple clients over a long time. APITEST facilitates this by providing command line options to execute the tests multiple times with one or more threads. Furthermore a user-defined wait between executing two test cases is supported to simulate the behaviour of real clients.

3.3.6. Test of Multithread Safety

Executing a unit test with multiple threads is important if you want to ensure that your software doesn’t deadlock itself after the first execution. Usually these tests are omitted since writing multithreaded test programs is significantly more work. APITEST facilitates these tests by it’s ability to define the number of executing threads on the command line. In this case the unit test is executed by multiple threads simultaneously. Furthermore, it is possible to randomise the order in which test cases are executed for every thread to simulate the behaviour of real clients. Passing this test does not guarantee that your code is multithread-safe, but it does give a good indication that it does not deadlock after a minute.

3.3.7. Simple Modification of Test Data

A throw-away test driver tends to hardcode required test data, thus complicating the migration to a new test environment, e.g. a new database for integration testing. To avoid this complication it is necessary to provide an easy way to access test data. This is accomplished by providing a configuration file support:

dbtest.1.branch_code=14000 dbtest.1.customer_id=730875734

Each entry in the configuration file consists of a key and a value whereas the key consists of the test suite name, the test case number and the name of the variable.

12

3.3.8. Integration of Regression Testing

When testing a module for accessing a database, the unit test typically doesn’t check the content of the results apart from checking certain invariants (e.g. there must be some results or this field can’t be null). APITEST provides the notion of a verbose mode to produce verbose output either to the console or to a file. This verbose output can be captured and explicitly compared against a reference to detect any differences (see 3.4. Perl Regression Test Framework)

APITEST enables every thread to use its own output file therefore it is possible to detect differences between verbose outputs of the threads. Since every thread is executing the same tests such a difference might indicate a race condition.

3.3.9. Simple Scripting

In reality there will always be a couple of broken tests which for various reason do not need to be executed every time. A solution is to specify all test cases and/or test suites which should be executed by a script file. The script file is then driven by the command line options.

3.3.10. Command Line Options

The unit test framework supports the following command line options: Command Line Option Description -C file Name of test configuration file -F file Script file with tests to execute -I Ignore failed tests -M threads Number of executing threads -R Random shuffle test sequence for each thread -S testsuite [-T testcase]] Test(s) to execute -O filename Name of output file -V Enable verbose output -W ms Time to wait between two test cases -X repeats Number of repetitions

3.4. Perl Regression Test Framework DROPTEST is a Perl regression test framework developed by Fred Wild [Wild97]. It has been used slightly modified in many projects under Windows 95,Windows NT, SINIX, AIX and SOLARIS.

3.4.1. Motivation

It was decided to implement a full-blown regression test suite for the SGML Server to prove that its behaviour is equivalent to the replaced components. One part of the regression test suite was where it is was drafted according to the specification. The other part consisted of a couple of hundred real-life requests from the production system to cover unspecified or under specified behaviour.

13

3.4.2. Basic Concepts

The whole program logic is based on the following directory structure:

Figure 5 - Droptest directoy structure

The main idea is to drop a regression test into the TEST directory (hence the name DROPTEST) The Perl script checks for every file found in TEST if there is a corresponding file in PRE and/or POST and executes them in the following order:

• PRE step (optional)

• TEST step (mandatory)

• POST step (optional)

In this manner the script provides an optional pre and post step which could do the following things (example):

• copy configuration information

• delete temporary files

• kills left-over processes

The basic idea is that every test step produces some sort of reproducible output. A test step is considered successful if the current output is identical to a reference output stored in the reference directory. A test step is considered unsuccessful if the current output is different from the reference output.

3.1.1 3.4.3. SGML Server Regession Testing

The regression test was executing with a copy of the real production database and was updated daily. Under these circumstances the regression test depended heavily on the ease of creating new reference data. This was accomplished by running the regression test against the original server written in C or FORTE to generate the reference data.

The SGML Server was tested with a PERL Script sending SGML requests to the SGML Server. The invocation of the PERL script was done as the actual test step of Droptest.

14

REQUEST=COMBINED_SEARCH USERID=ROWJ COM_FILER_NAME_SEGMENT=LEPCO COM_COMPANY_NR=L420000000 OUT_SORT_KEY_2=DOC_TYPE OUT_TYPE=D OUT_SORT_ORDER_1=D FILTER_INSIDER_DOCS=Y OUT_SORT_ORDER_2=A SEC_ONLY=N OUT_START_ROW=1 OUT_TOTAL_ROWS=Y OUT_MAX_ROWS=25 OUT_SORT_KEY_1=DOC_DATETIME_ARRIV_MIN Figure 3-6 SGML request data for retrieving financial documents

In the post step caching related tags were removed from the result and compared with the reference files. By means of such automated comparison, deviations in the new results could be readily detected. [Harrold98]

4 Conclusion

Testing often becomes the critical-path activity on a software project because it is postponed to the end of the software development process. With a test driven development approach, unit tests are developed in parallel to the software and considered as part of the software release.

The combination of the test tools with a Daily Build and Smoke Test [McCon96] proved to be highly effective during the incremental development. The layered testing approach is essential for an efficient software correction in a distributed system. By drilling down the various layers of the software it is possible to pinpoint problems and to debug the offending code very efficiently.

The usage of APITEST in combination with INSURE++ turned out to be invaluable to locate memory related problems which is crucial for a high volume financial service system. Since developers create and maintain unit tests for his/her work the software is never allowed to disintegrate.

The bugs encountered were distributed in the following way:

• Minor bugs that leave the product usable but the customer unsatisfied

• Severe bugs that leave the product hardly usable

• Fatal bugs that crash the server

15

0

20

40

60

80

100

120

140

160

minor 26 9 4 2 0 41

severe 27 20 5 10 4 66

fatal 11 14 13 6 0 44

all 64 43 22 18 4 151

droptest apitest build manual review all

droptest: detected by executing DROPTEST apitest: detected by executing APITEST build: problems during the build process manual: detected during manual testing and test case preparation review: detecting through reviewing the code Furthermore the encountered bugs could be classified in memory related problems (memory leaks, memory access violations, using already freed memory), implementation related problems and SGML related problems (formatting the SGML replies, missing SGML tags) as depicted in the final diagram:

0

20

4060

80100

120140

memory 9 20 3 32

business logic 19 23 15 57

sgml 36 0 0 36

all 64 43 18 125

droptest apitest manual all

memory : memory related bugs impl: implementation related bugs sgml: output related bugs in SGML Server All in all, it can be concluded that the use of a unit and integration test framework to detect errors in the components and their interfaces as soon as the code was compiled was a great advantage and contributed significantly to the lack of faults found in system testing where they are much more difficult to locate and remove. However, this required that the test be

16

designed into the system architecture from the start. This design for testability proved to be a critical success factor in the overall project. [Binder94b].

5 References

[BeGa98] Kent Beck, Erich Gamma: ’’ Test Infected: Programmers Love Writing Tests’’, http://members.pingnet.ch/gamma/junit.htm, 1998

[Beizer95] Beizer, Boris: Black-Box Testing, John Wiley & Sons, New York, 1995, p. 236

[Binder94a] Binder, Robert: ’’Testing Object-Oriented Systems -- A Status Report’’, American Programmer, Vol. 7, No. 4, April 1994, p. 22

[Binder94b] Binder, Robert: ’’Design for Testability in Object-Oriented Systems’’, Comm. Of ACM, Vol. 37, No. 9, Sept. 1994, p. 37

[Binder99] Binder, Robert: Testing Object-Oriented Systems, Addison-Wesley Pub., Reading, Mass., 1999, p. 58

[Bourne97] Bourne, Kelly: Testing Client/Server Systems, McGraw-Hill, New York, 1997, p. 19

[FewGra99] Fewster, Mark/ Graham, Dot: Software Test Automation, Addison-Wesley Pub., Harlow, G.B., 1999, p. 26

[GoF95] Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides: ‘‘Design Patterns -- Elements of Reusable Object-Oriented Software’’, Addision-Wesley Professional Computing Series, 1995

[Harrold98] Harrold,Mary Jean/Rothermel,Gregg: ’’Empirical Studies of a Safe Regression Test Selection Technique’’, IEEE Trans. on S.E., Vol. 24, No. 6, June 1998, p. 401

[Hetzel88] Hetzel, Bill: The Complete Guide to Software Testing, QED Information Sciences, Wellesley, Mass. 1988, p. 73

[KoomPohl99] Koomen,Tim/Pohl, Martin: Test Process Improvement, Addison-Wesley Pub. Harlow, G.B., 1999, p. 109

[McCon96] Steve McConnel: ’’Rapid Development’’, Microsoft Press, 1996

[Mey97] Bertrand Meyer: ‘‘Object Oriented Software Construction’’, 2nd Edition, Prentice-Hall Inc., 1997

17

[Sneed98] Sneed, Harry: ’’Automated Test Case Specification for Integration Testing of Distributed Objects’’ Proc. of EuroStar98, München, Dec. 1998, p. 137

[Wie96] Karl Eugene Wiegers: ’’Creating a Software Engineering Culture’’, Dorset House Publishing, 1996

[Wild97] Fred Wild: ‘‘Design for Testability’’, Dr.Dobb’s Journal 2/97

Thursday 7 December 2000

T9

A Case Study of Testing a Distributed Internet Application

Harry M. Sneed

Siegfried Goeschl Harry M. Sneed, MPA graduated from the University of Maryland in 1969, worked 3 years for the Navy Dept in Washington, D.C., later went to Germany where he worked for the Volkswagen Foundation and Siemens. In 1978 he founded together with Dr. Ed. Miller from SRA the first European Test Factory in Budapest. Later the factory became a tool development laboratory where tools for all phases of the software lifecycle were developed. Sneed lead the development of 9 software tools for the mainframe including 3 test tools as well as personally developing 26 tools for the pc.. In addition he has lead 23 user application projects to a successful conclusion, consulted customers and taught 16 different software engineering seminars on all subjects from project estimation to software maintenance and reengineering. After the collapse of the socialist institutes, Sneed went to the UBS in Zürich as a reengineering consultant, project leader and tool developer. Since 1996 he has become a free lance programmer, specializing in old COBOL and new C++. At the moment, he is working as a reengineering specialist for CaseConsult in Germany and as a quality assurance specialist for Software Daten Service in Austria, in addition to being a quest professor at the University of Regenburg. Sneed has authored 11 books on software development, testing, maintenance, management and migration in German language. In addition, he has published more than 120 papers in both German and English, including some 35 papers on testing and quality assurance. Sneed is a member of the German GI, the IEEE and the ACM. He can be contacted at [email protected] Siegfried GOESCHL, holds a BSc equivalent in Chemical Engineering from HBLVA Rosensteingasse and a Master’s degree in Computer Science from the Vienna University of Technology. Goeschl worked for 3 years as a Software Engineer and Technical Lead for BASYS in the area of process automation and visualisation before moving into the financial sector. In 1997 he went to London were he worked for DATASTREAM specialising in implementing CORBA-based systems and testing distributed applications. In 1999 he joined Software Daten Service in Austria as a technical architect for an Online Banking Application based on CORBA and JAVA. He can be contacted at [email protected]

presentation paper bio return to main menu t9validator retrieval updates ... cantata, and junit....

Documents