efficient mobile network acceptance testing

42
P R E S E N T A T I O N International Conference On Software Testing, Analysis & Review DEC 4-8, 2000 COPENHAGEN, DENMARK Presentation Bio Return to Main Menu T7 Thursday, Dec 7, 2000 Efficient Mobile Network Jorg Paul Paper Acceptance Testing

Upload: hathu

Post on 03-Jan-2017

223 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Efficient Mobile Network Acceptance Testing

P R E S E N T A T I O N

International Conference On

Software Testing, Analysis & ReviewDEC 4-8, 2000 • COPENHAGEN, DENMARK

Presentation

Bio

Return to Main Menu T7

Thursday, Dec 7, 2000

Efficient Mobile Network

Jorg Paul

Paper

Acceptance Testing

Page 2: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

1

Efficient Mobile Network Acceptance TestingEfficient Efficient Mobile Network Mobile Network Acceptance TestingAcceptance Testing

Jörg PaulJörg PaulMannesmann Mobilfunk GmbH

Am Seestern 1D-40547 Düsseldorf

[email protected]

DrDr. Dieter. Dieter Kreuer KreuerEricsson Eurolab Deutschland

GmbHEricsson-Allee 1

D-52134 HerzogenrathGermany

[email protected]

Thomas GärtnerThomas GärtnerEricsson GmbH

Fritz-Vomfeldestr. 26D-40547 Düsseldorf

[email protected]

Page 3: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

2

ContentsContentsContents

1. Mobile Network Acceptance Testing2. The NEXT/AIMS Tool Platform3. The GSM R8 Project4. Results5. Conclusions

1. Mobile Network 1. Mobile Network Acceptance TestingAcceptance Testing2. 2. TheThe NEXT/AIMS Tool NEXT/AIMS Tool PlatformPlatform3. 3. TheThe GSM R8 Project GSM R8 Project4. 4. ResultsResults5. 5. ConclusionsConclusions

Page 4: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

3

GMSC MSC

MSC

HLR

VLR

AuC

EIR

PSTN

SSS

BSC BSC

BTS BTS BTS BTS

BSS

MSMS

OMC

OSS

Basic Structure of a GSM NetworkBasicBasic Structure of Structure of a GSM Networka GSM Network

Page 5: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

4

•• differentdifferent releases forreleases for SSS, BSS SSS, BSS and and OSS OSS at at leastleastshipped once shipped once a a year by the vendor year by the vendor ((EricssonEricsson))

•• Test Test phasesphases forfor thethe newnew softwaresoftware::

1. Mobile Network Acceptance Testing1. Mobile1. Mobile Network Network Acceptance TestingAcceptance Testing

Unit Tes

ting

Function Tes

ting

System

Test

Type A

ccep

tance

Test

FSA / Rollo

ut

Feature

Testin

g

Page 6: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

5

•• any newany new HW HW or or SW SW release tested before integration release tested before integrationinto the commercial network into the commercial network („(„no harm no harm to to the networkthe network“)“)

•• joint test activity between vendor and operatorjoint test activity between vendor and operator•• testing done testing done in in the actual the actual live live network configurationnetwork configuration•• SUT not SUT not tested standalonetested standalone, , but as but as a a part of the full part of the full GSMGSM

networknetwork•• emphasis on emphasis on end-to-end end-to-end functional testingfunctional testing•• both blackboth black--box and whitebox and white--box testing performedbox testing performed

Type Acceptance TestingTypeType Acceptance Testing Acceptance Testing

Page 7: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

6

MSC

BSSPSTN HLRVLREIR

AuC Billing

PBX

ACD

VMS

CTISMSC VRU VRU SGSN

IN SCP

W-MSC

WAP

VoIP

Growth of Network ComplexityGrowth ofGrowth of Network Network Complexity Complexity

Page 8: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

7

Conflict Situation for TestingConflict Situation for Testing

Testing

Quality

Costs Time

Page 9: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

8

Network Complexity and Test CoverageNetwork Network Complexity and Complexity and Test Test CoverageCoverage

Complexity

Test Coveragewith Test Automation

TAcc duration

Test Coveragewithout Test Automation

10

15

20

25

30

35

40

1990 1992 1994 1996 1998 2000

Page 10: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

9

•• resource savings due to automation of regressionresource savings due to automation of regressiontestingtesting

•• speed-up ofspeed-up of test execution test execution•• more efficient test more efficient test plant plant utilisationutilisation (e.g. (e.g. at nights andat nights and

weekendsweekends))•• higher higher reproducibilityreproducibility of automated testing of automated testing•• enable testersenable testers toto concentrate on the new features of concentrate on the new features of aa

releaserelease

Automated Testing: Our goalsAutomatedAutomated Testing: Our goalsTesting: Our goals

Page 11: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

10

•• originaloriginal target application target application: : Statistical Usage TestingStatistical Usage Testing•• end-to-end functional testing ofend-to-end functional testing of mobile mobile networksnetworks•• both blackboth black--box and whitebox and white--box testing possiblebox testing possible•• suitable platform forsuitable platform for automation of automation of Type Type AcceptanceAcceptance

testing performed by Ericsson andtesting performed by Ericsson and Mannesmann MannesmannMobilfunkMobilfunk

3. The NEXT/AIMS Tool Platform3.3. The The NEXT/AIMS Tool NEXT/AIMS Tool Platform Platform

Page 12: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

11

Principle of NEXT/AIMSPrinciple of Principle of NEXT/AIMSNEXT/AIMS

NEXT AIMSServer

AIMSHardware

SUT

TENUX

SRSMS

Commands Actions

Calls

Commands Commands

SUT StatusSUT Status

Results Signals

Page 13: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

12

Integrated Automated Testing Platform for GSMIntegrated Automated Testing Platform for Integrated Automated Testing Platform for GSMGSM

Page 14: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

13

4. The GSM R8 Project4. 4. The The GSM R8 ProjectGSM R8 Project

CSS PSS BSS OSS

GSM R8

Page 15: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

14

Automated Test Objects: Automated Test Objects:

Data / Fax33%

SMS3%

Call Tests50%

Other Tests14%

Page 16: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

15

AutomatedAutomated Test Test Objects Objects ( (continuedcontinued):):

•• Call HandlingCall Handling•• Interworking withInterworking with BSS BSS and and GSM R7 GSM R7•• SupplementarySupplementary Services Services•• ChargingCharging•• Statistics and Traffic MeasurementsStatistics and Traffic Measurements•• Data andData and Fax Tests Fax Tests•• ShortShort Message Service Message Service•• APZ Regression TestsAPZ Regression Tests

Page 17: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

16

5. Results5. 5. ResultsResults

Grade of Automation: Test Objects

24%

52%

24%

fully automated TOsfully manual TOsmixed TOs

Page 18: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

17

Test Case Execution Times

5:04:22 5:04:225:04:22

0:00:00

0:05:00

0:10:00

0:15:00

0:20:00

0:25:00

0:30:00

0:35:00

0:40:00

0:45:00

0:50:00

0:55:00

Annou

ncem

ts APZ R

egres

sn

Call Barr

ings

Basic

Call Hnd

Cha

rging

Fax

/Data

Frau

d Prev

ntn

Interw

rk R7/R

8 Int

rw M

SC/BSC

Statist

. Sub

stm.

Overal

l

Test Object

MinimumMean Maximum

Page 19: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

18

Number of Repetitions

0,0%2,7% 0,3% 0,0%

8,0%

89.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6

Page 20: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

19

Execution Time Distribution

0%

2%

4%

6%

8%

10%

12%

14%

00:0

0-01

:00

01:0

0-02

:00

02:0

0-03

:00

03:0

0-04

:00

04:0

0-05

:00

05:0

0-06

:00

06:0

0-07

:00

07:0

0-08

:00

08:0

0-09

:00

09:0

0-10

:00

10:0

0-11

:00

11:0

0-12

:00

12:0

0-13

:00

13:0

0-14

:00

14:0

0-15

:00

15:0

0-16

:00

16:0

0-17

:00

17:0

0-18

:00

18:0

0-19

:00

19:0

0-20

:00

20:0

0-21

:00

21:0

0-22

:00

22:0

0-23

:00

23:0

0-24

:00

Page 21: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

20

Implementation Progress

0%

20%

40%

60%

80%

100%

120%

43/9

9

45/9

9

47/9

9

49/9

9

51/9

9

01/0

0

03/0

0

05/0

0

07/0

0

09/0

0

11/0

0

13/0

0

15/0

0

17/0

0

19/0

0

21/0

0

Calendar Week

Coded TCsVerified TCs

Page 22: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

21

Grade of Automation: Test Cases

20% ne wTe s ts (manual) 59%

automate d Re gre s s ion

Te s ts

80% mixe d Re gre s s ion

Te s ts

41% manual Re gre s s ion

Te s ts

Page 23: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

22

Relative Costs Comparison

66%

3%

16%

34%

0%

20%

40%

60%

80%

100%

STP hrs/TC mhrs/TC TAcc duration/TC total R8 TAccduration

automated Testmanual Test

Page 24: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

23

6. Conclusions: What we reached ...6. 6. ConclusionsConclusions:: What we reached ...What we reached ...

• Length of Type Acceptance reduced by 30%Length of Type Acceptance reduced by 30%

!! shorter time-to-market shorter time-to-market

•• more efficient test plant utilisation more efficient test plant utilisation

!! automated testing 24 hours a day automated testing 24 hours a day

•• resource savings during execution phase resource savings during execution phase

! test cases normally run unattended by the tester

Page 25: Efficient Mobile Network Acceptance Testing

mmobilfunk

mannesmanntelecommunications

24

What we learned ...What we learned ...What we learned ...

• stability of the test configuration is criticalstability of the test configuration is critical

•• manual testing in parallel to automated testing should bemanual testing in parallel to automated testing should be avoided avoided

•• no significant speed-up of test case execution no significant speed-up of test case execution (SUT was the limiting factor)(SUT was the limiting factor)

Page 26: Efficient Mobile Network Acceptance Testing

1

Increasing the Efficiency of Mobile Network Acceptance Testing by Test Automation

Jörg Paul, Dr. Dieter Kreuer, Thomas Gärtner

Joerg Paul graduated in physics at the Aachen University of Technology in 1995. At university he mainly worked in the fields of theoretical elementary particle physics and computational physics. In 1996 he joined the German GSM network operator Mannesmann Mobilfunk. Since then he has been working in several software testing and test automation projects. He is now project leader for the introduction of automated testing into the joint type acceptance project GSM R8 of Mannesmann Mobilfunk and Ericsson. Email address: [email protected]

Page 27: Efficient Mobile Network Acceptance Testing

2

Increasing the Efficiency of Mobile Network Acceptance Testing by Test Automation

Jörg Paul, Dr. Dieter Kreuer, Thomas Gärtner

Abstract: In the last decade, operators and manufacturers of mobile networks had to deal with a rapid growth of network complexity, which is still going on. Accordingly, the number of services offered to the customer and the number of network elements and interfaces incorporated in the network strongly increase from year to year, while the release cycles decrease. A lot of testing is performed to find the software defects of the new releases before commercial service. Both manufacturers and network operators are facing the problem how to perform an ever-increasing amount of test cases in a decreasing time interval. In order to solve this problem, the test automation tool platform NEXT/AIMS has been developed by Ericsson in close co-operation with Mannesmann Mobilfunk. NEXT/AIMS was designed to automate end-to-end functional testing that is performed in system or acceptance test. Already in the acceptance test of the current Ericsson software release GSM R8, which is the very first application of the tool, a grade of test automation of 47% has been reached. By means of the automated testing approach a considerable amount of human resources can be saved in the test execution phase, which could therefore be shortened by 30%. In this paper we are going to describe the NEXT/AIMS tool platform and the results drawn from its first practical application in the GSM R8 type acceptance project.

1. Introduction

Since the first operators of digital mobile networks have started service, a rapid technical development has taken place. One example for the evolution of digital mobile networks is the Global System for Mobile Communications (GSM) (see [1] and [2]. GSM is a standard for wireless mobile networks which offers various kinds communication services to the fully mobile customer, like telephony, data / fax transmission, short message service (SMS) or Intelligent Network (IN) services. Currently, packet switching technologies like GPRS (General Packet Radio Service) are being introduced into the GSM networks [3]. The coming third generation mobile telephony standard UMTS (Universal Mobile Telephony System) will be designed to fully support mobile multimedia applications like video conferencing [4].

The introduction of many new services leads to a more and more complex network. Not only do the single network elements become more complex due to the increasing number of supported features, but also the number of different network elements needed to provide a broad range of services is growing steadily.

The rapidly growing network complexity leads to a real challenge for testers. In spite of the growing number of features, the release cycles of hardware and software for the network even tend to become shorter due to the strong demands of the very dynamic mobile communications market which promotes competition between the operators. It is very important to find remaining software defects before the product is released to commercial service. Therefore, both the manufacturers and the operators of mobile networks have to perform a number of test cases which is nearly exponentially growing in a time frame

Mannesmann Mobilfunk GmbH Am Seesstern 1 40547 Düsseldorf Germany Tel.: +49 211 533 3799 Fax: +49 211 533 3804 [email protected]

Ericsson Eurolab Deutschland GmbH Ericsson-Allee 1 52134 Herzogenrath Germany Tel.: +49 2407 575 447 Fax.: +49 2407 575 651 [email protected]

Ericsson GmbH Fritz-Vomfelde Str. 26 40547 Düsseldorf Germany Tel.: +49 211 534 2118 Fax.: +49 211 534 2356 [email protected]

Page 28: Efficient Mobile Network Acceptance Testing

3

which is constant or even decreasing. If the testing efficiency cannot be increased significantly, a loss of test coverage and also test quality cannot be avoided. It will no longer be possible to test neither all the new functions of the system under test (SUT) nor the old functions (regression tests) with sufficient depth under the existing time constraints.

In addition to the various tests performed by the vendors (e.g. system test), Mannesmann Mobilfunk as an operator of one of the largest GSM networks world-wide, performs a thorough acceptance test of any new component to be introduced into the operative network. This test activity is called Type Acceptance and will be performed for each new or changed component in advance of the integration into the live network. The goal of the Type Acceptance is to assure that any new hardware or software component will not damage the commercial network. Therefore, it is crucial that Type Acceptance testing is performed in a test environment which ideally should be identical to the real commercial network. Otherwise, there is no way to find faults related to special parameter settings, external interfaces or details of the configuration used in the field. Those faults usually are not visible in an isolated laboratory configuration.

In general, a Type Acceptance project is a joint project between Mannesmann Mobilfunk and the corresponding vendor of the SUT. As Ericsson is one of the main vendors of GSM network elements used by Mannesmann Mobilfunk, there are several large Type Acceptance projects run by both companies every year. Therefore, both Mannesmann Mobilfunk and Ericsson are facing the problem of an exploding number of test cases, which can no longer be compensated by additional human resources. As a consequence, both companies jointly have been working on methods to increase the efficiency of both system and acceptance testing significantly since 1996.

The introduction of test automation turned out to be a very promising measure to optimise testing in a way that test coverage and quality can be kept sufficiently high. A large number of test cases performed in a Type Acceptance are end-to-end functional tests, which verify the system functionality from the customer’s point of view. Therefore, one of the key difficulties of the project was to develop at test system that simulates the mobile end-user and is able to execute high-level functional tests with real GSM terminal equipment (black-box testing). But at the same time, a large number of test cases require internal information about the status of the SUT (white-box testing). In order to develop such a test system, it was necessary to combine the experience and know-how of both the network operator and the manufacturer.

In 1999 both companies decided to implement automated testing in the Type Acceptance of the Ericsson switching software release CME201 R8 CSS. The goal of the project was to automate as many of the existing regression test cases as possible. The NEXT/AIMS tool platform used for the project is described in section 2 of this paper. In section 3 we are going to describe the test scope for the GSM R8 test automation project, and section 4 summarizes the results we obtained.

2. The NEXT/Aims platform

The automated test case execution during the type acceptance was based on a tool platform developed by Ericsson in close co-operation with Mannesmann Mobilfunk.

To understand the architecture and implementation of the tool platform, one must consider, that its actual target application is automated black box testing focussing on Statistical Usage Testing (StUT) [7]...[10]. The intent of StUT is to describe real user behaviour via a stochastic traffic model. The tool platform can then run a simulation of the model and thereby automatically generate a large number of test cases that are immediately executed on the system test plant. Apart from checking the correctness of the execution, quality figures such as Mean Time to Failure (MTTF) or other Quality of Service Figures can be determined.

Page 29: Efficient Mobile Network Acceptance Testing

4

Performing a performance test on a telecom system requires putting the exchanges under a considerable traffic load. Load generators are the only means of generating such a high load. Additionally, a Type Acceptance test platform requires control of the terminal equipment to access all the functions available to the users. This means controlling actual mobiles, modems, etc. Furthermore, it must be possible to enter operator commands, because some tests require activation/deactivation of services, or checking of internal switch status information. Such commands are usually entered into the exchange via a dedicated interface in a particular man-machine language (MML). Finally, some global execution control must be in place to synchronise the actions of the test tools on the various interfaces. For instance, a test case might set up a configuration via the MML interface, initiate a time-variant background traffic load via the load generator, and at the load peaks perform calls with real mobiles.

Our test platform is shown in figure 1. On top, the global execution is controlled by a software tool called NEXT (Network Executor), which has access to a repository of test cases (TC’s). NEXT controls a load generator MGTS and another tool AIMS (Air interface and mobile subscriber simulator, consisting of hardware and software components) to access real terminal equipment. Fax and data terminals are not directly controlled via AIMS, yet (this is planned for the upcoming release), but since nearly 40% of the automated R8 Type Acceptance test cases consist of fax/data test cases, a preliminary solution was implemented involving a fax/data server. This server has several modems connected (mobile, PSTN, ISDN) and runs both a fax and a data server program that use the modems to perform all kinds of fax and data transmissions.

Ericsson telecom exchanges feature an MML interface provided by a unit called the “IOG” (Input/Out-put Group). NEXT has an interface to the IOG, too, allowing to send MML commands and to receive printouts. Finally, NEXT can control a tool dubbed PCM-LIG (PCM Link Interrupt Generator) which can perform controlled interrupts of the PCM links between exchanges, thus simulating hardware failures.

The system under test is a small GSM network consisting of the nodes MSC (Mobile Switching Centre), BSC (Base Station Controller) and BTS (Base Transceiver Station). NEXT is directly connected to the MSC and BSC nodes via their IOG MML interfaces. The commercially available load generator MGTS is

Figure 1: Integrated automated testing platform for GSM

Page 30: Efficient Mobile Network Acceptance Testing

5

used to put load onto the signalling links of the MSC exchange (A-G). The AIMS tool interfaces mobiles or fixed telephones (BL = Both way Line circuit, simulating PSTN phones, or phones on a Private Access Branch Exchange or PABX). PABX’s have a 2 Mbit/s PCM connection to the Primary Rate Access Interface, PRA, of the MSC. Using coaxial cables, mobile radio signals are fed to the base stations via a Semiconductor Roaming Simulator, SRS, simulating the air interface between mobiles and base stations.

Lacking a commercial solution, AIMS was created by our group to control the keypad of a mobile (or any other telephone), to read out its display (if supported by the telephone), and to monitor power, ringing, and in-band signalling tones. The latter tones are decoded and recognised, as are test tones sent by AIMS to continuously monitor voice connections (voice checks). Furthermore, AIMS can send and recognise DTMF tones, which can e.g. be recorded as a substitute for unrecognisable voice announcements. Finally, AIMS controls the attenuation in the SRS. The SRS has 4 inputs for mobile HF signals and 4 outputs for base station HF cables. To connect more mobiles, it is either possible to combine the HF signals of several mobiles on a single input, or to use several SRS units connected in cascade. AIMS is presented in more detail in [5].

The NEXT tool, which controls the test execution, has also been created by our project group. NEXT is based on Petri nets enhanced with a command language. Blocks of commands, executed upon occurrence of a Petri net transition, can be included in each transition, e.g. to control external tools. Token values can be used as parameters for the commands in the command blocks. Token values are also used to import signals from the connected test tools (e.g. a token 4 showing up in a place named ringing might signify, that mobile 4 is ringing). NEXT allows designing, executing, and monitoring test models. Many more details on NEXT and examples of its Petri nets can be found in [5] and [6].

For testers who merely wish to execute existing test programs, we created the NEXT Graphical User Interface for Testers (GUI-T). Within GUI-T, test models are merely files. GUI-T allows executing a test suite, i.e. a series of sequential or parallel batches of tests. GUI-T also manages test resources (such as mobiles, IOG-connections etc.), so that test models can pick from the pool of available resources whatever they need during execution. This allows a flexible resource allocation during parallel execution.

Prior to execution, test cases to be used with the platform are designed and tested by Ericsson testers. This is actually the most time consuming task, involving the creation of reusable modules, splitting existing test instructions into small, reusable pieces, and running the test case under different fault situations. This effort is usually higher than the preparation of test cases for manual execution, but pays off with repeated execution during the Type Acceptance and other test phases.

A test case with NEXT consists of an initialisation step, an execution step, and a restoration step. The initialisation step serves to connecting to the test tools and claiming the resources. In many test cases, certain parameters in the system under test are also checked and modified here. In the execution step, test commands are sent to the system under test and the received signals are compared with expected results. Any deviation is evaluated as a fault and the test case directly jumps to the restoration step. Here, where the test case eventually ends up even with no detected faults, all the modifications of the initialisation step are undone and the test resources are released again.

When Ericsson testers have finished designing a set of test cases, test experts from Mannesmann Mobilfunk are invited to validate the execution of the test cases and to approve their correctness – or to disapprove it and to demand modifications. As watching the test execution for hundreds of test cases is very time consuming, the more viable way of exchanging and validating execution logs was often adopted, especially when test cases occurred in many variations of a common basic test case.

A test case approved by Mannesmann Mobilfunk is not to be touched anymore, which can always be verified by employing a configuration management (CM) tool. Approved test cases are stored in a suitable repository. Prior to a test campaign in the TAcc, GUI-T is used to combine test cases from this repository into a test suite for successive execution. Test cases that do not disturb and are not disturbed by the execution of other test cases may be specified to run in parallel, if the available test resources allow so. Otherwise, test case execution is sequential. Test resources are assigned to test cases executed in parallel on a first come-first serve basis with no optimisation, which we found to be no problem, as very few test cases actually allow parallel testing activities at all.

Page 31: Efficient Mobile Network Acceptance Testing

6

For each test case, two log files are automatically created during execution: the Automated Test Report (ATR) which reports on the execution steps, expected and observed results including resource and timing information. The test case designers need to program the output, which goes into the ATR. The other log is the Automated Test Log (ATL), where all the messages exchanged with connected tools are stored in the order of occurrence, with no influence of the tester. This log is sometimes useful for fault tracing or proving the correctness of a test case. During and after the execution, both files can be opened and browsed from within GUI-T. The ATR contains the test result (passed or failed) which can also be viewed on-line in GUI-T during the test suite execution.

3. Test scope of the GSM R8 CSS project

The GSM standard is still being enhanced every year. Most of the vendors of switching systems and base station systems built so-called "releases" at least once a year. These releases incorporate all new and enhanced features and usually consist of software and hardware components. The current release developed by Ericsson is named GSM R8. GSM R8 contains various updates for the different GSM subsystems, the Switching Subsystem (SSS), the Base Station Subsystem (BSS) and the Operation and Maintenance Subsystem (OSS). Due to the introduction of GPRS, the SSS is in fact split up into the classical Circuit Switching Subsystem (CSS) and the Packet Switching Subsystem (PSS), which has a similar structure as a router-based IP network.

After GSM R8 had undergone the Ericsson software quality assurance process, it entered the Type Acceptance process. The aim of the Type Acceptance is to find the remaining software defects in the software and to verify that the new software release to be installed will not damage or disturb the network operated by Mannesmann Mobilfunk. In a network which is as complex as a GSM network, many of the defects found in the testing process are in some way related to the network configuration. In order to find the errors that would most likely occur in operation, any new software release has to be tested in the live network configuration, otherwise it might be impossible to find these defects. That means that the same software configuration has to be established, the same market corrections and adaptations have to be loaded and the same exchange properties and parameter settings have to implemented as in the field. Furthermore, an acceptance test for any GSM network element (e.g. an MSC) is never performed standalone. All the other network elements of the full GSM chain have to be present, too, and many of the Type Acceptance Tests are end-to-end functional tests involving real terminal equipment.

Figure 2 shows a sketch of an example test configuration of the GSM R8 Type Acceptance. Although the SUT is the MSC, the test configuration is in fact a small GSM network, which contains three MSC nodes and three Base Station Controllers (BSC). The MSC is part of the CSS and fulfils no packet switching functionality yet. In order to trace the mobile customer who is roaming through the network, the HLR (Home Location Register) and the VLR (Visitor Location Register) are needed. The HLR is a database that contains the basic subscription information of the customer, e.g. his service profile, and permanently is updated with the subscriber’s status (attach/detach, present location etc.). The VLR database stores the information of all subscribers that are roaming in the specific MSC area the VLR is responsible for. Normally, MSC and VLR are realized on the same physical network node. The BSC is responsible for the maintenance of the base transceiver stations (BTS), locating the mobile stations, rate adaptation and data compression of voice calls (TRC, transcoder rate adaptor centre). It also handles the signalling and payload traffic between the mobile stations and the switching system.

During the execution phase of a Type Acceptance project, the test configuration needs to be changed quite often, because certain test objects in general need a very special environment to be tested. However, in a test environment that changes frequently, configuration faults are likely to occur, which sometimes causes problems that are extremely hard to detect. In order to find these configuration faults in advance of testing, which is always a tedious and time-consuming job if done manually, we designed a special test script (Petri net) to check the set-up of the network automatically. The script can be run easily before any test session without additional manual effort. By means of the Configuration Check, a lot of problems could be detected and solved even before testing actually started.

Page 32: Efficient Mobile Network Acceptance Testing

7

For an operator of a mobile network it is extremely important, that all the features of the network that are directly visible to the customer are free of errors. Therefore, a major part of the tests performed during Type Acceptance are end-to-end functional tests with an emphasis on black box testing techniques. In order to find exactly the errors the customer would encounter, the tester often takes the customer's point of view and verifies the services offered to the user under all relevant initial conditions. Every time the software changes, these services have to be tested again in order to assure that no defects have entered the software (regression testing). Starting with the first GSM releases the pool of regression tests has grown tremendously. Already in the last years it has turned out to be impossible to perform all the regression tests and the tests for the new functionality in the given time frame. With the NEXT/AIMS tool platform we have managed to automate 47% of the regression tests performed for the CSS part of the GSM R8 Type Acceptance.

The customer accesses the network through the mobile station (MS). One crucial result of former acceptance tests was that it is not sufficient to simulate the mobile phone. Although the interface between the mobile station and the GSM network is a standardised interface, the communication between the mobile and the network is a non-trivial matter and caused a lot of troubles in the past. In contrast to methods applied in protocol conformance testing (see [11]), end-to-end functional testing is always performed with commercially available mobile phones. Therefore, NEXT/AIMS accesses the SUT via the same interfaces that are used by the customers of the network operator.

In addition to this, many test cases require the ability of NEXT to interact with the SUT via the command line interface of the SUT. The behaviour of the network nodes is examined by sending commands and analysing printouts, like it is done in the operation and maintenance centres of a network operator. In addition to the black box tests, there are also tests checking single functions of the SUT with respect to internal status information, which can be accessed via the command line interface to the SUT. These tests are implemented as white box tests.

The tests, which are performed, cover the whole range of user interaction and operation and maintenance procedures. This is necessary to check, if operation and maintenance influences or disturbs traffic handling.

Figure 2: Sketch of an example test configuration of the R8 project.

Page 33: Efficient Mobile Network Acceptance Testing

8

For GSM R8 we have concentrated on the regression tests of former releases, because these tests have to be repeated in every Type Acceptance project. For GSM R8, which is the first application of our tool, we have restricted ourselves to the CSS part only. But in the BSS area there is also a huge potential for test automation, which might even be higher than in the CSS, due to the various BTS types and combinations that have to be tested with nearly identical test scenarios. In the framework of GSM R9 we are going to implement automated testing in the BSS area, too.

3.1 Test Objects Selected for Automation

At the moment there are three basic services provided to the GSM customer, which are telephony, circuit-switched data & fax and the Short Message Service (SMS). The SMS enables the user to send simple text messages with a maximal length of 160 characters from the mobile phone to another mobile or the Internet. The basic GSM data service has a maximum bandwidth of 9.6 kbit/s, which has been extended recently by the introduction of High-Speed Circuit Switched Data (HSCSD, see [12]). In addition to these basic services GSM offers, there is a large number of so-called Supplementary Services, which are defined in the GSM recommendations [13]. Services like call forwarding or call barring are just two examples of the Supplementary Services, which can be de-/activated, configured and utilised by either the customer or the network operator (e.g. for barring of outgoing calls for a specific subscriber). The corresponding test object Supplementary Services covers most of the services, which are available for the customers. The test tool performs the typical actions of GSM users and checks the results by monitoring the phones and the behaviour of the network elements (nodes).

All the possible interactions of the customer with these three basic GSM services combined with the Supplementary Services form a large and very important group of test cases in every Type Acceptance. In total 69% of all test cases, which were performed during the acceptance test for GSM R8 CSS were related to basic telephony, the short message service or data/fax calls.

As the NEXT/AIMS tool platform was especially designed to perform these types of tests without human interaction, we have managed to automate a large fraction of them. In total, 47% of all test cases which were executed in the acceptance test for GSM R8 CSS were performed fully automatically, and 92% of these test cases contained either voice, fax or data calls or short message transmission.

Especially the test object mobile fax and data transmission has proven to be an ideal candidate for automated testing, because there are various modes to operate the data service all of which have to be tested. This results in a set of test cases that have to be repeated several times with different parameter settings of either the terminal equipment or the network. The test tool checks all provided fax and data services, e.g. all data rates, connection elements and compression modes used for all possible call types (mobile originating, mobile terminating and mobile- mobile calls). There are analogue (PSTN) and digital (ISDN) modems and terminal adapters used to test the whole range of equipment, which is used by the subscribers. In total, 33% of the test cases we performed automatically were either data or fax test cases.

The short message service (SMS) has become more and more popular among the customers of mobile operators. The test object Short Message Service tries to cover all traffic scenarios related to the point-to-point SMS1. Examples for basic scenarios are sending messages from one mobile phone to another one, which are received immediately, sending multiple messages, and receiving messages after attaching the mobile station. Short messages are transmitted in a signalling channel of the Common Channel Signalling System No. 7 (see [14]). For this reason, testing the SMS sometimes requires to analyse the signalling information, which is exchanged between the network and the mobile station. At the moment this is not possible with the NEXT platform, which provides no protocol analyser functionality yet. Therefore, the protocol information was checked manually by using commercially available protocol analysers. Compared to the fax and data test object, the set of SMS test cases is quite small (less than 5% of all automated test cases). 1 The point-to-multipoint SMS transmission was not included in our test automation program yet. This scenario will be implemented with the next release.

Page 34: Efficient Mobile Network Acceptance Testing

9

Although, the SMS traffic volume in the network has increased tremendously during the last three years, the telephony service is still the most important service offered by today’s mobile network operators. For this reason, all the different aspects of voice services are tested very carefully by Mannesmann Mobilfunk for any upcoming software release. One of the most simple test objects is Basic Call Handling, which tests the fundamental GSM call scenarios for different subscribers who are roaming through the test network.

For a GSM network operator it is not sufficient to verify that all the services work very well. It is essential that also the charging functionality works properly. In GSM, the charging is done by the operator’s billing platform, which processes the call records written by the MSC. These call records contain the essential information for each call set-up in a specific MSC area (regardless whether it is voice, data or fax), for example the A- and B-party numbers, the starting time and the call duration. The records are stored on a hard disk of the Input/Output Group (IOG) of the MSC and are periodically transferred to the billing system. In the Charging test object, all relevant call types are performed fully automatically by the tool platform and the corresponding call records are saved. As NEXT neither has the ability to check if the generated call records are correct, nor has an interface to the operator’s billing platform, the call records still have to be analysed manually or transferred to the billing system where the post-processing is made.

In order to be able to plan the capacity of the network, the operator needs detailed information about the traffic volume carried by each network node and the amount of signalling messages processed by the signalling network. The Ericsson switch provides this information by a large number of counters, which measure the individual events relevant for the later analysis. The test object Statistics and Traffic Measurements checks whether these counters are correct and can be relied upon. For this purpose, a large set of call scenarios is performed fully automatically by NEXT/AIMS. The corresponding traffic measurements are stored by the MSC and transferred for post-processing. As NEXT/AIMS is not able to do the post-processing itself, the data still has to be checked by the testers.

After a new release was tested successfully, it has to be rolled out in the whole network. Usually, the BSC is not updated in the same night as the MSC. Therefore it is necessary to check whether the new MSC release is compatible to the old BSC release and vice versa. When upgrading a network, it is neither possible nor achievable to update the whole network in one step. In consequence, there is a period where both release versions are used. The Interworking test object proves that all services can interwork between all network elements.

As described above, NEXT provides a command line interface to the system under test (e.g. the MSC), the so-called TENUX interface. Via TENUX it is possible to execute maintenance commands on the SUT or to obtain result printouts, which enables us to perform also test objects which are beyond the scope of simple call tests, for example the APZ Regression Tests. The APZ2 is the core of an Ericsson exchange. It consists of a pair of central processors, at least one pair of support processors and up to 1024 regional processors controlling up to 16 control modules. The test proves central functions like software upgrades, hardware replacements and alarm reception. Since all parts of an AXE (the hardware platform all Ericsson GSM nodes are based on) are redundant, nearly all maintenance works should be possible without traffic disturbances.

Although we have reached a grade of automation of 57% for the regression tests that were performed in the acceptance test of the release GSM R8 CSS, there are still a lot of test cases that could not be automated with current release of the tool platform, for example the signalling tests. The signals which are exchanged by the different GSM nodes are rather complex and have to be decoded using a protocol analyser. The most common protocol analysers on the market have no open interface to send commands and receive result printouts. In consequence, the test automation of these tests had to be postponed.

Other tests probably will never be automated at all. These are usually tests which require some kind of external interaction which cannot easily be triggered by a test automaton, e.g. manipulating the hardware of the SUT in order to check if the hardware fault detection and recovery works properly. In principle, one could imagine that in the future even these tests could be automated, too, but it is questionable if the benefit gained by this would justify the high implementation effort.

2 APZ is not an acronym.

Page 35: Efficient Mobile Network Acceptance Testing

10

4. Results from the R8 CSS Project

4.1 Preparation Phase

The GSM R8 activities related to automated testing started in spring 1999, long before even defining the set of test cases to be executed, because adequate time for programming the test cases was needed. The baseline for selecting test cases for automation was the previous release, GSM R7, so the test descriptions from this release were investigated for potential test cases for the GSM R8 automated TAcc. This meant of course, that new features of GSM R8 would not be subject to automated testing, but this restriction was in accordance with Mannesmann Mobilfunk’s and Ericsson’s plans to only apply automated testing in regression tests. On the one hand, this means that test descriptions are already available as instructions for the designers of automated test cases. On the other hand, human testers got the opportunity to get acquainted with the new features by manually testing them, which will help in programming automated test cases for the subsequent release.

A team of EDD3 and MMO4 test experts thus walked through all the test cases of the former GSM R7 TAcc and identified a large number of test cases that could be automated with the existing NEXT/Aims test automation platform. In relative numbers, based on the final selection of test cases used in the R8 TAcc (including the new features), 52% of the test objects had no automation potential at all. 24% could be even completely automated, with a remaining 24% of test objects with a mixed population of automated and manual tests. In terms of test cases, the R8 TAcc had a 20% fraction of new test cases (that were excluded from test automation by edict) and an 80% fraction of regression tests. From the regression test cases, 59% could be automated, while 41% had to be performed manually. In terms of the total number of test cases, this amounts to 47% test automation vs. 53% manual testing. This limit was set mainly by the number of features supported by the test platform but also the nature of some test cases as mentioned above.

The programming activities for the automated tests began in October 1999. Initially, the progress was not directly visible as the test case designers at EDD first built a set of modules that would be used later by the individual test cases. Therefore, the test case development started very slowly. The result, however, was a super-generic test case implemented with NEXT that was turned into an individual test case by just feeding it with the right parameters, which vastly accelerated the implementation of test cases. It was stated by the designers that coding (not including testing!) such a test case was in the order of magnitude of 15-30 minutes (although we have taken no measurements on this). However, coding was the least time consuming activity during the preparation phase. Test descriptions had to be read and understood (some needed to be completely rewritten, as they were too cryptic). Test code needed to be verified itself in the test plant, which usually caused a significant overhead for configuring the test plant and the automated test platform properly. Disturbance by parallel testing activities was a major problem. Sometimes, manual testers reloaded the switch software or re-plugged network connections while an automated test suite was running, which of course led to a total failure. Adding to the problems was the replacement of some of the switches with new hardware components that were not all received and installed by the original schedule. Proper resource scheduling and communication between the different test teams can definitely be improved in future automated TAcc projects.

But also the test platform developed at another Ericsson subsidiary, EED5, caused some setbacks. Parsing the MML-printouts of the switches proved to be difficult, as the thousands of possible printouts were originally meant for human testers rather than automated analysis. Thus, quite often a newly encountered printout required a modification of the parser. Finally, the new fax/data platform was not ready on time and had a few bugs that were not detected at EED due to the limited capability of the system test plant at EDD with respect to fax/data features.

Figure 3 shows the progress in coding and testing according to the weekly reports by the test case design team. As stated above, the design of test cases started with a delay due to the common modules being 3 Ericsson GmbH, Düsseldorf, Germany 4 Mannesmann Mobilfunk GmbH, Düsseldorf, Germany 5 Ericsson Eurolab Deutschland GmbH, Herzogenrath, Germany

Page 36: Efficient Mobile Network Acceptance Testing

11

developed first. Progress stagnated around the turn of the year due to the holiday season. The speed of coding then picked up rapidly with a very sharp increase in week 8. This was due to the fax/data test cases, which encompassed roughly a third of all the test cases, and which were only variations of 4 basic test cases

with different parameters (transmission rates etc.), so that they could easily be obtained by copy-pasting from the initial test cases with a few minor modifications. In the end, the more difficult test cases were implemented, and the speed of coding slowed down again. There was even a reduction in the number of test cases (with the 100% mark being based on the final set), as a few test cases were taken out of the R8 TAcc (features not supported).

It was also found that some test cases that were already coded had duplicates in other in test objects, so that they could be removed. Figure 4.1 also shows the progress in verifying the TCs, which was initially hampered by the above mentioned problems with test plant scheduling. The graph also shows a small drop in the number of test cases in week 16, as MMO demanded a change in the way the fax/data test cases were verified, so that the affected test cases needed to be re-tested. The reports ended in week 21, short of the verification reaching 100%, as a few test cases could not be run successfully and were taken out of the TAcc. It was later found that this was in fact a problem of the test plant configuration rather than the test platform.

Not plotted is the validation time, which was spent both by MMO and EDD employees. Validation means that the MMO-experts are demonstrated the correct behaviour of the test cases. Due to the large number of test cases and the limited availability of system test plant (STP) time as well as of the MMO experts, most of the test cases were validated by EDD handing logfiles to MMO.

Not taking into account the initial creation of modules, which was a one-time activity, the average effort to implement, verify, and validate one test case was close to 7:50 hours or roughly one working day. This includes the whole of the daytime activities that were done during the test case development. It also includes the move from GSM R7 to GSM R8, which required modifications of some test cases and re-testing of all those that were not directly implemented on GSM R8. As mentioned, the coding work only comprised less than half an hour of this time. No matter what language is used for implementing the test cases, a significant overhead can be expected in practice for similar projects, considering the first application of automated testing and the related problems.

4.2 TAcc Phase

Then came the busy phase of the TAcc itself. In theory, one could create a huge test suite of all the automated test cases and run it on the system under test. Reality was quite different, though. During the TAcc period, there were more than 50 executions of test suites that varied in length from a single test case to a whole test object. The main reason was that the system was reconfigured many times during the type

Implementation Progress

0%

20%

40%

60%

80%

100%

120%

43/9

9

45/9

9

47/9

9

49/9

9

51/9

9

01/0

0

03/0

0

05/0

0

07/0

0

09/0

0

11/0

0

13/0

0

15/0

0

17/0

0

19/0

0

21/0

0

Calendar Week

Coded TCs

Verif ied TCs

Figure 3: Progress during test case development

Page 37: Efficient Mobile Network Acceptance Testing

12

acceptance and it was never possible to run but a few test objects on one particular configuration. Configuration changes could not be avoided, on the other hand, as the test plant resources were scarce and so it was not possible to set up a generic configuration that would suite all the test objects equally well.

4.21Testing Efficiency

It is quite interesting to consider the hours, during which the automated tests were run. Figure 4 plots the relative number of test cases that were started during each of the daytime hours. While there is a clear peak centred in the early afternoon, as would be expected for office activities, the remainder of the time was used approximately evenly for running the test cases. Thus, automated testing is clearly a 24h activity. Tripling the available time for testing by using the test resource around the clock is the major advantage of test automation.

4.22 Speed of Execution

How long do test cases run? Figure5 provides an idea. For most of the automated test objects, execution times are plotted showing the fastest, the average, and the longest execution time. The overall values are

shown at the outer right. The execution times vary in a wide range, and contrary to the expectations of a layman, they are in the order of minutes to hours rather than seconds. The limiting factor here is the system under test, not the execution speed of the test platform. Some Charging test cases require less than a minute to be executed (make a call and check the charging information), while a certain Interworking test runs for more than 5 hours, as it shall test whether a call can be performed with such a long duration. The average test case takes in the order of 15 Figure 5: Execution times vs. Test Objects

Test Case Execution Times5:04:22 5:04:22 5:04:22

0:00:000:05:000:10:000:15:000:20:000:25:000:30:000:35:000:40:000:45:000:50:000:55:00

Annou

ncem

ts

APZ Reg

ressn

Call Barr

ings

Basic

Call H

nd

Chargi

ng

Fax/D

ata

Fraud P

revntn

Interw

rk R7/R

8

Intrw

MSC/BSC

Statist

. Sub

stm.

Overal

l

Test Object

Minimum

Mean

Maximum

Execution Time Distribution

0%

2%

4%

6%

8%

10%

12%

14%

00:0

0-01

:00

02:0

0-03

:00

04:0

0-05

:00

06:0

0-07

:00

08:0

0-09

:00

10:0

0-11

:00

12:0

0-13

:00

14:0

0-15

:00

16:0

0-17

:00

18:0

0-19

:00

20:0

0-21

:00

22:0

0-23

:00

Figure 4: Distribution of TC Execution over the day

Page 38: Efficient Mobile Network Acceptance Testing

13

minutes. A human tester is not much slower, here, but first, the automated testing runs without any human attendance at all, while manual testing usually involves at least one or two Ericsson testers and an MMO auditor. Secondly, as we have seen above, the automated test may run during nighttime hours and over the weekend. The MMO auditor only needs to review the logs, and he will only have a closer look at the logs of test cases that end with the attribute “FAILED”.

4.23 Quality of Automated Testing

It is very difficult to compare the quality of the automated and manual test in terms of the detected faults. On one hand, automated testing is more sensitive to faults, as it pays attention even to small deviations e.g. of the pulse duration of the busy tone. On the other hand, the automated test cases were based on manual tests from the preceding TAcc, so the potential to find errors was the same. Or not quite the same, as only regression tests were automated, that is, test cases for features that were already used in the last software release so that most of bugs should have been fixed meanwhile. The result was as follows: from all the detected unspecified behaviours, automated testing found 34%. As we have seen, the fraction of automated test cases was 47% of all the test cases. One might have expected a similar fraction for the detected unspecified behaviours, but as manual testing was more focussed on the new, error-prone features, this expectation is not met. The only conclusion that can be drawn from these figures is that automated testing does in fact find faults, but whether it is better or worse at finding them compared to manual testing cannot be decided on the basis of the available data.

4.24 Cost Comparison

If the execution speed of automated tests is not that high, have there been any savings? Figure 6 answers this with a clear yes. The leftmost three bars indicate the relative durations of an automated test case compared to a manual one as measured in test plant usage (STP hours), manhours, and TAcc duration hours. The rightmost bar compares the actual duration of the automated and the manual TAcc testing parts.

These figures need some more elucidation. The STP hours are a plain comparison on a per test case basis, how long the execution of an automated test case is compared to the manual execution. To obtain the manual figure, which is normalised to 100% here, the number of manually executed TAcc test cases was divided by the number of office hours. The figure for the automated testing is based on the average length of a test run according to the test logs. It is hard to compare the figures on an equal basis as the time for configuring the test plant and putting together the test suites was not measured here. But the trend is clear: the manual test plant usage is in the same order of magnitude as the automated test plant usage. The system

under test deter-mines the speed of execution, not the tester or test machine.

The manhours per test case were obtained by multi-plying the above figure by the mean number of persons involved. In the case of manual testing, these are at least two, one from MMO and one from Ericsson. In the case of automated testing, there only

needs to be one person involved in putting together the test suite and starting it. It was estimated that a tester

Relative Costs Comparison

66%

3%16%

34%

0%

20%

40%

60%

80%

100%

STP hrs/TC mhrs/TC TAccduration/TC

total R8 TAccduration

automated Testmanual Test

Figure 6: Costs of automated vs. manual testing

Page 39: Efficient Mobile Network Acceptance Testing

14

only needs to be attending the tests for 1/10 of the time. Therefore, the relative amount of man-time for automated testing is vastly smaller than for manual testing.

What do the testers do during the execution? Mostly, they are at home and possibly sleeping. Many of the automated tests were performed outside the usual office-hours. This is shown in the third bar. The automated testing figure just represents the average duration of an automated test case, while the manual 100% bar is based on the actual duration of the TAcc, including weekends and none-office-hours. As manual testing is limited to office hours, the time efficiency is roughly 4 times lower than for automated testing, and so the 66% from the STP hour bar become 16% when the actual TAcc duration is considered.

This was based on a per test case basis. The rightmost bar shows the actual relation between the manual R8 SSS TAcc, based on the number of TAcc days, and the automated one, based on the sum of all the periods during which test cases were run (including repetitions and periods of less than two hours between test executions to account for preparation times). The numbers of executed test cases were roughly equal, so the bars should have the same length. However, due to the more efficient resource usage of automated testing, there is a threefold increase in speed compared to manual testing. And this was achieved with a very small team of testers, compared to a much larger team of testers involved in manual testing.

How about the effort spent on designing the test cases? We have no exact figures on the amount of hours spent for preparing the manual tests, but roughly, the automation of the tests required about the same amount of hours or at most twice as much as preparing the manual tests. However, one should not base a cost comparison on these figures, as this was the first set-up of a huge database of automated tests. This initial effort occurs only once for automated testing. After that, only the new test cases from the former TAcc which enter the regression test pool need to be automated. We have seen that this fraction is only about 20% of a TAcc with a potential for automation of about 50%. Thus, only 10% of the new test cases will need to be automated for the next phase, as opposed to 47% that were automated within this project. Hence, the effort for the next phase will definitely be smaller with automated testing than with manual testing. More gain is expected, when test cases from earlier phases (system test etc.) can be taken advantage of, and when the test case database is not only employed during TAcc, but also for testing of correction releases of the GSM software.

4.3 Additional Benefits

There are a number of benefits of automated testing that cannot be easily measured, but which should nevertheless not be forgotten. One of them is the reproducibility of automated test cases. In our investigations at Ericsson, we have found that some faults only occur under certain timing conditions. A manual tester may by chance find such a fault, but when he tries to verify, whether he just hit the wrong key on the mobile or whether he really encountered a fault, he may not be able to reproduce the same behaviour due to a slightly different timing. With automated testing, the execution is timed very precisely and hence time critical faults are reproduced with much better accuracy.

Furthermore, automated testing does most of the boring tester tasks. Writing reports and executing test cases that mostly pass is not very challenging. Automated testing only leaves the identification of the source of a fault to the testers, which not just improves his or her efficiency, but also creates a more motivating working climate. The same is true for the test case designers, who worship programming more than writing test instructions.

Another advantage of automated testing is that the skill of the persons running the test suites doesn’t need to be high. Setting up a test suite and running it is a task for a student or a newly employed person. Experts are only needed for programming the test cases. If the test instructions are detailed enough, they even only need to be consulted occasionally by the test case designers, which are more software designers than testers. Experts in GSM are more difficult to find nowadays, so automated testing can help lessen the strain on these valuable resources as well.

Besides benefits, there is also one drawback, though. A very important factor for automated testing is trust, which has to be gained from the testers. When test cases would not pass during the preparation phase, testers not involved in test case programming always blamed the test platform first. In nearly all the cases,

Page 40: Efficient Mobile Network Acceptance Testing

15

the fault could later be traced to the system under test, but this had to be demonstrated by executing the same test case manually. The test platform was actually running very stable with only two crashes in 2 months. It may take a while of practical experience until all testers fully acknowledge the benefits of test automation.

5. Conclusions In the Type Acceptance project for the release Ericsson GSM R8 CSS we have successfully used the NEXT/AIMS tool platform to automate 47% of the test cases that were executed. As the automated test cases can run fully automatically with only a small human effort required for starting and supervising the test campaign, we have therefore saved considerable human resources during test execution. Of course, as it was the first application of the tool platform, a large effort had to be spent for implementation, testing and validation of test cases. But once programmed, the test cases can be used in all the following acceptance tests and the initial investment pays off. Furthermore, we have reduced the length of the Type Acceptance execution phase by 30%. Therefore automated testing contributes to a reduction of the time-to-market for a new release.

However, we have learned that automated testing did not speed up the test case execution significantly. Although the tool itself is able to execute test cases much faster than any human being, the limiting factor is the SUT, which has to react to the actions initiated by the test case, e.g. by executing commands or sending result printouts. Therefore the automation gain lies in the fact that automated test cases can run unattended by the tester, which increases the efficiency of test plant utilisation. However, the stability of the test configuration has turned out to be critical for test automation. A severe configuration fault, which was not detected before the test started, can easily ruin the whole test campaign, which is normally detected in the next morning. The same statement holds true for manual testing in parallel to a running automated test campaign, which therefore should be avoided. If you keep this in mind and organise the test sessions in a way that automated testing can run undisturbed by other activities in the test plant, test automation can be a great success for a company.

6. References

[1] S. H. Redl, M. K. Weber, M. W. Oliphant: “An Introduction to GSM”, Artech House, 1995

[2] M. Mouly, M. B. Pautet: “Current Evolution of the GSM System”, IEEE Personal Communications Magazine, October 1995

[3] ETSI specification GSM 03.60: “Digital Cellular Communications System (Phase 2+); General Packet Radio Service”

[4] The UMTS Forum: “The Path towards UMTS – Technologies for the Information Society”, UMTS Forum Report No. 2, 2000

[5] Dieter Kreuer, Simon Hoff, Jürgen Sauermann: “Facing the Challenge of Growing Telephone Network Complexity with Automated Testing,” Proc. EuroStar 98 conference, Munich, 1998.

[6] Dieter Kreuer: “Applying Test Automation to Type Acceptance Testing of Telecom Networks: A Case Study with Customer Participation,” Proc. Automatic Software Engineering ASE 99, Cocoa Beach, FLA, USA, Oct. 1999.

[7] R.C. Linger: “Cleanroom Software Engineering for Zero-Defect Software,” Proc. of the 15th International Conference on Software Engineering, IEEE Computer Society Press, Baltimore, 1993.

[8] J.D. Musa: “Operational Profiles in Software Reliability Engineering,” IEEE Software, March 1993, pp. 14-32.

[9] J.A. Whittaker M.G. Thomason: “A Markov Chain Model for Statistical Software Testing,” IEEE Transactions of Software Engineering, October 1994, pp. 812-24.

Page 41: Efficient Mobile Network Acceptance Testing

16

[10] Michael R. Lyu (Editor), Handbook of Software Reliability Engineering, McGraw-Hill, 1995, pp. 756-57.

[11] ITU-T Recommendation X.290: “OSI Conformance Testing Methodology and Framework for Protocol Recommendations for ITU-T Applications”

[12] ETSI specification GSM 02.34: “Digital Cellular Communications System (Phase 2+); High Speed Circuit Switched Data – Stage 1”

[13] ETSI specification GSM 02.80: “Digital Cellular Communications System (Phase 2+); Supplementary Services – General Aspects”

[14] ITU-T Q.700: “Introduction to ITU-T signalling System No. 7”

Page 42: Efficient Mobile Network Acceptance Testing

Thursday 7 December 2000

T7 Efficient Mobile Network Acceptance Testing Jorg Paul

Jorg Paul graduated in physics at the Aachen University of Technology in 1995. At University he mainly worked in the fields of theoretical elementary particle physics and computational physics. In 1996 he joined the German GSM network operator Mannesmann Mobilfunk. Since then he has been working in several software testing and test automation projects. He is now project leader for the introduction of automated testing into the joint type acceptance project GSM R8 of Mannesmann Mobilfunk and Ericsson.