Requirement-based Testing

A Presentation by

Harry Sneedfor the

John von Neumann Society Budapest, 27 May 2009

The many Dimensions of Software Testing

Test Objects• Units (Modules/Classes)• Components• Subsystems• SystemsTest Types• Functional Testing• Performance Testing• Load Testing• Usability Testing

Test Approaches• Random Testing• Creative Testing• Systematic TestingTest Bases• Code-based testing• Interface-based testing• Reference-based testing• Model-based testing• Require.-based testing

Software Units

ComponentsSystems

Functional Specification

(Oracle)

In 1978 Bill Howden introduced the notion of a test oracle in connection with Functional Testing. It means testing the System against the Requirement Specification (functional & non-functional Requirements)

A Test is always a Test against Something

Oracle

CodeTest inputsTest results &

Actual paths

SpecificationExpected results Expected paths

Potential Paths

SpecifiedPaths

compare Results & Paths

Corresponds to a Unit Test (White-Box)

Code-based Testing = Testing the Program against itself

CodeTest inputs Test results

Specification Expected results

compareData

Representative Values

Interfaces/Files/Tables

Specified Results

Interfaces/Databases

Corresponds to an Integration Test (Grey-Box)

Data-based Testing = Testing the Program against the Data

Interfaces/Files/Tables

CodeTest inputs Test results

Specification Expected results

compare

Files/Tables/Panels

Specified Results

Corresponds to a System Test (Black-Box)

Results

Files/Tables/Panels

Specified Inputs

Requirement-based Testing = Testing the Program against the Specs

StaticAnalysis

ExecutionTest

Verf. Cond.Generator

SyntaxAnalysis

SyntaxErrors Semantic Errors Exec Errors Struct Errors

Graph Checking+

Call Checking+

Units Consistency+

Mode Checking+

Asserted/Actual Use

+Set/Use Checking

Test Results+

Execution Errors

+Coverage

Report+

Assertion Exceptions

Annotated Listing

+VC'S

+Symbolic

Execution of Expressions

+Simplified

VC'S

Indented Program Listing

+Diagnostics

Correct CorrectCorrectCorrect

SourceProgram

Programwith Assert

"Correct"Source Prog.

Assertions taken from Specs(RSL)

The Mother of all Test Tools = RXVPRequirement Evaluation & Verification Package

FORTRAN

Dr. Edward Miller from General Research Corporation

The idea was to put dirty software into a washing maschine and get out cleaned software .

P1

P2

P3

P4

Entry

Exit

Test Object = Procedure or Module

Test Case = Path thru Test Object

1. Path

2. Path

3. Path

5. Path

Test Cases were extracted from the Code

4. Path

PreconditionsPostconditions

BeforeState After

State

The Budapester Testlabor

• The Budapester Test Laboratory was set up by Harry Sneed and Dr. Ed Miller from Software Research Inc in cooperation with the SZKI and Szamok in May, 1978. This was the first known attempt to outsource the testing activity.

• The customer was Siemens, Munich, which was developing a large scale, distributed software system for the German Railroad – the Integrated Transport System (ITS).

Test Process in the Test Labor

• Compilable Modules were delivered by Siemens every week on a magnetic tape.

• Sources were statically analyzed to produce flow and structure diagrams.

• Paths were marked thru the flow diagrams with magic markers.

• Conditional variables were assigned values to confirm or negate every decision.

• Values were assigned at test time to the test driver before each test case execution.

• Results were logged to be checked later against the expected results.

Experience with the Test Labor

• In the first half year 128 modules with 60881 SPL statements passed through the test labor.

• 4378 code branches were tested with a coverage of 89.7 % by 1544 test cases.

• 192 program errors were discovered via dynamic analysis.

• 1396 design and code deficiencies were discovered via static analysis.

• The costs of the quality assurance remained below 4 DM pro statement. This was less than 10% of the development costs.

Summary of first 6 Months

Comp Modules Stmts Testcases

Branches Coverage Design &Code Defis

ProgramErrors

A 6 4029 138 5

B 37 7588 232 603 91% 130 22

K 71 40735 1064 2843 87.5% 868 143

N 6 2847 101 140 94% 110 14

S 8 5682 147 792 95% 150 6

Total 128 60881 1544 4378 91.9% 1396 190

Resume on Module Testing

• In one year 281 Modules with 134 K stmts were tested with 3864 test cases at a cost of DM 75,- per test case = DM 289.800

• The minimum branch coverage was 85%• 403 Errors were reported at a cost of DM 150,- per error =

DM 60.450• Later in the system test by the customer, another 386

errors were uncovered.• The intensive module test had uncovered only 51% of the

total errors found.• This demonstrates the fact that module testing of the code

against itself is not sufficient. The software as a whole must be tested against the requirements.

StoreMeasure

R-Net

ReceiveSignals

ASSERT

ASSERT

ReportEngine ok

ASSERT

CheckMeasure

ReportSignalError

ASSERT

sendAlarm

ASSERT

Invalid Measure Valid Measure

Validation Points

MonitoringEngine

Safe Range Out of safe Range

RXVP progressed to test the systems against Requirements Nets

ReceiveSignal

Measuringdevice

&

StoreMeasure

Message Log

CheckMeasure

Report ok AlarmNurse on Duty

AlarmDisplay

ReportMeasure Error

SystemDisplay

3. test case

2. test case

1. test case

R-Net-Entry

invalid MeasureValid Measureand

or

or Engine problemEngine ok

Engine Monitoring System

Requirement Specification

with RSL

Extracting Test Cases from Requirement Nets

Action DescriptionIf Account_Balance minus

the withdrawal is morethan the credit limitallow the withdrawal

In case the withdrawalcauses the Account_Balanceto fall below the creditlimit refuse the

withdrawal.

Account_Balance minus the withdrawalis more than the credit limitWithdrawal causes the Account_Balanceto fall below the credit limitAllow the withdrawal

Refuse the withdrawal

Test Case 1 2 3 4

Y Y N N

Y N Y N

- X - -

X - X -

Service Function Test Case Generation

Conditional keywords arescanned.

Each Condition is treatedseparately and combinedwith the other conditionsin the same action to givethe number of testcases = nr_conditions2

Extracting Test Cases from Business Rules

One Use Case can have one or more Test case

Normally, there are at least 2 Test Cases

Positive outcome

Negative outcome

CashWithdrawal

Not enoughcash

Accountoverdrawn

Cashdelivered

WrongPIN Number

InvalidCard

Use Case =

Test Cases = Use Case Paths

Extracting Test Cases from Use Case Descriptions

Test Cases

Functional

Non-Functional

Objects

Actions

States

Conditions

Requirements Text

Extracting Test Cases from Requirement Documents

Actions upon objects:The customer account should be deleted.The invoice should be sent within 30 days.

Object states:The customer account is overdrawn.The customer is an employee.

Rules:If the account is overdrawn a notice should be sent.Whenever someone fails to pay within 30 days, a reminder should be sent.Customer accounts should be deleted, if they are overdrawn more than 3 months.In case of a system crash, the recovery routine should be started.

Grammar types indicating Declarations to be tested

[RF 41] Notification can be divided into action and non-action (=information) notification.

[RF 42] In case of an “action needed” notification, the system provides a link to the place where the action is needed.

[RF 43] Notifications are added to a notification interface, which contains information aboutThe reason of the notificationThe kind of the notificationThe type of the notificationThe date/time of the notification

If action needed: A link to the place where the action is needed.

[RF 44] The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”. If the action is finished, the user name is stored by the notification.

[RF 45] If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF 43]) so that each user can see the notification. The notification is only added to the notification table if the data is unequal to the current event data (see also [RF 30]).

Sample Requirements Text

TestCase-Id BaseType Requirement Relation TestType TestCase Description--------------------------------------------------------------------------------------------------------------------------------------------------------DFI0163 Require RF_43 perform action Notifications are added to a notification interface, which contains information about:

The reason of the notificationThe kind of the notificationThe type of the notificationThe date/time of the notification

DFI0164 Require RF_43 checks rule If action is needed a link to the place where the action is needed.--------------------------------------------------------------------------------------------------------------------------------------------------------TestCase-Id BaseType Requirement Relation TestType TestCase Description--------------------------------------------------------------------------------------------------------------------------------------------------------DFI0165 Require RF_44 perform action The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”.DFI0166 Require RF_44 checks rule If the action is finished, the user name is stored in the notification.--------------------------------------------------------------------------------------------------------------------------------------------------------TestCase-Id BaseType Requirement Relation TestType TestCase Description--------------------------------------------------------------------------------------------------------------------------------------------------------DFI0167 Require RF_45 checks rule If at least one provider changed his feed data changing starting

time, a notification is added to the notification table (see [RF_43]) so that each user can see the notification.

DFI0168 Require RF_45 checks rule The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]).DFI0169 Require RF_45 checks rule The user choose Provider A as primary source, so the event time is the time of Provider A.DFI0170 Require RF_45 checks rule If Provider B updates his time to the time of the event, no notification is added.DFI0171 Require RF_45 checks rule If Provider B sends a time different to the event time, a notification is sent.

Test Cases extracted from the Requirement Text

The order entry system must be able to handle at least 1000 customer web pages at one time. A customer order must be processed within 2 seconds. The article database may contain up to 10.000 articles. The system should be able to produce as many as 25.000 bills in one billing run. A billing run should not take more than 2 hours. If the system crashes it should be back up and in operation within 15 minutes. Database backups should take place every 4 hours. Customers should be checked both by number and by password. Passwords should be rejected after 3 tries, customer number should be rejected after 2 tries.

Sample Non Functional Requirements

1. 999 customer orders at one time

2. >1000 customer orders at one time

3. response time per order < 2 sec.

4. response time per order >2 sec.

5. 9.999 article records

6. > 10.000 article records

7. 24.999 bills in one billing

8. >25.000 bills in one billing

9. run time of billing < 2 hours

10. run time of billing >2 hours

11. If system crash, recovery < 15 minutes

12. If > 4 hours, database secured

13. Submit 3 passwords where 3rd password is correct

14. Submit 3 passwords where all 3 are incorrect

15. Submit 2 customer number where 2nd number is correct

16. Submit 2 customer number where both are incorrect

Typical Non-Functional Test Cases

A Test Case:

has a unique identifier serves a purpose refers to a Testobject refers to a requirement or use case has a precondition state has a postcondition state has a set of required input parameters has a set of expected results has a predecessor test case has a given environment has an owner has a status has a date of execution

Properties of a Test Case

An international standard is required for the minimumProperties of test cases!!!

TestCase ID: IC-Order-7Test Case – Purpose: to test a fulfilled orderTest-Object: Customer – OrderPrecondition: Ordered Article is on stock.

Article. Amount is > Order.Position.Amount(Article. Amount - Order. Position.Amount ) > minimum – Amount

Postcondition: Article.Amount = a Pre.Article.Amount – Order.Position.AmountNo supply order

Inputs: Order.Cust-Nr = 200100Customer.Credit = ’A’Order.Position-Nr = 2Order.Position.Art – Nr = 4711Order.Position.Amount = 3Article.Amount = 10Minimum – Amount = 7

Results: Text-Msg = „Article order fulfilled”Environment: MS – Internet, Windows – XPUsecase tested: Customer.Order.ProcessingStatus: testedDate: 29. 8. 2005.

Functional Test Case Specification

TextAnalysis

TestCases

RepositoryLoader

Test CaseDB

RepositoryUnloader

TestScriptsTest

Testerenhances & refinesTest Cases

TesterexecutesTest Cases

Text AnalyzergeneratesTest Cases

Test Script Generatorgenerates Test Scriptsfor Test Automation

From Requirements via Test Cases to Test Scripts

Text

Processor

TxtAnalTextPre

Processor

TxtScan

Object

List

ObjectModel

TestCases

RuleViolations

SpecMetrics

RepositoryEntries

Objects- Use Cases - Actions - States - Rules

TextExcelXML

Format RulesRupp Rules

SizeQuantityComplexity

Sections- Use Cases - Objects - Actions - States - Rules

TextAnal = Tool for extracting Test Cases from English and German Language Documents

Requirements Analysis Demo

• Bet&Win Requirement Document• Setting Key Words• Scanning for Objects• Analyzing the Document• Extracting Test Cases• Collecting Requirement Metrics• Documenting Cross References• Generating a test design

Egovernment Web Site with three applications and circa 33 K Java Stmts

1844 Test cases were specified of which 1272 were generated

1495 Test cases were executed = 81% Test Case coverage

352 Test cases discovered at least one error = 24% Hit rate

450 Errors were reported by testers = 13 per KDSI

56 Errors were reported by users after release = 11% of all errors

System test discovered 89% of all reported errors = test efficiency

System test lasted 4 months from January to May 2005

392 Test days were booked against the project

= 3.8 Test cases per Testertag

= 1.1 Errors reported per tester day

Customer was satisfied with the test result.

Egovernment Web Site with three applications and circa 33 K Java Stmts

1844 Test cases were specified of which 1272 were generated

1495 Test cases were executed = 81% Test Case coverage

352 Test cases discovered at least one error = 24% Hit rate

450 Errors were reported by testers = 13 per KDSI

56 Errors were reported by users after release = 11% of all errors

System test discovered 89% of all reported errors = test efficiency

System test lasted 4 months from January to May 2005

392 Test days were booked against the project

= 3.8 Test cases per Testertag

= 1.1 Errors reported per tester day

Customer was satisfied with the test result.

Experience with Requirement-based Testingof the Public Internet Portal for the state of Saxony

Conclusions

• Test methods have not really improved since the 1970s.

• We are still using the basic approaches introduced back then.

• Test tools have however improved significantly.• Today there are more and better testing tools.• The future lies in test automation.• Only through automation can we really reduce test

costs and increase the test ROI.• Test ROI = (TestBenefits – TestCosts) / TestCosts