bridging the gap between logical requirements-based test cases and physical test data generation...
TRANSCRIPT
Bridging the Gap between logical requirements-based Test Cases and physical Test Data Generation
Presentation for Swiss Testing Day
by Harry M. Sneed, ANECON, Vienna
• The unresolved problem of system testing• Requirement-based testing• From Requirements to logical Test Cases• From logical Test Cases to a Test Design• From logical Test Cases to Test Data• From logical Test Cases to Test Results• Generating Test Data• Validating Test Results• Can the Gap between the test concept and test
execution be bridged???
The Unresolved Problem of System Testing
• At the top of system testing there are logical test cases formulated in natural language based on the requirement specifications.
• At the bottom of system testing there are physical test data objects such as HTML pages, XML interfaces and SQL databases.
• The problem in system testing is how to go from the logical test case level to the physical test data, i.e. to bridge the gap between test case specification and test data generation.
SWISS-1
LogicalTest
Cases
DesignDocument
DataAnalysis
DataSchema
<supply_order> <supply_id type = “integer“/> <supply_article type = “integer“/> <supply_name type = “string“/> <supply_amount type = “integer“/></supply_order>
If article_stock<minimum_stock { create supply_order where supply_id = asupply_id + 1; supply_article = article_id; supply_name = article.name; supply_amount = supply.stock * 2;}
RequirementDocument
RequirementAnalysis
TestData
Miracle
Objects &Attributes
The Miracle of System Testing
SWISS-2
Test if when amount on stock exceeds minimum stock level a supply order is
produced
ObjectAssignment
Test DataProcedures
ExtractsLogical
Test Cases
From Requirements to Test DataSWISS-3
Test DataGenerator
LogicalTest
Cases
TextAnalysis
DatabaseSchemata
InterfaceSchemata
GUIDefinitions
ValueAssignment
ValueTables
ExistingData
RequirementDocument
DB TablesInterface
FilesGUI
Inputs
SQL XML/WSDLHTML / XSL
HumanTransactions
WithUser Objects
& Preconditions
PreAsserts
SQL XML HTML
Input Objects
Script
English/German Human Interaction
1 2 3
4
5
TesterTester
Steps to Test Data Generation
SWISS-4
The TestAnalyzer extracts logical test cases from the requirement text - one for the object action, object state and object condition.
Tester refines the test cases by adding pre conditions to each test case.The precondition refers to the objects used by the system as inputs.
Tool displays all logical objects from the test cases and all physical input objects from the data model. The tester assigns logical objects to physical objects. Tool generates an assertion procedure for each physical object.
Tool displays attributes of all physical objects. The tester assigns them value domains, specific values, relations or expressions. The tool generates tables of test data.
Tool joins generated test values with existing test data to generate system inputs - GUI contents, data base tables and messages.
1
2
3
4
5
ObjectAssignment
Test DataProcedures
ExtractsLogical
Test Cases
From Requirements to Test ResultsSWISS-5
Test ResultValidator
LogicalTest
Cases
TextAnalysis
DatabaseSchemata
InterfaceSchemata
GUIDefinitions
ValueAssignment
ValueTables
ExistingData
RequirementDocument
DB TablesInterface
FilesGUI
Inputs
SQL XML/WSDLHTML / XSL
HumanTransactions
LogicTest Cases withUser Objects &Post Conditions
PostAsserts
SQL XML/WSDL HTML
Script
English/German Human Interaction
1 2 3
4
5
TesterTester Data Model
Steps to Test Result Validation
SWISS-6
The TestAnalyzer extracts logical test cases from the requirement text - one for the object action, object state and object condition.
Tester refines the test cases by adding post conditions to each test case.The precondition refers to the objects used by the system as outputs.
Tool displays all logical objects from the test cases and all physical output objects from the data model. The tester assigns logical objects to physical objects. Tool generates an assertion procedure for each physical object.
Tool displays attributes of all physical objects. The tester assigns them value domains, specific values, relations or expressions. The tool generates tables of expected values.
Tool compares system outputs with previous outputs and with the expected values.
1
2
3
4
5
Refinement of Test Cases
SWISS-7
Test Case Test Purpose Test Objects Pre/Post Conditions
OrderEntry01 Order should be rejected if customer is not known
Order
Customer
Rejected (post)
Unknown(pre)
OrderEntry02 Order should be rejected if customer credit rating is too low
Order
Customer
Credit Rating
Rejected (post)
Known (pre)
Too low (pre)
OrderEntry03 Order should be rejected if article ordered is not on stock.
Order
Article
Rejected (post)
Not exists (pre)
OrderEntry04 Order should be backlogged, if amount on stock is less than order amount.
Order
Amount on Stock
Order Amount
Backlogged (post)
< order_amount (pre)
=> amount_on_stock (pre)
OrderEntry05 Order should be fulfilled if amount on stock is adequate.
Order
Amount on Stock
Fulfiled (post)
> order_amount (pre)
OrderEntry06 Amount on stock should be reduced by amount of order.
Input on Stock
Amount of Order
- order_amount (post)
< amount_on_stock (pre)
OrderEntry07 When amount on stock is less than minimum on stock level, a supply order is produced.
Amount on Stock
Minimum Stock
Supply Order
< minimum_stock (pre)
> amount_on_stock (pre)
Exists (post)
2
Assigning Logical Objects to Physical Objects
SWISS-8
<supply_order> <supply_id type = “integer“/> <supply_article type = “dec“/> <supply_name type = “string“/> <supply_amount type = “integer“/></supply_order>
Test Case Test Purpose Test Objects Usage
OrderEntry09 When amount stock is less than minimum stock level, a supply order is produced
Amount on stock
Minimum stock level
Supply order
Input
Input
Output
<Article> <Article_id type = “integer“/> <Article_name type = “string“/> <Article_stock type = “integer“/> <Minimum_stock type = “integer“/> <Supply_stock type = “integer“/></Article>
Output Input
3
Assigning Value Domains to Physical Data Fields
SWISS-9
4
Article
Article_idArticle_nameArticle_stockMinimum_stockSupply_stock
Supply_idSupply_article_idSupply_nameSupply:amount
IntegerStringIntegerIntegerInteger
IntegerIntegerStringInteger
Supply_Order
Article_Ids
Begin = 0Internal = +10
Article_Names
BookMagazineCD_Disk
Article_Stock
Minimum = 50Maximum = 9999
Minimum_Stock
Constant = 50
[RF 41] Notification can be divided into action and non-action (=information) notification.
[RF 42] In case of an “action needed” notification, the system provides a link to the place where the action is needed.
[RF 43] Notifications are added to a notification interface, which contains information aboutThe reason of the notificationThe kind of the notificationThe type of the notificationThe date/time of the notification
If action needed: A link to the place where the action is needed.
[RF 44] The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”. If the action is finished, the user name is stored by the notification.
[RF 45] If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF 43]) so that each user can see the notification. The notification is only added to the notification table if the data is unequal to the current event data (see also [RF 30]).
Original Requirements TextSWISS-10
<Requirement id = "RF_43"> <RequirementHeader> <RequType>Function</RequType> <RequOwner>your_name</RequOwner> <RequStatus>defined</RequStatus> <RequLabel>RF_43</RequLabel> </RequirementHeader> <RequirementBody> <RequTextLine> Notifications are added to a notification interface, which contains information about</RequTextLine> <RequTextLine>* The reason of the notification</RequTextLine> <RequTextLine>* The kind of the notification</RequTextLine> <RequTextLine>* The type of the notification</RequTextLine> <RequTextLine>* The date/time of the notification</RequTextLine> <RequTextLine>* If action is needed: A link to the place where the action is needed.</RequTextLine> </RequirementBody> </Requirement><Requirement id = "RF_45"> <RequirementHeader> <RequType>Function</RequType> <RequOwner>your_name</RequOwner> <RequStatus>defined</RequStatus> <RequLabel>RF_45</RequLabel> </RequirementHeader> <RequirementBody> <RequTextLine> If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF_43]) so that each user</RequTextLine> <RequTextLine>The notification is only added to the notification table if the data is unequal to the current event data (see also
[RF_30]).</RequTextLine> <RequTextLine>Provider B has an other starting time than provider A. The user chose Provider A as primary source, so the
event time is the time of Provider A.</RequTextLine> <RequTextLine>If Provider B updates his time to the time of the event, no notification is added.</RequTextLine> <RequTextLine>If Provider B sends a time different to the event time, a notification is send.</RequTextLine> </RequirementBody> </Requirement>
Requirements extracted from the Requirement Text SWISS-11
TestCase-Id BaseType Requirement Relation TestType TestCase Description--------------------------------------------------------------------------------------------------------------------------------------------------------DFI0163 Require RF_43 perform action Notifications are added to a notification interface, which contains information about:
The reason of the notificationThe kind of the notificationThe type of the notificationThe date/time of the notification
DFI0164 Require RF_43 checks rule If action is needed a link to the place where the action is needed.--------------------------------------------------------------------------------------------------------------------------------------------------------TestCase-Id BaseType Requirement Relation TestType TestCase Description--------------------------------------------------------------------------------------------------------------------------------------------------------DFI0165 Require RF_44 checks rule If the action is finished, the user name is stored in the
notification.--------------------------------------------------------------------------------------------------------------------------------------------------------TestCase-Id BaseType Requirement Relation TestType TestCase Description--------------------------------------------------------------------------------------------------------------------------------------------------------DFI0166 Require RF_45 checks rule If at least one provider changed his feed data changing starting time, a notification is added to the notification table (see [RF_43])
so that each user can see the notification.DFI0167 Require RF_45 checks rule The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]).DFI0168 Require RF_45 checks rule The user choose Provider A as primary source,
so the event time is the time of Provider A.DFI0169 Require RF_45 checks rule If Provider B updates his time to the time of the event, no notification is added.DFI0170 Require RF_45 checks rule If Provider B sends a time different to the event time, a
notification is sent.
Test Cases extracted from the Requirement Text
SWISS-12
TestCase Id = "DFI0167" > <Requirement Id = "45" > <RequirementType>Require</RequirementType> <RequirementName>RF_45</RequirementName> <Priority>High</Priority> <Responsible>Analyst</Responsible> </Requirement> <TestCaseInfo> <TestCaseType>rule</TestCaseType> <TestGoal>check:The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]).</TestGoal> <TestEnvironment>MS-Windows</TestEnvironment> <TestObjekte> <TestObject>notification</TestObject> <TestObject>notification_table</TestObject> <TestObject>event</TestObject> </TestObjects> </TestCaseInfo> <TestCaseConditions> <PreceedingCase>DFI0166</PreceedingCase> <Trigger>Notification is received</Trigger> <PreConditions> <PreCondition> the data is unequal to the current event data(see also [RF_30]). </PreCondition> </PreConditions> <PostConditions> <PostCondition> notification is only added to the notification table </PostCondition> </PostConditions> </TestCaseConditions>
Test Cases in XML Format with Objects
SWISS-13
Oracle Data Tables/*==============================================================*//* Table: BASEEVENT */create table DFI.BASEEVENT (ID NUMBER(18) not null,EVENTNAME varchar(250),STARTDATETIME date not null,CANCELLED date,UPDATEABLE int not null,EVENTDATE date not null,PROGRAM varchar(100),ERRORCODE number,ERRORTEXT varchar(2000))/*==============================================================*//* Table: FEEDEVENT */create table DFI.FEEDEVENT (ID NUMBER(18) not null,BASEEVENT_ID NUMBER(18),FEEDEVENTID varchar(100),EVENTNAME varchar(250),STARTDATETIME date,CANCELLED date)/*==============================================================*//* Table: NOTIFICATION */create table DFI.NOTIFICATION (ID NUMBER(18) not null,EVENTNAME varchar(30),EVENTDATE date not null,EVENTSPONSOR CHAR(10))
Physical Objects defined in SQL Database Schema SWISS-14
<xs:element name="Competition"><xs:complexType>
<xs:sequence> <xs:element ref="Stages"/></xs:sequence><xs:attribute name="id" type="IdType" use="required"/><xs:attribute name="sponsor" type="xs:string"/><xs:attribute name="name" type="xs:string"/><xs:attribute name="startDate" type=„date"/><xs:attribute name="endDate" type=„date"/>
</xs:complexType></xs:element><xs:element name=„Round">
<xs:complexType><xs:attribute name="number" type="IdType" use="required"/><xs:attribute name="name" type="xs:string"/><xs:attribute name="startDate" type=„date"/><xs:attribute name="endDate" type=„date"/><xs:attribute name="legs" type="IdType" default="1"/><xs:attribute name=„replaysPossible" type=„boolean" use="required"/>
</xs:complexType></xs:element>
Physical Objects defined in XML Input Data Schema
SWISS-15
BASEEVENT ID
EVENTNAME
START DATETIME
CANCELLED
UPDATEABLE
EVENTTDATE
PROGRAM
ERRORCODE
ERRORTEXT
number(18) not null,
varchar(250),
date not null,
date not null,
int not null)
date not null,
varchar(100),
number,
varchar(2000
NOTIFICATION ID
EVENTNAME
EVENTDATE
EVENTSPONSOR
number(18) not null,
varchar(30),
date not null,
char(10))
Competition Id
Sponsor
Name
StartDate
EndDate
Idtype
string
string
date
Date
Round Number
Name
StartDate
EndDate
Legs
Replayspossible
Idtype
string
date
date
Idtype
boolean
Event BASEEVENT
Notification
Table
NOTIFICATION
Notification Competition
Round
Assignment of Logical Objects to Physical Objects
SWISS-16
file: DFI_Test_02; if ( object = "BASEEVENT" ); assert new.ID = “4711"; assert new.EVENTNAME = “XXXXXXXXXXXXXXXXX"; assert new.STARTDATETIME = “2006-10-07"; assert new.CANCELLED = "2006-10-05"; assert new.UPDATEABLE = "99999999999999"; assert new.EVENTDATE = “2006-10-07"; assert new.PROGRAM = “YYYYYYYYYYYYYYYYYYY"; assert new.ERRORCODE = “77"; assert new.ERRORTEXT = “ZZZZZZZZZZZZZZZZZZ“; endObject; if ( $object = „ComFEEDEVENT" ); assert new.ID = “4712"; assert new.BASEEVENT_ID = BASEEVENT.ID; assert new.FEEDEVENTID = FEEDEVENT.ID; assert new.EVENTNAME = BASEEVENT.EVENTNAME; assert new.STARTDATETIME = “2006-10-07"; assert new.CANCELLED = "2006-10-05";endObject;if ( $object = “NOTIFICATION" ); assert new.ID = “4713"; assert new.EVENTNAME = “XXXXXXXXXXXXXXXXX“; assert new.EVENTDATE = “2006-10-15"; assert new.EVENTSPONSOR = “YYYYYYYYYYYYYYYYYYYY";endObject;End DFIDB;
SWISS-17
Initial Assignment of Values to Database Attributes
file: DFIDB; if ( $Test = " DFI0166 " );
if ( $object = "BASEEVENT" ); assert pre.ID = “4711"; assert pre.EVENTNAME = “Manchester/Liverpool Match"; assert pre.STARTDATE = “2006-10-07"; assert pre.CANCELLED = "2006-10-05"; assert pre.UPDATEABLE = „6856"; assert pre.EVENTDATE = “2006-10-07"; assert pre.PROGRAM = “English football league"; assert pre.ERRORCODE = “77"; assert pre.ERRORTEXT = “Match result not verified“; endObject; if ( $object = „Competition" ); assert pre.ID = BASEEVENT.ID; assert pre.Sponsor = “McDonalds“; assert pre.name != BASEEVENT.EVENTNAME; assert pre.StartDate != BASEEVENT.STARTDATE; assert pre.EndDate = “2006-10-07"; endObject;
if ($object = “NOTIFICATION" ); assert post.ID = Competition.ID; assert post.EVENTNAME = Competition.name; assert post.EVENTDATE = Competition.StartDate; assert post.EVENTSPONSOR = Competition.Sponsor;
endObject; endTest;
SWISS-18
Specific TestCase related Assignment of Values
<TestProcess id = "DFI_Test_02" name = "Automatic_Data_Feed_Test" type = "Batch"><TestObjectives>
<TestObjective>To test the Automatic Data Feed Function</TestObjective> <TestObjective>To test if the XML files from the Providers are processed correctly</TestObjective><TestObjective>To test if the DFI Database is updated correctly</TestObjective><TestObjective>To test if the correct Notifications are sent</TestObjective>
</TestObjectives><TestEnvironment>
<Hardware>Sun-Sparc Server</Hardware></TestEnvironment><TestPrerequisites>
<TestPrerequisite>DFI Database must be initialized with Testdata</TestPrerequisite><TestPrerequisite>One or more XML Files must be generated</TestPrerequisite><TestPrerequisite>DFI System must be put into operation</TestPrerequisite><TestPrerequisite>XML Files must be copied into the system queue</TestPrerequisite>
</TestPrerequisites><TargetRequirements>
<TargetRequirement>RF-01-Existing_Functionality_is_covered</TargetRequirement><TargetRequirement>RF-45-Notification_is_generated_if_at_least_one_Provider_changes_his_feed_data</TargetRequirement><TargetRequirement>RF-46-User_can_restrict_data_by_multiple_Filters</TargetRequirement>
</TargetRequirements><TargetUseCases>
<TargetUseCase>UC-01-Import_Data_Feed</TargetUseCase><TargetUseCase>UC-01-01-Propagate_new_Feed_Event</TargetUseCase>
</TargetUseCases><TargetObjects>
<TargetObject>BaseEvent</TargetObject><TargetObject>FeedEvent</TargetObject><TargetObject>Notification</TargetObject><TargetObject>Rule</TargetObject><TargetObject>TeamLeague</TargetObject>
</TargetObjects><TestResults>
<TestResult>Feed Files have been parsed and separated</TestResult><TestResult>Feed Events have been added</TestResult><TestResult>Feed Events have been updated</TestResult><TestResult>Event Notifications have been stored</TestResult><TestResult>Update Notifications have been stored</TestResult><TestResult>UpdateEvent Notification has been sent</TestResult>
</TestResults>
SWISS-19Assignment of Use Cases to a Test Process
<ProcessSteps> <sequence> <ProcessStep name = "Generate_XML_Feed_File"> <selection>
<ProcessStepAction>Generate an XML feed file</ProcessStepAction> </selection> </ProcessStep> <ProcessStep name = "Load_XML_feed_File"> <selection>
<ProcessStepAction>Load the XML feed file</ProcessStepAction> </selection> </ProcessStep>
<ProcessStep name = "System_parses_XML_feed_File"><selection>
<ProcessStepAction>System parses the XML Feed file</ProcessStepAction></selection>
</ProcessStep> <ProcessStep name = "System_stores_Parse_Results"> <selection>
<ProcessStepAction cond = "if no parsing errors">System stores good Parse results for processing</ProcessStepAction><ProcessStepAction cond = "if parsing errors">System stores bad Parse results as error file</ProcessStepAction>
</selection> </ProcessStep>
<ProcessStep name = "Process_Events" cond = "if no parsing errors"> <selection>
<ProcessStepAction cond = "if new event and no other event exists">System creates new event</ProcessStepAction><ProcessStepAction cond = "if new event and no other event exists">System adds Feed Event
reference</ProcessStepAction><ProcessStepAction cond = "if new event and no other event exists">System sets Base Event properties</ProcessStepAction><ProcessStepAction cond = "if new event and no other event exists">System checks active Rule</ProcessStepAction><ProcessStepAction cond = "if new event and no other event exists">System checks if active Rule is
fulfilled</ProcessStepAction><ProcessStepAction cond = "if new event and no other event exists and rule not fulfilled ">System sends Create Notification to Bookie</ProcessStepAction><ProcessStepAction cond = "if new event and no other event exists and rule fulfilled">System updates base event attributes</ProcessStepAction><ProcessStepAction cond = "if new event and no other event exists and rule fulfilled">System sends non-action Notification to Bookie</ProcessStepAction>
</selection> </ProcessStep> </sequence></ProcessSteps>
SWISS-20Assignment of Use Case Steps to Process Steps
<TestCases><TestCase>
<TestCaseId>DFI0166</TestCaseId><TestCaseDescription>If at least one provider changed his feed data e.g. changing starting time,
a notification is added to a notification interface </TestCaseDescription><TestRequirement>RF-45</TestRequirement><TestUseCase>UC-01-01</TestUseCase>
</TestCase><TestCase>
<TestCaseId>DFI0167</TestCaseId><TestCaseDescription>The notification is only added to the notification table if the data is unequal to the current event data </TestCaseDescription><TestRequirement>RF-45</TestRequirement><TestUseCase>UC-01-01</TestUseCase>
</TestCase><TestCase>
<TestCaseId>DFI0168</TestCaseId><TestCaseDescription> The user choose Provider A as primary source,
so the event time is the time of Provider A </TestCaseDescription><TestRequirement>RF-45</TestRequirement><TestUseCase>UC-01-02</TestUseCase>
</TestCase></TestCases><TestEndCriteria>
<TestEndCriterium>All Feed File variations have been processed</TestEndCriterium><TestEndCriterium>All invalid Feed data has been recognized</TestEndCriterium><TestEndCriterium>All target database tables have been updated</TestEndCriterium> <TestEndCriterium>All database attribute values correspond to expected values</TestEndCriterium><TestEndCriterium>All feed data has been matched</TestEndCriterium><TestEndCriterium>All unmatched data has lead to a notification</TestEndCriterium><TestEndCriterium>All update events have been sent to V5</TestEndCriterium><TestEndCriterium>All notification data values correspond to expected values</TestEndCriterium><TestEndCriterium>All notifications have been acknowledged by the bookie</TestEndCriterium>
</TestEndCriteria>
SWISS-21
Assignment of Logical Test Cases to a Test Process
Test Process Data Flow
Data FeedUC-01UC-16UC-17
UpdateEvent
Validieremit
WSDLValXMLInterface
WSDLInterface
FeedFile
Generieremit XMLGen
XML
Rule
BaseEvent Delivered NotificationTeam
League
FeedEventDelivered
FeedEventNotification
Type
Validiere mitCSVVal
Generiere mitCSVGen
SWISS-22
+----------------------------------------------------------------------------------------+| WSDL Response Validation Report || || File: DFI-XML Params: Y Y Y Y || Object: Notification Date: 26.02.06 || Type : WSDL System: WebService || || Key Fields of Response(new,old) |+----------------------------------------------------------------------------------------+| New:Notification || Old:Notification |+----------------------------------------------+-----------------------------------------+| Non-Matching Fields | Non-Matching Values |+----------------------------------------------+-----------------------------------------+| RecKey:4711 | || New: EventName | Manchester/Arsenal || Old: EventName | Manchester/Liverpool |+----------------------------------------------+-----------------------------------------+| RecKey:4713 | || New: Eventdate | 2006-10-11 || Old: Version | 2006-10-12 |+----------------------------------------------+-----------------------------------------++----------------------------------------------+-----------------------------------------+| Total Number of old Responses checked: 10 || Number of old Responses found in new File: 10 || Number of old Responses not in new File: 00 || Number of new Responses found in old File: 10 || Number of new Responses not in old File: 00 || Total Number of Attributes checked: 70 || Total Number of non-Matching Attributes: 07 || Percentage of matching Attributes: 90 % || Percentage of matching Responses: 100 % |+----------------------------------------------------------------------------------------+
Test Result Validation against the Post Conditions SWISS-23
The Solution to System Testing
• The solution to the system testing problem lies in the association of objects and events at the logical level to objects and events at the physical level.
• The logical objects referred to in the system requirements must be linked to the physical objects of the system architecture – the GUIs, interfaces and database tables.
• The logical events referred to in the system requirements, e.g. use cases, must be linked to the physical events of the system – the online transactions and batch processes.
SWISS-24