testware hierarchy for test automation
DESCRIPTION
Software Test Professionals Conference & Expo 2010.TRANSCRIPT
1
Testware Hierarchy for
Test Automation
Gregory Solovey
The proposed process allows to perform test automation in parallel with coding, and can be the base for continuous integration and agile testing
2
Developer : unit test
Team: feature test
Project: regression & sanity test
Product: system & load test
Technology: interoperability test
In the traditional approach automated tests are added during subsequent test process phases
In the proposed approach automated tests are reused while transitioning to the subsequent test process phases
Test Automation
CREATION
EVOLUTION
3
In the traditional approach test is sequential (formal relay)
In the proposed approach test is almost parallel(multi-threaded, with different priorities)
Agile Testing
Requirements Design Coding Unit test
Scenarios
Test plan Testing
Testware Testing
4
Processes
Provide decomposition/
abstraction
Design Tests
Build Harness
Automate Testing
There Is Nothing New Under the Sun
Divide
and
rule
It is
eas
ier t
o
perc
eive
erro
r
than
to fi
nd tr
uth
Giv
e m
e a
leve
r
and
a fu
lcru
m a
nd I
shal
l mov
e th
e
wor
ld
He who
lives
mus
t be
prep
ared
for
chan
ges
5
Processes
6
Processes
Provide decomposition/
abstraction
Define hierarchy
Present as a formal model
Design Test
Order error set
Cover all faults
Build Harness Provide
controllability/ observability
Automate Testing
Write scripts in test language
Separate business functions from
implementation
Architecture/ Requirements docs
Test docs
Design docs
Testware
Processes
7
Hierarchy
8
Processes
Provide decomposition/
abstraction
Define hierarchy
Present as a formal model
Design Test
Order error set
Cover all faults
Build Harness Provide
controllability/ observability
Automate Testing
Write scripts n test language
Separate business functions from
implementation
•Systems hierarchy
•Models hierarchy
•Errors hierarchy
•Tests hierarchy
•Harness elements hierarchy
•Report hierarchy
•Testware hierarchy
Order is Heaven's first law
9
04/13/2023 10
Process: Provide object decomposition/abstraction Sub-process: Define object hierarchyPurpose: Simplify the test design process
• As a rule of thumb, the size of a subsystem to be tested should be in the order of a hundred elements (objects, states, messages)
• Represent the system hierarchically, from a high level description to a less abstract one, until the desired level of detail is achieved
• Vertical decomposition is achieved by breaking down an element from black box to white box; from functional to structural model
• Horizontal decomposition is achieved by ensuring the independence of the model's elements.
Algorithms
Use cases
Sub-system
System
Object Test Harness Auto
04/13/2023 11
Process: Provide object decomposition/abstraction Sub-process: Define object hierarchyPurpose: Simplify the test design process
• Provides a natural way to divide the work• Achieves the fastest way to find most critical
errors by applying tests to the highest level of object abstraction first
• Defines the boundary between system, feature, unit test
Algorithms
Use cases
Sub-system
System
Object Test Harness Auto
04/13/2023 12
Process: Provide object decomposition/abstraction Sub-process: Define object hierarchyPurpose: Simplify the test design process
Algorithms
Use cases
Sub-system
System
Nowadays the systems’ architecture provides the right level of decomposition:
In the vertical view the direction is to separate the business and implementation details, allowing for independent multi-layer development, where the lower level layers are unaware of the higher layers
In the horizontal view the direction is towards object reuse (component based development), with the ability to assemble the objects in various contexts and access them via APIs
Hierarchy in SW architecture: (application (middleware (operating system(firmware (drivers))))Hierarchy in HW models: (processors (registers (flip-flops (gates)))Hierarchy in Protocol models: (SIP(TCP(IP (PPP(DS)))))
Object Test Harness Auto
04/13/2023 13
Process: Provide object decomposition/abstraction Sub-process: Define object hierarchyPurpose: Simplify the test design process
Algorithms
Use cases
Sub-system
System
For systems where this condition is not satisfied the tester has to provide the decomposition, document independent objects, review assumptions with systems engineers, estimate risk of such assumptions.
The management has to decide whether to accept the risk that test is incomplete or request architectural changes to satisfy the test requests.
Object Test Harness Auto
04/13/2023 14
Business function Model Test
Process: Provide object decomposition/abstraction Sub-process: Present as a formal modelPurpose: Formalize the test design process
• If the architecture/requirements are presented as a set of business functions, then only subject matter experts can define tests, based on their expertise. There are no formal rules on how to build tests from business-oriented requirements; therefore, it cannot be proved that the tests are complete and consistent.
• If the architecture/requirements are presented as a set of formal software engineering models, then testers can use well-known test design approaches to cover all model errors.
Object Test Harness Auto
04/13/2023 15
Business function Model Test
Process: Provide object decomposition/abstraction Sub-process: Present as a formal modelPurpose: Formalize the test design process
Data structure
Condition; function
Structural: algorithm; ladder diagram
Functional: State machine; Syntax; Object-oriented
Compound: Instruction set
Object Test Harness Auto
For systems where requirements are described as business functions the tester has to transform the plain, narrative text into formal statements (models analogies to UML), and have the result reviewed for consistency
04/13/2023 16
Business Model Errors Test
• A test cannot prove the correctness of an object, but can identify its errors. A test is a set of object stimuli that forces the object to produce different responses in the presence or absence of errors.
• Errors are possible distortions of the object-to-test’s model: its elements, structure or behavior.
Process: Design testSub-process: Order the error setPurpose: Increase error resolution
Object Test Harness Auto
04/13/2023 17
Process: Design testSub-process: Order the error setPurpose: Increase error resolution
4
1
32
54
1
3
2
regression test
each test case attempts to cover as many errors as possible (to have fewest cases)
new feature test
each test case attempts to cover as few errors as possible (to reach highest error resolution)
Object Test Harness Auto
04/13/2023 18
Model Errors Method Test
Process: Design test Sub-process: Cover all faultsPurpose: Guarantee the coverage with a minimal test set
Model MethodElementary: Variable; Condition; Function
Boundary analysis; Graph coverage; Value changes
Structural: Data/control flow; Algorithm
Nodes/ Branches/ Path coverage
Functional: State machine; Syntax; Object-oriented
States and transition functions coverage; Rules coverage; Properties, methods coverage
Compound: Instruction set Sensitive Path
Object Test Harness Auto
04/13/2023 19
a = b or c;f = a & d;if (f) then {} else {}
D = {a1, a0, b1, b0, c1, c0 , d1, d0, f1, f0 }
b
cd
bcd ExpectedResult
Errors
101 0 a0 b0 d0 f0
011 0 c0
001 1 a1 b1 c1 f1
110 1 d1
bcd ExpectedResult
errors
000 1 f0
111 0 a0 d0 f0
001 1 a1 b1 c1
110 1 d1
011 0 c0
101 0 b0
Regression test
Feature test
f1
a0 d0 f0
a1 b1 c1
c0 b0
d1
Graph model
Error set
Error tree
Program Object Test Harness Auto
19
04/13/2023 20
Process: Build the test harness Sub-process: Provide controllability/observabilityPurpose: Guarantee the test execution
execute
compare
control
observe
Object to test
A test harness is a set of “instruments” that provides the ability to execute test cases:
• CLI commands to access APIs or test functions• GUI hidden objects to access database or send transactions
Object Test Harness Auto
subsystem
21
Recommendation for CLI commands: • Provide routing to a particular CLI command, using UNIX-like syntax
(“cd”, “ls”, “help”, “show”)• Print the states and values of the object or group of objects in TLV
format • Access all object APIs: create, activate, discover, delete, send, etc• Report errors with the keyword “ERROR”• Redirect system outcoming messages to CLI interface • Do not perform any checks other than syntax at the CLI level• Return “OK/ ERROR” results from the command
Recommendation for GUI objects: • Use unique names for application objects • Use naming conventions• Use automation-friendly controls
Process: Build the test harness Sub-process: Apply standardsPurpose: Simplify automation
Object Test Harness Auto
Process: Automate testingSub-process: Separate business function from implementation detailsPurpose: Provide testware maintenance
Object Test Harness Auto
Developers and testers have different preferences for test tools: • Developers prefer to use open source tools (to be able to modify
them) and write test cases in a programming language (C, VB, C#, PERL, EXPECT)
• Testers feel comfortable using production tools (access to technical support) and write test cases in specialized test languages (test framework)
Programming language Test language
variables, arrays, hashes TS,UC,TC,TA w/ parameters
if, case compare
for, while, until loop for TS,UC,TC,TA
function, subroutine, procedure TC {setup, test, compare, reset}
algorithm unconditional sequence22
Process: Automate testingSub-process: Write script in a test language Purpose: Simplify debugging
Script: capture & playback
Test Set: subsystem level
Use Case: requirements level
Test Case: condition level
Test Action: transaction level
Object Test Harness Auto
Sample framework shows various test object levels: (TS (UC (TC (TA))))
23
Process: Automate testingSub-process: Write script in a test language Purpose: Guarantee reporting diversity
Object Test Harness Auto
level verdict report
TS subsystem metrics product management
UC feature full/partial/no functionality project management
TC test case pass/ fail developer/tester
TA transaction ok/ error/ timeout developer/tester
24
25000 20 5000 200 250 250 50 50SUM UNIQUE Test
TS 2
UC1
TC 1
TA 1 .TA 2
… TC q
TA 1 TA 2
… UC n
TC 1
TA 1 TA 2
… TC w
TA 1 TA 2
… TS n
UC 1
TC 1
TA 1 TA 2
… TC r
TA 1 …. TA 2
… UC m
Process: Automate testingSub-process: Separate business function from implementation detailsPurpose: Provide testware maintenance
TS
•XML config file
UC
•Test management system
TC
•Libraries/ Scripts
TA
•Test Language
Object Test Harness Auto
Sample testware hierarchy for an LTE embedded middleware system:• XML configuration file contains all test sets , which are invoked
based on changes to subsystems detected during the build process (for continuous integration)
• The test management system contains the use cases• The libraries contain the test cases (set-execute-compare-reset
phases)• The test language describes possible interactions with the object-
to-test
25
Process: Automate testingSub-process: Separate business function from implementation detailsPurpose: Provide testware maintenance
Test file call Test file call
Test file
Test case
Compare
Action
Action
Action
ActionAction
…
…
Test phase
Reset phase
Test caseTest case
Test case== definition part ========================
Use case
Test case callTest case callTest case call
Use case
Test case callTest case callTest case call
Use case
Test case callTest case callTest case call
== control part ==============================
TEST LIBRARIES
TEST CONTAINERS
TEST SCRIPTS
Object Test Harness Auto
26