a model driven tool for automated system level testing of middleware turker keskinpala abhishek...

A Model Driven Tool for Automated System Level

Testing of MiddlewareTurker KeskinpalaAbhishek DubeySteve Nordstrom

Ted BaptySandeep Neema

Institute for Software Integrated SystemsVanderbilt University

Nashville, TN USA

STV'06 WorkshopMarch 30, 2006

Outline

Motivation and Challenges Model Driven Approach Tool Structure Test Series Definition Modeling

Language Example Test Series Definition Conclusions and Ongoing Work


Motivation and Challenges Many middleware framework configurations are

generally represented by XML Configuring functional operation of components Component deployment in the cluster

Frameworks are complicated and need to be tested Application topologies Parameter variations

Need for many test configurations


Motivation and Challenges

Hand writing test configurations Cumbersome Error prone Inefficient

Big configuration space of control and deployment parameters of applications

Response to changes in hardware resources or other criteria Difficult to scale up manually written XML

configurations

Usual outcome: Under tested software


Goals

Testing large sets of system configurations Simple powerful specifications Tool to take specifications to instances

Configure series of tests from the specifications

Help finding “corner cases” Potential functional errors Potential performance problems Unexpected system interactions


Model Driven Approach

Domain Specific Modeling Use domain specific models to encapsulate relevant

parts of the system properties

Provide multiple view points Decouple types, resources, dataflow and

deployment aspects

Generative modeling Vary range of behavioral and structural system

parameters

System generation Automatically generate configurations from higher

level models


Tool Structure

Test Series Definition Models


Test Series Definition Define tests using a high level modeling

language called Test Series Definition Modeling Language (TSDML) Define application types Form a template test case from those types Define dataflow association between applications

by connection rules Define deployment associations for applications Define the variance on the system size and

structure Define variance on the application parameters Define available resources

Automatically create series of test configuration models by replicating and adapting the template test case


Tool Structure


Generator (1-*)

Test Configuration Models



Form the configuration layer Each test configuration model capture:

Specific dataflow scenario Specific deployment scenario Configuration Schema

Represents single system representation to be tested

Automatically generate single configuration file per configuration model

Test driver scripts can also be generated


Tool Structure


Generator (1-*)


XML Configs Test Scripts

Generator (1)

Application Runtime (System Under Test)

Test E

xecutio

n &

V

alida

tion

En

gin

eApplications


System Under Test

XDAQ Framework High performance distributed middleware

framework developed at CERN Will be used for Compact Muon Selenoid

(CMS) experiment Run-time support for clustered applications Predefined functions to configure and

control the applications and their parameters

XML is used for configuration of XDAQ and its applications


Test Series Definition Modeling Language (TSDML)

Parameterized model Varying system structure

(size) Iterators: e.g. i,j,k = {start,

step, stop} Replicators: e.g. r1=n*i,

r2=m*j Connectors: Boolean

function of iterator, and replicator

If src=(i,r1) and dst=(j,r2) C=f(src,dst) C=2*src==3*dst

i

r1

App1

j

App2

r2

Connector


Iterator-Replicator Example

Assume i = 1:1:2, j= 1:1:4, r1 = 2, r2 = 4

r2

i

r1

App1

j

App2

Test Case i j #App1 #App2

1 1 1 2 4

2 1 2 2 8

3 1 3 2 12

4 1 4 2 16

5 2 1 4 4

6 2 2 4 8

7 2 3 4 12

8 2 4 4 16


Connector Examplei = 1:1:2, j= 1:1:4,

r1 = 2, r2 = 3CR = src=floor((dst-1)/(j*r2/i*r1)) + 1

i

r1

App1

i

App2

r2

Test Case i j #App1 #App2

1 1 1 2 4

2 1 2 2 8

3 1 3 2 12

4 1 4 2 16

5 2 1 4 4

6 2 2 4 8

7 2 3 4 12

8 2 4 4 16Connector


Connector Example (ctd…)

App1

App2

App1

App2

App2

App2

(1)

(2)

(1)

d

s

t

s

r

c

Rule:

src=floor((dst-1)/(j*r2/i*r1)) + 1

(2)

(3)

(4)

1=floor(0/2) + 1 => TRUE1=floor(1/2) + 1 => TRUE1=floor(2/2) + 1 => FALSE1=floor(3/2) + 1 => FALSE


Test Series Definition Modeling Language (TSDML)

Varying application parameters with parameter generator Has a function attribute Function of iterator Function attribute may be:

Internal function e.g. result = src * 256 where src = i

Matlab script e.g. lut = [10 20 55 88 400 800

1025 4490 7342]; result = lut[src]; Internal lookup table


Example TSDML


Example TSDML

2*i

1:1:8 1:1:1

1:1:1

2*i16*

i

1*i


Conclusion

Domain specific modeling for capturing relevant system parts

Vary system structure and application parameters from high level model

Generate series of test configurations from a single high level model

Generate the configuration representation

Mitigate the complexity of creating many system test configurations


Ongoing Work

Add another layer on top of TSDML System model

Application behavior Target framework

Analyze the models Assess important parts of the system

to exercise and test


Thank you!

A Model Driven Tool for Automated System Level

Testing of MiddlewareTurker KeskinpalaAbhishek DubeySteve Nordstrom

Ted BaptySandeep Neema

Institute for Software Integrated SystemsVanderbilt University

Nashville, TN USA

a model driven tool for automated system level testing of middleware turker keskinpala abhishek...

Documents

stv06 workshop

test configurations

template test case slide

app2 r2 test caseij

r1 app1 j app2 test

false slide

tested software slide

tn usa slide