mts: controllable test objects

MTS: Controllable Test Objects

Robert V. Binder 2nd International Workshop on End-to-end Test Script Engineering

July 16, 2012 Minneapolis

Copyright © 2012, Robert V. Binder

Overview

• Testing in Flatland

• Overview of MTS

• Lessons Learned

• Q&A

July 16, 2012 MTS - Controllable Test Objects 2

The xUnit Phenomena

jUnit

cppUnit


Incremental Testing Framework

TestObject

TestCaseTestSuite -Contains

0..1 0..*

0..1

-Contains

0..*

TestRunner TestLog+Contains

1 1

MyTestSuite

-Uses

1 1

MyTestObject

«uses»


Testing in Flatland

• Run A

p q r s t v

• Run B

r s t u

• Run C

t u

A p q B C p

q

r

s

t

u

B r s C

C t u

TestCases TestSuites


Straight Track


xUnit’s One Track Strategy

• Define and nest groups

• Run any sequence of groups

• No support for group or member

– Repeat

– Select

• Obstacle to any rich strategy

– Select, repeat

– Skip, Halt

– Time

– Broadcast

– Synchronize


Composition Isn’t Enough

• atom == test case

• ( ) group

• * quantification

• | select (or)

• How to do ( p | q )

• How to do (t ( p | q ))*

xUnit is like reg exp without * ? + |

A formal language without alternation and Kleene star


Composition Isn’t Enough Why Repeat?

• Iterate over collection

• Iterate over generation strategy

• Performance test over interval

• Capacity test to limits

Why Select?

• Respond to environment configuration/conditions

• Skip – Depends on earlier

– Don’t halt long complex test run

– Count failures for reliability estimate


Straight Track


Switching Tracks


MTS ARCHITECTURE AND FEATURES


MTS: Any App, Any Platform

Agent Host

Client Host Under Test

Agent HostConsole Host

MTS Console

MTS Test

Agent

MTS Test

Agent

Host Under Test may be

TEST RUN

REPORTS

MTS Remote

Agent

Client

Under Test

Server Host Under Test

MTS Remote

Agent

Server

Under Test

Cell Phone

PDA

Desktop

Server

Embedded Processor

Network Equipment

Access Point

Base Station

Client Host Under Test

MTS Remote

Agent

Client

Under Test

Server Host Under Test

MTS Remote

Agent

Server

Under Test


MTS Approach

• Test harness for distributed, end to end testing

– xUnit: composable test objects

– TTCN: adapters, concurrency, …

– Tcl: defacto standard for test script automation


About TTCN-3

• Procedural DSL similar to CORBA IDL

• Point of Observation and Control (adapters)

• Modular, but not necessarily composable

• Well-suited to model-based testing – Generate TTCN from MSCs,

FSMs, etc.

– Generate impl source from TTCN

– Compile & run impl

• Same test suite can be used on many implementations


Tcl

• Open source interpreter

• LISP-ish evaluation

iut.set {1} iut.set {$x} iut.set {[genX{$y}]} iut.set {[eval $codeForX]}


Tcl • Open source

interpreter

• LISP-ish evaluation

• Object-oriented

[incr Tcl]


MTS::TestObject

PPC 2002

YourTestObject

+getTestObjectClassName()

+getTestObjectHeadline()

+getTestObjectFileName()

+getTestObjectDir()

+getRemoteHostStatus()

+getTestRunRepeatCount()

+getTestRunIteration()

#require()

#setup()

#test()

#cleanup()

#ensure()

#check()

#run()

#submit()

#userLogEntry()

#userLogImage()

#evaluateCompareFrame()

#replayTestActionSequence()

+testObjectHeadline

MTS::TestObject

+require()

+setup()

+test()

+cleanup()

+ensure()

MyTestObject

MTS Framework

+connect()

+open()

+close()

+test()

+record()

Adapter

+getImage()

WM5

+getServiceWSDL()

+parseWSDL()

+queryForDictionary()

+queryForXML()

WebService


TestObject methods Skip rest of test object when its preconditions aren't met. Automatically called when a TestObject is run.

require

Create starting conditions for a test; allocate necessary resources. setup

Run tests. The user's implementation is expected to execute test actions leading to one or more test verdicts.

test

Restore pretest conditions; release resources the user test object has claimed.

cleanup

Skip or halt rest of a test run when postconditions aren't met. Automatically called after cleanup completes.

ensure


Using require ::itcl::class aTestCase{ inherit TestObject ::itcl::body aTestCase::require {} { if {[getRemoteHostStatus] eq "Connected"} { set remoteHostconnection true } else { userLogEntry "Remote Host not ready, skipping" set remoteHostconnection false } return $remoteHostconnection } ::itcl::body aTestCase::test {} { … }


Completion Actions return keyword

Current Test object Caller Test object Test objects run stack Counters

OK No effect, Destruct (ensure)

No effect. No effect. No change.

SKIP Destruct Continue with the next test object.

No effect. No change.

HALT Destruct Immediately exit the method that called this test object.

Terminate the test run. Percolate the HALT to the run stack.

No change.

EXCP Capture current Tcl errorInfo. Destruct the current test object.

Immediately exit the method that called this test object.

Terminate the test run. Percolate the EXCP to all test objects in the run stack.

Increment error counter.


Using ensure and Completion Actions ::itcl::class checkDir {inherit TestObject ::itcl::body checkDir::require {} { … } ::itcl::body checkDir::setup {} { … } ::itcl::body checkDir::test {} { … } ::itcl::body checkDir::cleanup {} { … } ::itcl::body checkDir::ensure { set currentDir [pwd] if { ! [regexp {mydir} $currentDir]} { return SKIP } else { return OK } }


MTS run command

run <target> repeat <times> <arg list>

• “Sources in” target file TestObject subclass

• Runs this TestObject

• Returns control to the caller

• Hardcoded target run myTestObject

• Target name in a variable run $anyTestObject

• Function computes target run [eval whichTestObject]


Selection and Repetition in MTS How to do ( p | q ) How to do (t ( p | q ))*

::itcl::body pq::test{} { if {[rhStatus]} { run remoteTest } else { run localTest } }

::itcl::body mySuite::test{reps} { run strategy1 -repeat $reps } ::itcl::body strategy1::test{} { run t run pq }


TestObject helpers

Start other test objects or create a test suite run <target>

Evaluate and log pass/fail conditions check <expression>

Time any action during a test run Timer <method>

Object interface for test run as a whole testRun <method>

Monitor remote host resource utilization rpm <method>

Check the remote host getRHostStatus myHost

Perform data-driven tests submit <args>

Annotate test run reports userLogEntry <args>


IMPLEMENTING MODAL CLASS TEST PATTERN


Modal Class Test Pattern α

ω

Two PlayerGam e ( )

~( ) ~ ( )

Gam e S tarte d

P la ye r 1

Served

P la ye r 2

Served

p2 _S tart ( ) /

s im u lateVolle y( )

p1 _W insVolle y( ) /


p1 _W insVolle y( )

[th is.p 1_ Score ( ) < 20 ] /

th is.p 1AddPoin t( )


p1 _S tart ( ) /


p2 _W insVolle y( )

[th is.p 2_ Score ( ) < 20 ] /



p2 _W insVolle y( ) /

s im u lateVolle y( )p1 _W insVolle y( )

[th is.p 1_ Score ( ) == 20] /


p2 _W insVolle y( )

[th is.p 2_ Score ( ) == 20] /


p1 _IsW in ner( ) /

retu rn TR UE;

P la ye r 1

W o n

P la ye r 2

W o n p2 _IsW in ner( ) /

retu rn TR UE;

Ga m e S ta rted

P la yer 3

Serve d

p 3_ S tart ( ) /

s im ulateVo lley( )

p 1_ W insVo lle y( ) /


p 3_ W insVo lle y( )

[this.p 3_ Sco re ( ) < 2 0] /

th is.p3 Add Point ( )




p 3_ W insVo lle y( )

[this.p 3_ Sco re ( ) == 2 0] /

th is.p3 Add Point ( )

P la yer 3

W on p 3_ IsW in ne r( ) /

return TR UE;

αTh ree P la yerGa m e ( ) / Two P la yerG am e ( )



Tw oP layerG am e ( )

ω~( )

TwoPlayerGame

ThreePlayerGame

+TwoPlayerGame()+p1_Start( )+p1_WinsVolley( )-p1_AddPoint( )+p1_IsWinner( )+p1_IsServer( )+p1_Points( )+p2_Start( )+p2_WinsVolley( )-p2_AddPoint( )+p2_IsWinner( )+p2_IsServer( )+p2_Points( )+~( )

+ThreePlayerGame()+p3_Start( )+p3_WinsVolley( )-p3_AddPoint( )+p3_IsWinner( )+p3_IsServer( )+p3_Points( )+~( )

TwoPlayerGame

ThreePlayerGame

Game Started

Player 1Served

Player 2Served

p2_Start( ) /simulateVolley( )

p1_WinsVolley( ) /simulateVolley( )

p1_WinsVolley( )[this.p1_Score( ) < 20] /this.p1AddPoint( )simulateVolley( )

p1_Start( ) /simulateVolley( )

p2_WinsVolley( ) [this.p2_Score( ) < 20] /this.p2AddPoint( )simulateVolley( )


p1_WinsVolley( )[this.p1_Score( ) == 20] /this.p1AddPoint( )


p1_IsWinner( ) /return TRUE; Player 1

WonPlayer 2Won

p2_IsWinner( ) /return TRUE;

α

ω

ThreePlayerGame( ) /TwoPlayerGame( )

~( )

Player 3Served

p3_WinsVolley( )[this.p3_Score( ) < 20] /this.p3AddPoint( )simulateVolley( )


Player 3Won

p3_IsWinner( ) /return TRUE;

~( )

p3_Start( )/simulateVolley( )





~( )


Modal Class Test Design

Player 1 Served

Player 2 Served

Player 3 Won

omega

Player 3 Won

Player 3 Served

Player 3 Served

Player 1 Served

Player 2 Won

omega

Player 2 Won

Player 3 Served

Player 2 Served

Player 2 Served

Player 1 Served

Player 1 Won

omega

Player 1 Won

Player 3 Served

Player 2 Served

Player 1 Served

Gam eStartedalpha

3 p2_Start( )

6 p1_WinsVolley( )[this.p1_Score( ) < 20]

2 p1_Start( )

9 p2_WinsVolley( ) [this.p2_Score( ) < 20]

8 p2_WinsVolley( )

7 p1_WinsVolley( ) [this.p1_Score( ) == 20]


1 ThreePlayerGame( )

4 p3_Start( )

5 p1_WinsVolley( )

14 p1_IsWinner( )

15 p2_IsWinner( )

17 ~( )



16 p3_IsWinner( )

11 p3_WinsVolley( )

1

2

3

4

8

11

7

6

9

11

10

5

12

13

8

5

17

14

17

15

17

16

*

*

*

*

*

*

Sneak path tests not shown Note iterations


Implementing Modal Class Design xUnit

• TestCase for each transition (20) • Hand hack TestSuite for each

root to leaf path (17) and one to run them all – addTest – addTestSuite

• How to iterate transitions? • Embed strategy in single large test

object? – Very smelly

• Develop code generator that writes test suites?

MTS

• TestObject for each transition (20)

• Top level TestObject uses run to call transition test objects – Iteration trival

• Parameterized strategies

– N+

– Markov

– Etc.

• Parameterized state spec


Dude, where’s my object?

1 1

TestCase objects TestSuite objects

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

16

xUnit MTS

TestObject objects

Player 1 Served

Player 2 Served

Player 3 Won

omega

Player 3 Won

Player 3 Served

Player 3 Served

Player 1 Served

Player 2 Won

omega

Player 2 Won

Player 3 Served

Player 2 Served

Player 2 Served

Player 1 Served

Player 1 Won

omega

Player 1 Won

Player 3 Served

Player 2 Served

Player 1 Served

Gam eStartedalpha

3 p2_Start( )

6 p1_WinsVolley( )[this.p1_Score( ) < 20]

2 p1_Start( )


8 p2_WinsVolley( )



1 ThreePlayerGame( )

4 p3_Start( )

5 p1_WinsVolley( )

14 p1_IsWinner( )

15 p2_IsWinner( )

17 ~( )



16 p3_IsWinner( )

11 p3_WinsVolley( )

1

2

3

4

8

11

7

6

9

11

10

5

12

13

8

5

17

14

17

15

17

16

*

*

*

*

*

*


Controllable Test Objects


Design Goal TTCN xUnit MTS

Select, Iterate, any level Yes No Yes

Generate with MBT strategies Yes Yes Yes

Logical/Physical separation Yes No Yes

Platform agnostic/robust Yes ? Yes

Channel agnostic/robust Yes ? Yes

Distributed control Yes No Yes

Minimize Test Agent resources NA No Yes

Minimize brittleness Yes No Yes

Intuitive interaction No Yes Yes

Composable No Yes Yes

MTS hard problem queue • Protocol to drive remote target host IUT from test host

– Bi-directional, lightweight (BEEP ?) – Platform and stack agnostic – Target environment control (remote command line) – Distributed adapter RPC

• Forced reflection to generate adapters • Logical/physical mapping in generative models • Scalable asynchronous remote agents

– Control strategy – Provisioning – Protocol


Question Time


Bob Binder [email protected] www.robertvbinder.com

mailto:[email protected]

mailto:[email protected]

http://www.robertvbinder.com/

http://www.robertvbinder.com/

mts: controllable test objects

Technology

complex test runcapacity

test case

test suite

test verdicts

test testactions

mts approach test harness

user test object

test script automationjuly