with inferred macro events effective generation of gui tests monkey see, monkey...

1

Monkey See, Monkey Do:Effective Generation of GUI Testswith Inferred Macro Events

Markus ErmuthMichael Pradel

TU Darmstadt

2

Motivation

How to test complex GUIs?

Manual Analysis-based Random

2

Motivation


� Realistic event sequences� Huge effort


2

Motivation


� Automatic� Scalability issues

Analysis-basedManual Random

2

Motivation


� Automatic and scalable� Used in practice


3

Problem

Header:� Mouseover

Items:� Mouseover� Mouseout� Click

3

Problem

� Effective testing requirescomplex, realisticsequences of events

� Probability to hit them bychance: Extremely small

3

Problem

� Effective testing requirescomplex, realisticsequences of events

� Probability to hit them bychance: Extremely small

Observation:UI-level events 6= Logical events

4

This Talk

Monkey see, monkey do

� Learn usage patterns from users

� Imitate them during test generation

5

Events vs. Macro Events

Event (implementation level)� Type, target

Macro event (logical)� Finite state machine� Transitions = abstracted events

over,header

over,item

over,item

out,item

click,item

out,item

click,item3

6

Overview

GUI application

Traces

Macros

Tests

Record user actions

Infer macro events

Test generation

6

Overview

GUI application

Traces

Macros

Tests

Record user actions

Infer macro events

Test generation

Fully automatic

7

Recording User Actions

Trace: Sequence of events

...over, headerover, item1click, item1out, item1...

...over, headerover, item1out, item1over, item2click, item2out, item2...

etc.

8

Inference of Macro Events

TraceMacroevents

Frequentsubsequ.mining

ClusteringFiniteautomatainference

8


TraceMacroevents

Goal: Identify recurring patterns andremove noise

Adapted CloSpan algorithm [Yan et al., 2003]

� Bounded length of subsequences� Structural relations between events



8


TraceMacroevents

...over, headerover, item1click, item1out, item1...

...over, headerover, item1out, item1over, item2click, item2out, item2...



8


TraceMacroevents

...over, headerover, itemclick, itemout, item...

...over, headerover, itemout, itemover, itemclick, itemout, item...



8


TraceMacroevents

Goal: Group related subsequences

Prefix clustering:� Same initial event ; same cluster



8


TraceMacroevents

over, headerover, itemclick, itemout, item

over, headerover, itemout, itemover, itemclick, itemout, item



8


TraceMacroevents

Goal: Summarize sequences into macros

Adapted k-tails algorithm [Biermann, Feldman, 1972]

� Optimized state merging� Structural relations between events



8


TraceMacroevents

over,header

over,item

over,item

out,item

click,item

out,item



9

Test Generation

� Interleave random testing withmacro replay

� Pick and replay macros based onavailable events

� Replay active macro until reaching afinal state

10

More Details in the Paper

11

Implementation

Client-side web applications

Builds on WebAppWalker� Framework for UI-level testing� Firefox add-on� Strategies for selecting events

https://github.com/michaelpradel/WebAppWalker/

12

Evaluation

Effectiveness and efficiency?

Setup:� 4 real-world applications� 16 usage traces� Comparison with random

testing

13

Visited Pages

How many pages do thegenerated tests reach?

Significantimprovementsfor 3/4applications

0

5

10

15

20

25

0 100 200 300 400 500

#vi

site

dpa

ges

# triggered events (Drupal)

macrorandom

14

Branch Coverage

How many branches do thegenerated tests cover?

Significantimprovementsfor 3/4applications

3000

3500

4000

4500

5000

5500

6000

0 100 200 300 400 500

#co

vere

dbr

anch

es

# triggered events (Zurmo)

macrorandom

15

Covered Usage Scenarios

How many usage scenarios do thegenerated tests cover?

0

2

4

6

8

10

add content

previewcontent

comment article

edit content

delete content

simple search

advanced search

user search#te

sts

cove

ring

usag

esc

enar

io

macrorandom

16

Performance

� Inferring macro events

� 13 seconds – 85 minutes

� One-time effort

� Test generation

� 0.7 – 1.3 seconds per event

� Only 8% slower than random testing

17

Future Work

� Cross-application macro learning

� Lightweight, in-production gatheringof traces

� Scalability of inference algorithms

18

Conclusion

� Macro events:Abstract UI events into logical events

� Infer and apply macros:More effective GUI testing

� Human knowledge improvesautomated testing

18

Conclusion

� Macro events:Abstract UI events into logical events

� Infer and apply macros:More effective GUI testing

� Human knowledge improvesautomated testing

Thanks!

with inferred macro events effective generation of gui tests monkey see, monkey...

Documents