<Insert Picture Here>

Automating JFC UI application testing with Jemmy.

Alexandre (Shura) IlineJava SE and JavaFX Quality architect.

The following is intended to outline our general product direction. It is intended for information purposes, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.

This presentation shares experience gotten from testing such products as

JRockit JavaFX AT

What's in scope

• All sorts of definitions “Testing”, “UI testing”, “Test automation”• Jemmy library• Automation approaches effectiveness and cost.• Bunch of source code

• Basic Jemmy operations• Less basic operations• Switches and settings

• Tricks and tips• Misconceptions

What's not in scope

• The economical effects• A.k.a. what's “quality”• Why to automate

• Automation cost vs. manual testing cost• When to automate• What to automate

• Non-economical benefits of automation

A.k.a. let's leave it for later. :)

UI testing … by Wikipedia

«GUI software testing is the process of testing a product that uses a graphical user interface, to ensure it meets its written specifications.»

UI testing … most often ...

«Checking whether usage of a product UI leads to results expected by the the person who performs testing»

●Start text editor●Push «File/Open»●Verify file chooser directory●Select some file●Verify editor area content●Verify application title●Verify buttons availabilities●....

Sample test scenario

Find Do VerifyPass Pass

Pass

Fail Fail Fail

Failure analysis

Find next controlTo perform operation

Verify that expectedState reached

Perform necessary actions

UI test workflow

Test automation ... for me

«Building a process which exercises and reports certain product characteristics while run unattended.»

Test automation is like development

• Putting logic in a code• Same lifecycle:

• Requirements• Design• Implementation

• Same set of problems• Bugs • Instabilities

• Scope • defined through test specification/plan

Test automation is not like development

• Big fat dependency – the tested product• vs many libraries and platform

• Many small programs• vs one big program

• Does one thing good – reports status• vs does many thing ... good

• Perfectness is not the goal• other than maintenance cost, ease of use

Jemmy

Jemmy history• Started as a tool to tests TeamWare UI (1999)• Used for NetBeans extensions (2000)• Official test tool for NetBeans (2001)• Open-source (2001)• Is used for NetBeans and extensions since then• Adopted as a test tool for Swing (~2003)• Used outside SUN Microsystems (next slide)• Jemmy v3 created (2008)• The test tool for JavaFX SDK (2008 – now)• Extended to support SWT• Test tool for JRMC

JemmyCore

JemmySGJemmy v2

AWT & Swing

JemmyRemote

JemmyFXRemoteJemmyFXJemmySWT

JRMCJavaFX Authoring Tool

JavaFX

LCDUINetBeans

JDK

Jemmy and extensions

Jemmy v3

JemmyAWT

Jemmy v2 JFC

Same VM.

Test code runs in the same VM as the application code• Benefits

• Full access to the application UI objects• As well as the application domain object (sometimes).• Control over event queue

• Drawbacks• Impacts the tested UI

• Options• Run application from test• Run test from application• Use accessibility hook

Using Jemmy

Demo

Operators

Test codeTest code

JFrameOperatorgetTitle(),

...

JFrameOperatorgetTitle(),

...

JComboBoxOperatorgetSelectedItem(),

selectItem(...)

JComboBoxOperatorgetSelectedItem(),

selectItem(...)

JTextFieldOperatorgetText(),

typeText(...), ...

JTextFieldOperatorgetText(),

typeText(...), ...

JButtonOperatorgetText(),

push()

JButtonOperatorgetText(),

push()

Operators

Demo

Threading

“noblocking”operation

periodicalchecks

waiting waitingaction action action

action

WaitingTimeouts

Demo

Event queue

User

UIOtherthreads

UIOtherthreads

Test

Actions

Verifications

Lookup

Event queuewith test

Event queue tools

Demo

Drivers

Test code

ButtonDriverpush(...)

...

TextDrivertypeText()

...WindowDriver

ListDriverslectItem(...)

...

JButtonOperator

JTextFieldOperator

JComboBoxOperator

JFrameOperator

Drivers

Demo

Dispatching modes

• Events• Proper events in a proper order are dispatched to a

component making it thinking some user actions are performed.

• Shortcut• Same as events except the events are posted into the event

queue at once. • Robot

• java.awt.Robot is used

Implemented with driver sets

Dispatching models

Demo

Image

• Fragile• Needed for custom controls

• Comparison• Pixel-to-pixel• %s of pixels• Color distance

• Average• Maximum

Verifying images

Demo

Automationapproaches

33

33

Application UI

Product UIProduct UI

34

34

Coordinates

• click(134,32) //selects some record• click(215,122) //hits “Properties”• sleep(5) //sleeps to let dialog be painted• click(64,182) //expands color combo• click(235,182) //selects Gray• click(235,212) //hit OK

35

35

Widgets

• Find “Car records” frame• Find table• Select “1abc234” cell• Push “Properties” button• Wait for “1abc234” dialog• Select “Gray” color in combo box• Push “OK”

36

36

Prod

uct U

IPr

oduc

t UI

Dom

ain

mod

el

Dom

ain

mod

el

Car record

ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN

Test

Widgets or coordinates

Cars testWidgets

Demo

38

38

UI Primitives

• Find car list frameCarListFrame list = new CarListFrame()

• Open properties dialog for car “1abc234”CarDialog propDialog = list.carProperties(“1abc234”);

• Set color to graypropDialog.setColor(Color.GRAY);

• Apply changespropDialog.ok();

39

39

LibraryPr

oduc

t UI

Prod

uct U

ID

omai

nm

odel

Dom

ain

mod

el

Car record

Test libraryTest library

ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN

CarListFrame CarDialog

Test

Cars testUI Library

Demo

41

41

Domain model

• Set color to gray for a car “1abc234”

new CarRecord(“1abc234”). setColor(Color.GRAY);

Underneath the cover, CarRecord class does all described earlier

42

42

Domain libraryPr

oduc

tU

IProd

uct

UI

Dom

ain

mod

el

Dom

ain

mod

el Car record

Domain test libraryDomain test library

ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN

UI test libraryUI test libraryCarList CarDialog

CarRecord

Test

Cars testDomain Library

Demo

Automation effectiveness

The formula?

TD + *T

S NR

TM * N

RN

C*E

A =N

C*

EA – automation effectiveness

To be used for every particular product.

NR and N

C are unique for a product.

TM is a characteristic of a test suite.

Smaller TD and T

S - higher the E

A.

Coefficient depend on the way you write your tests

Coordinates Widgets UI Library Domain library0

1

2

3

4

5

6

7

8

1.1

3

5

7.5

10.5

0.1 0.05

Td/TmTs/Tm

Td and T

s together

Coordinates Widgets UI Library Domain library0

5

10

15

20

25

30

1.13

5

7.5

24

12

2.41.2

25.1

15

7.48.7

TdTsTd+(Ts*Nc*Nr)

TD and T

S for N

C=3, N

R=8, T

M=1

Coordinates Widgets UI Library Domain library0

0.5

1

1.5

2

2.5

3

3.5

0.96

1.6

3.24

2.76

Ea

EA for N

C=3, N

R=8

Misconceptions

• Automated testing replaces manual testing• it will find all the bugs• it will find more bugs• it does the same amount of work

• Create once use forever• This is easy• This is too hard• No need to create test plan• “Will just buy the XXX tool and it will solve all out

problems”

Resources

jemmy.dev.java.net

<Insert Picture Here>

Automating JFC UI application testing.With Jemmy.The technical aspects. Alexandre (Shura) IlineJava SE and JavaFX Quality architect.

Backup slides

Coverage

• Implementation coverage• Line, block, condition, sequence, ...

• Specification coverage• How well the tests covering the functional specification

• Public API coverage• Whether the public API is tested fully

• UI coverage• Whether the tests cover UI fully

• Combine with bug density• First cover the areas with more bugs

• Combine with code complexity• First cover the more complicated code.

Development

Continuous build

Commit

BuildExecuted

automaticallyafter commit

Success

Analysis.Rollback!!!

Code changes

NoYes.

PromoteCode is compilableTest further

Continuous build with testing

Commit

Build Success

TestingIs it working?

Passed

Analysis.

Rollback!!!

Code changes

Testchanges

No

No

Test furtherBuild is good

Code line is healthyGo on ...

Yes

Yes. = Compilation

successful

Pre-integration

Commit

Code changes

Testchanges

Testing Passed

No

Yes

How to benefit from UI automation?Use it!

• Sanity• Pre-integration• Attach to bug reports• Make is a quality criteria• Run it for every build• Show it to the boss :)• Don't forget to show it to the dev. team