1

There is an A for everything

Theoretical side of things

Definitions Vocab ISTQB

Good practices

Things to remember

Lessons to learn

Practical (realistic ) side of things

Activities and their benefits

Examples

2

Test Automa

tion Framew

orksRoxana Cracana

December 4th, 2013

Summary :

• Define test automation framework

• Types of automation frameworks

• Create your own framework

• Focus Points

• Layer your framework

3

A set of assumptions, concepts and practices that constitute a work platform and support for automated testing.

What is a Test Automation Framework?

Is responsible for:• Creating a mechanism to hook into or drive

the application under test• Creating methods to automate test cases• Defining the format in which to express

input data and expected results• Executing the tests• Reporting results

Properties:• Correct and complete• Reliable• Reusable • Scalable• Easy to use• Easy to maintain• Fast

4

Types of automation testing frameworks

Linear

Modular

Data Driven

Keyword Driven

Hybrid

Test Driven Development

Behavior Driven Development

Agile Frameworks

Automation Testing Frameworks

5

Record and Playback is not an framework

What is…• Capture interaction with system and replay it• Popular approach among commercial automation

tools

Advantages• Very easy and fast to create initially• No programming skills needed• Easiest way to learn the features of the testing

tool

Disadvantages• Does not test anything unless checkpoints added• Very fragile – often single change in UI can break

all tests• No modularity or reuse• Hard to maintain

6

Linear testing framework

What is…• Non-structured scripts interact directly

with the SUT • Also produced by capture and replay tools

Advantages• Fast way to create script• Automation expertise not required, just

tool knowledge

Disadvantages• No modularity or reuse• Test data is hard coded into the script• Very fragile – one change in the system

may break all scripts• Plenty of separates scripts• Never a good basis for large scale

automation

SUT

Test script 1

Test script 2

7

Modular testing framework

SUT

Test script 1

Test script 2

Test library

Test method 1

Test method 2

Test method 3

What is…• Interaction with the SUT done by functions in

a test library

Advantages• Higher level of code reuse• Easier script maintenance

Disadvantages• Building test library requires initial effort:

takes time and programming skills• Test data embedded into scripts

8

Data-driven testing framework

SUT

Test Data 1

Test Date 2

Parser

Test script

1

Test script

2

Test library

Test method 1

Test method 2

Test method 3

What is…• Test data taken out of test scripts –

usually externalized in tabular format

Advantages• Changes to the test scripts do not affect

the test data• Test scripts can be executed with

multiple sets of data

Disadvantages• Creating scripts requires programming

skills and time• Initial effort creating data parsers can be

high• May be an overkill for simple needs

Test data for InvalidLoginUsername Password Error Message

The field E-mail is mandatory abc123* The field E-mail is mandatory

john.smith@gmail.com

The field Password is mandatory

john.smith abc123*The format of the e-mail address entered is not correct.

john.smith@gmail.com

abc1The password must have between 5 and 12 characters.

john.smith@gmail.com

a1b2c3d4e5f6gThe password must have between 5 and 12 characters.

john.smith@gmail.com

a1b2c3d

Your e-mail address or password is not correct or is not registered at Mango.com

9

Keyword-driven testing framework

SUT

Test script 1

Test script 2

Test library

Keyword 1

Keyword 2

Keyword 3

What is…• Utilizes data tables and self explanatory

keywords to indicate the actions performed on SUT

Advantages• Provides high code re-usability• Test tool independent • Tests can be designed with or without

SUT• Very good for large scale use

Disadvantages• Initial effort can be really high

Test : ValidLogin

Action Object Parameters

Navigate Browser http://shop.mango.com/RO/mango

Set Email john.smith@gmail.com

Set Password 123abc*

VerifyText Username John Smith

Close Browser

10

Hybrid testing framework

What is…• Is the combination of modular, data-

driven and keyword driven testing frameworks

Advantages• Integrates the advantages of all the other

frameworks.

Disadvantages• Very complex

11

Which type of framework to choose?

Test framework Advantages Disadvantages

Modular Reusable functionsHierarchical structure

Test data within the scripts limits reusabilityTest script is dependent on software

Data driven Improved maintainability Dependency on technical expertiseTest script is dependent on software

Keyword driven Ease of maintenanceScalabilityLess dependency of software

Dependency on technical expertiseRequires large effort

Hybrid Integrates the advantages of all the other frameworks

Increased complexity

Create your own framework!

12

Focus point

Good testing frameworks have layers

• Layer 1: Control level• Layer 2: Business level• Layer 3: Test level

Build layers over existing automation tools!

13

Focus point

Test error and failure handling

• Decide when to fail your test• Decide when to fail your test

suite• Think about tests setup and

tests recovery

Soft assertion vs Hard assertion

14

Focus point

The key artefact of a testing framework is

REPORTING • Summary report• Detailed report• Reports repository• Traceability• Metrics

Integrate early logging and reporting into the framework!

15

Focus point

Object Repository

• Also called Application map• Stores AUT objects• Multiple ways to represent it

Identify the best way to represent OR!

16

Focus point

Continuous automation

• Parallelize test execution • Distribute test execution• Schedule test execution• Automate test machine setup • Automate test precondition• Automate notifications

Integrate test execution into a continuous integration tool!

17

Layer your framework – Control Level(1)

Step 1: Control types• Identify all control types from AUT

• Standard controls e.g. : button, dropdown, edit, checkbox, radio button

• Composed controls e.g. : radio button group, checkbox group, menu, table, tab group

• Custom controls • Implement one class for each type of

control

Step 2 : Control methods• For each type of control identify user

interaction• Action e.g. : click button, select value in

dropdown, type value in edit• Verification e. g. : verify selected value in

drop down, verify all values from table• Implement one method for each user

action• Integrate in each method error

handling and reporting

18

Layer your framework – Control Level(2)

Dropdown actions• Select value• Select index• Verify selected value• Verify all values• Verify value exist• Verify value not exist• Verify number of values• Verify values are sorted• Verify is displayed• Verify is not displayed• Verify is enabled• Verify is disabled• Verify background colour• Verify label• Verify tooltip

Edit actions• Type value• Clear value• Append value• Enter value• Verify value• Verify is displayed• Verify is not displayed• Verify is enabled• Verify is disabled• Verify background colour• Verify label• Verify tooltip

19

Layer your framework – Control Level(3)

Dropdown actions

• Select value• Select index• Verify selected value• Verify all values• Verify value exist• Verify value not exist• Verify number of values• Verify values are sorted

Edit actions

• Type value• Clear value• Append value• Enter value• Verify value

General actions• Verify is displayed• Verify is not displayed• Verify is enabled• Verify is disabled• Verify background colour• Verify label• Verify tooltip

20

Layer your framework – Control Level(4)

Logging

Error handling :Hard assertion

Edit control – Verify Value method

21

Layer your framework – Business Level(1)

Step 1: Business Units• Identify business units for

AUT (business unit : page, section, widget)

• Create one class for each business unit

Step 2: Application Map• For each business unit,

identify all objects that have a business value

• For each object identified:• Associate an unique alias

map• Associate properties that

unique identify the object into the business unit

Step 3: Business Flows• For each business unit

identify scenarios or business logic flows that can repeat into test case

• Implement one method for each flow

22

Layer your framework – Business Level(2)

Business Unit:Header

Business Unit:Register

Business Unit:HomePage

Business Unit:Start Session

23

Layer your framework – Business Level(3)

Application MAP for Start Session Edit_Email

Edit_PasswordLink_ClickHere

Text_Errors

CheckBox_AutoLogin

Button_StartSession

Button_Register

24

Layer your framework – Business Level(4)

Start SessionBusiness Unit

25

Layer your framework – Test Level(1)

Test implementation :• Establish setup and recovery scenarios• Identify a test framework (e.g. NUnit,

JUnit, PyUnit) and the best way to create test suits and test cases

• Parameterize test cases if necessary• No waits or code into the tests, only calls

to control and business methods

Test execution :• Identify tools for test execution• Schedule execution• Automate test preparation

26

Layer your framework – Test Level(2)

TC01 Verify invalid email in Login.

Precondition

Email "roxana.cracana@gmail" is already registered with password ‘123abc*’

Navigate to url http://shop.mango.com/RO/mangoClick link “START SESSION”

Step No Actions Expected results

1Type "roxana.cracana" in field "E-mail"

"E-mail" has value "roxana.cracana".

2Type correct pasword "123abc*" in field "Password"

"Password" has value "*******".

3Click 'START SESSION' button

Error is displayed :”The format of the e-mail address entered is not correct.”

Test Case vs. Test Script

Keyword driven test script

27

Layer your framework – Test Level(3)

Data driven test script

28

Conclusion

Characteristics:

• Modularity• Configurability• Business oriented• Exception handling• Easily maintainable• Logging & reporting

Is applicable for :

• Smoke tests• Regression tests• Internationalization tests• Client side monitoring and

performance tests

Can be used to automate :

• UI Web application• UI Desktop application• UI Mobile application

A hybrid automation testing framework

29