automation as a project - lessons learnt

Poznań Testingand Quality Group

Automation as a project –

and Quality Group

Automation as a project –lessons learnt

Zbyszek Moćkun

Agenda

• Introduction• Introduction

• Business

• Technology review

• Management

• Summary

• Discussion

Few words about the author

Introduction

Few words about the author

Experience and domains

Experience Domains

• 7 years in tests

• Three different companies

• Several domains

• Product based and project

• Different methodologies

• Mobile infrastructure

• Billing and Roaming

systems

• Business Assurance

application

• Medical

• Content Management

• Different methodologies

(Waterfall, Scrum, Kanban)• Content Management

Systems

Automation vs. profitability

Business

Automation vs. profitability

It’s all about the money

http://www.br-automation.com

How to talk with stakeholders

• Time of money return• Time of money return

• Use statistics/metrics

• Remeber that statistics show incorrect data (only regression

issues should be calculated)

• It’s not about how many issues were found

• Statistics fall down without continuous investement

• Code resistance on the same tests

• Decrease release time (time for manual tests)

• Number of issues found influence on time needed for

regression

How to start?

• Show metrics• Show metrics

• Why does we need it?

• How much does it cost?

• What does we want to achieve?

• What are the goals?

• Small steps

• First step should be cheap and quick

• Show results for stakeholder

• Plan and estimate next steps

• Gather metrics and statistics

Approaches

Product-based

Project-based

Product against project

Product-based Project-based

• Scalability

• Modularity

• Coverage – as much as

possible

• Expensive - infinite number

of runs

• No need of scalability

• Only basic modularity

• Coverage – main features

• Cheap - finite number of runs

• Reuse for other projects (not

always possible)

• No place for planning new

• Reuse for other products

• Planing new features and

improvements

• No place for planning new

features and improvements

• Manual tests are cheaper at

the most cases

Technology

Automation

Technology

One place for automation management

Continuous Integration

One place for automation management

Where should automated tests be run?

• On developer machine if possible, before commit

• As sanity check on other machines

Developer machine Nightly Iteration Staging Production

• As sanity check on other machines

• Developer iterative work (migration as example)

• Should we run autoamted tests on production (live)?

• Independent test data

machine

CI tool

Reports

• Nightly tests – preferred frequency

• Build results management (disc space)• Build results management (disc space)

• Reports improvements

• Minimalize time spend on report (verification/debugging)

• Display as much data as possible for failures

• One issue failing several tess script

• Environment errors - stability

• Red reports effect

• Issues are not fixed

• A lof of failed tests cases

• Noone look at the results

Reports

How to build good test suite?

Dependent or

independent?

How to build good test suite?

Independent and random

• All tests are independent

• Pre – precondition (set require state)

• Post – postcondition

• Test – test script

• Rec – recovery procedure after test failure (pre i post should have defined too)

• LogData – gather all logs after failure, very important for „random” failures

• Random order

• Repeat counter

• Dependent only for CI and adult teams (founded issues are

fixed as fast as possible)

How to manage test data? Load them or

Test data

How to manage test data? Load them orcreate?

Does it work?

Record & Play

Does it work?

Selenium IDE as example

• Stability issues

• Recorded values are not meaningfull• Recorded values are not meaningfull

• Require rewrite

• User-extension.js (write own commands)

• Quick and easy debug

• Project-based – preferred as cheap and quick

• Product-based – as starting point

• More info link

Product against project

• Quick and easy (+)

Cost return (+)

HTML Java

• Cost return (+)

• Coverage – max 10%-20% (-)

• Modularity using user-

extension.js (=/-)

• No need to environment

setup (+)

• No scalability (-)

• Cnvert to other languages

• Flexibility (+)

• Scalability (+)

• Modularity (+)

• Coverage (+)

• Development knowledge (+/-)

• Environment setup (-)

• Cnvert to other languages

• HTML Suites (works with CI)

• Integrated reports

• No convert possibility (+/-)

• Debug time (-)

• Reporting (JUnit, TestNG) (-)

Survey

Two similar application and different technologies based on

Selenium: Selenium:

• HTML based on Selenium IDE

• Java based on Selenium RC/Eclipse

HTML – 50% more test cases written

Java – 20% more time spend on maintenance/debugging

Which technology would you like to use?

Java devs choose HTML/Selenium IDE as more friendly

and faster

Management

Automation

Management

Manage, monitor and plan

Coverage

Manage, monitor and plan

20/80 rule

Coverage

Manage your coverage and use it for planning manual tests

• Remember about 20/80 rule

• Power function

• 100% - is it possible?

• Network architecture as example (100% achieved)

• Human factor caused that 100% is not achievable

• Red report effect

Defect management

• Red report effect• Quick fixes for CI• Labels

Who should write and maintenance

Automation responsibility

Who should write and maintenanceautomated tests?

Developer helping hand

Can developer deliver automated tests together with

functionality?functionality?

• Who should write test cases?

• Scripts written using invalid test cases

• Scripts written for not working functionality

• Lack of time – functionality is more important

• Control needed (sometimes devs forget)

• Two unsuccessful attempts

Triangle: Budget, Time, Scope (Aim)

Automation as a project

Triangle: Budget, Time, Scope (Aim)

There is no success without:

• Continuous work• Gather statistics for stakeholders• Gather statistics for stakeholders• Divide budget on manual and automated tests• Planning the future – where want to be in next year?• Control your costs• Remember – it’s all about the money

Automated Exploratory Tests for Regression

The future

Automated Exploratory Tests for Regression

Aspects

Client

AnalyticsLayout Functionality Performance

Symptoms

Layout

Content

CodeLink checker

URL strategy

JS errors

Response (404)Analitycs

Heuristic Oracle

The balance between comparison and comparison not at all

Layout Content Code Performance

Robots

• Scripts• Shared between projects• Shared between projects• Easy and quick to run

JS errors Responses(404) W3C Analytics Link checker Url strategy

Poznań Testing and Qaulity Group

Discussion & beer

Poznań Testing and Qaulity Group

automation as a project - lessons learnt

cognifide limited

commercial confidence

release time time

tests billing

autoamted tests

manual tests number

improvements manual

test datahow

Education

ucpm lessons learnt programme

lessons learnt vpuu whole

lessons learnt part 6

lessons learnt part 5

lessons learnt (march 2011) 1 lessons learnt

lessons learnt

lessons learnt part 4

lessons learnt by living

container terminal automation experiences & lessons...

community fiber lessons learnt

automation: lessons learnt from sesar by michael standar

mvp academy: lessons learnt

lessons learnt -whitepaper from openminted...

lessons learnt, challenges faced_pp21_final

integrity lessons learnt

learndesk -project – lessons learnt

pbmr fuel development laboratories - lessons learnt · ·...

aus heatstroke lessons learnt

lessons learnt report

lessons learnt from nlep