Test Automation – 10 (sometimes painful) Lessons learned

Michael Palotas & Dominik Dary Quality Engineering Team Europe

eBay International

April 2012

Founded in 1995

Based in San Jose, California

27770 Employees worldwide

eBay Inc. Facts

104 million active buyers and sellers worldwide

68 Billion USD transaction volume 2011

2000 USD

transaction volume every second

Platform Facts

9 petabytes of data In our Hadoop and Teradata clusters

2 billion page views each day

75 billion database calls each day

Data Facts

70,000,000 times eBay Inc’s apps have been downloaded

5 billion USD transacted 2011 on mobile devices

1 million items are listed with using mobile apps every week

Mobile Facts

Who we are

Michael Palotas Head of Quality Engineering Europe

E-mail: mpalotas@ebay.com

Dominik Dary Staff Software Engineer in Test

E-mail: ddary@ebay.com

Inspiration: http://patrickwilsonwelsh.com/?p=32

Lesson 1: Flip the testing triangle

Unit

Tests

E2E Automation

Tests

Manual Tests

Unit Tests

E2E

Tests

Manual

Tests

Integration Tests

**Swiss Testing Day Presentation 2010: “Mit Testaspekten früh, intelligent und redundanzfrei testen” Melanie Späth

Image Source: http://www.masternewmedia.org/images/social_software_impact_individual_organizations.jpg

Lesson 2: Everyone needs to know what is

automated

Use Test Aspects**

• Test aspects briefly describe

“why” and “what” has to be

tested, without describing

“how”

• Enables for early testing

because tests can be written at

the same time the user story is

created

• Test aspects are structurally

derived

Advantages

• Clear overview for all team

members about test coverage

• Simplicity helps you to do

quick reviews and

walkthroughs with your product

owner

• Enables you to identify manual

and automated test duplicates

• Traceability between test

cases and specification

A test tool/testing framework won't

teach your testers how to test

A clear testing process is mandatory

for test automation

Test Aspects and modeling of the

Biz-Domain layer is the key success

factor for your test strategy

Lesson 3: A Tool is not a Strategy

Lesson 4: Write the right tests

Source: http://farm1.static.flickr.com/119/269335955_03a0ff4b67_o.jpg

Automate the business critical use

cases of your customer

Focus on end-to-end testing rather

than UI component tests

Design testability into your

product

Lesson 4: Write the right tests

Lesson 5: Instant Feedback is essential

D T D T D T D T D T D T D T D T

Development Test

Lesson 6: Automation is software development

Skillset & Knowledge

• You need people with software

development skills and an

appetite for testing

• Automation mindset is essential

• Good understanding of the

architecture of the application

under test is required

Engineering practice

• Task oriented development

helps you to get things done

• Follow the basic rules like lean

development principles

• Design the test before coding it

• Do code reviews

• Your developers can give you

very good feedback about your

code quality

Lesson 7: Maintainability is king

Sourc

e:

htt

p:/

/geekandpoke.t

ypepad.c

om

/.a/6

a00d

83

41

d3

df5

53

ef0

134

86

02

ac19

97

0c-p

i

Definition of Done

Reviews

Document the essentials

Don’t build a framework

Long Leash

Lean Test Automation • Eliminate Waste

• Build Quality In

• Create Knowledge

• Defer Commitment

• Deliver Fast

• Respect People

• Optimize the Whole

Lesson 7: Maintainability is king

Technical Debt • Activities that result in

technical debt include

• Postponing

documentation and

writing tests

• attending to TODO

comments and

• tackling compiler and

static code analysis

warnings

• Knowledge that isn't

shared around the

organization

• Code that is too confusing

to be modified easily

Use the same programming

language as development

Developers can write or maintain

tests if needed

Enables for a cross functional team

where everyone can work on nearly

every task

Lesson 8: Speak the same language as the

developers

Lesson 9: Invest into the test infrastructure

Speed up and simplify

the creation of test

data

Offer and use web

services

Web based test data

creation tools can be

used in test auto-

mation and manual

testing

Automated tests only do what you have implemented:

at the same speed, same order, with exactly the same

mouse clicks

Manual testing allows you to go off the beaten path

Test automation in combination with exploratory

testing combine the best of both worlds

Lesson 10: Manual testing is still very important

Quality = Manual + Automated Testing

Conclusion of our ten lessons learned

1. Flip the testing triangle

2. A tool is not a strategy

3. Everyone knows

what’s automated

4. Write the right tests

5. Instant Feedback is

essential

6. Automation is software

development itself

7. Maintainability is king

8. Speak the same

language as the

developers

9. Invest into the test

Infrastructure

10.Manual testing is still

very important

Thank You! Questions or Comments?