Deadly Traps of Test Automation

STC JULY 2014

THAO VO

A landscape

2

1. Automation idol

2. UI addiction

3. Over misusing on commercial tools

4. Poor collaboration

5. Lazy maintenance

6. Intensive ROI

Source: http://www.lifehack.org/articles/productivity/how-to-manage-common-productivity-traps-for-improved-productivity.html

3

Trap #1: Adore Automation

Source: http://www.theguardian.com/media/shortcuts/2014/mar/16/could-robots-be-journalist-of-future

Trap #1: Adore Automation

Symptoms • Depending on automation for all testing

activities

• Absence of code reviews

• Absence of exploratory testing

• Absence of UX testing

• All tests are built by developers

4

Source: http://www.calgolftech.com/1-problem-with-buying-golf-clubs/

Trap #1: Adore Automation

5

Treatments Prevent comparison between

manual and automation

Use automation in conjunction with manual techniques

Use automation to provide a baseline – a sanity check

Source: http://cureforhives.org/chronic-idiopathic-urticaria-treatments

6

Trap #2: Love UI so much

A Pyramid

7

GUI

Tests

Acceptance Tests

Integration Tests

Unit Tests

Trap #2: Love UI so much

Symptoms • Limited technical skills

• Cannot understand the application or system architecture

• Huge proportion of tests are being run via the UI

• No collaboration with developers

• Difficult to setup test context

• Test results are slow & fragile 8

Source: http://www.bostonsearchgroup.com/blog/ceo-search-break-through-medical-devices-for-wound-closure/

Trap #2: Love UI so much

Treatments • Limit the investment in UI

automation

• Invest more at lowest possible level with clear test purposes

• Collaborate with developers

• Make sure automation gives a fast feedback

9

10

Trap #3: Over misusing on commercial testing tools

Source: http://www.fastcoexist.com/1679718/your-bad-eating-habits-are-contagious

Trap #3: Over misusing on commercial testing tools

Symptoms

11

A commercial tool forms the basis a testing strategy

Only certain teams or individuals can access the tool or run tests

Developers and testers have not been consulted in tool selections

Source: http://www.sfgate.com/news/article/Apple-exploring-cars-medical-devices-to-reignite-5239850.php

Trap #3: Over misusing on commercial testing tools

Treatments • Use open source tools wherever

possible

• Use tools that can be extended or integrated into existing development tool chain

• Use tools that match specified test intent

• Use clear matrix tool evaluation

• Keep educating!!!

12

Source: http://stugalactic.co.uk/student-shopping-medicine.aspx

13

Trap #4: Too pride to collaborate when creating

tests

Source: http://www.jonathankettleborough.com/tag/collaboration-2/

Trap #4: Poor collaboration

Symptoms • Automated testers are isolated

from team: duplicated test scripts, too much tests but inefficiency

• Poor automation design

• Maintainability or compatibility issues

• No definition of quality: unclear, not match the desired system quality

14

Source: http://thaineuro.org/category/pharma-and-healthcare/healthcare/

Trap #4: Poor collaboration

Treatments • Collaborate to create good tests

and prevent duplication

• Ensure good technical practices and design

• Collaborate to improve quality definition and verification

• Limit investment in UI based automated tests

15

Source: http://medicine1.info/medicine/

16

Trap #5: Too lazy maintenance

Source: http://www.tastyhuman.com/the-illusion-of-8-hour-workday-are-you-hard-working-or-just-lazy/

Trap #5: Too lazy maintenance

Symptoms • Test suite has not been run

recently – state is unknown

• Test suite requires manual intervention

• Duplication within automation code

• Small changes trigger a cascade of failures

17

Source: http://orangedocfamilymedicine.com/

Trap #5: Too lazy maintenance

Treatments • Automated tests should be

executed using a CI environment

• Ensure tests are always executable – even if the system is not being actively developed

• Ensure collaboration between developers and testers

18

Source: http://www.psmag.com/blogs/the-101/the-spectrum-of-color-response-take-your-medicine-51195/

19

Trap #6: Cut costs through automation

Source: http://insuranceknock.com/questions/155/saving-money-with-2-phrases

Trap #6: Cut costs through automation

Symptoms • Using a predicted ROI to

reduce testing budget

• Hidden investments are not counted

20

Source: http://bloomintegrativehealth.com/news-updates/

Trap #6: Cut costs through automation

Treatments • Clear objectives and reasons

for automation

• Cost-benefit analysis should include ongoing maintenance

• Quality is the key

21

Source: https://today.duke.edu/2013/03/operationmedicinedrop

Summary

22

23

Q&A

References

• [1]. Testing Experience – published No. #21, #23 - http://www.testingexperience.com/

• [2]. 7 Deadly Sins of Agile Software Test Automation – Adrian Smith http://www.slideshare.net/adrianlsmith/7-deadly-sins-of-software-test-automation

• [3]. Manual Test cannot be automated http://www.satisfice.com/blog/archives/58

• [4]. Testing Club Vietnam 2014 Trends in SW Testing – Paul Holland

24

Profile

• Thao Vo – QA Architect at KMS Technology Inc. (http://www.kms-technology.com/); Board member of HCMC Software Testing Club (http://www.hcmc-stc.org/).

• Email: thaovo@kms-technology.com or thaovo@hcmc-stc.org or vvthao@yahoo.com

• LinkedIn: http://vn.linkedin.com/pub/thao-van-vo/24/31a/b35

25

© 2014 HCMC Software Testing Club

THANK YOU