testdroid: maximize the benefits from your test automation investment

1

webinar

Maximize the Benefits from Your Test Automation Investment

14 November 2013

Ville-Veikko HelppiTechnical Product [email protected]

mailto:[email protected]

3

Agenda• Common Challenges in Today’s App Lifecycle• How Test Automation Can Shorten Your App’s

Time-to-Market and Improve the Quality?• Return on Investment with Test Automation• Different App Verticals – What is Important?• 10 Tips for Professional Mobile App Testing• Testdroid Update• Q&A

webinar

© Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved.

4



webinar


5

Challenges in Today’s App Devwebinar


• Failures: Both SW and HW cause failures• Emulators do not offer realistic platform for dev• Already installation of app fails – not tested at all • Application crashes, leaks memory, consumes too

much resources, behaves not as meant to behave• UI scaling/display/resolution – common problem• Application performance is bad, draining battery

Fact: Most typical application fails when tested across different device variants used by consumers

6

Why Do They Fail?webinar


7



webinar


8

How to Test: Manual vs. Automation• Costs & Assets– Time, People, Infrastructure, Tools, Training

• App Development & Testing Cycle – Together– Testing after Daily Builds (Continuous Integration)

• Tools & Technology – Frameworks– Myriad of frameworks available to help you

• Test Coverage & Reusability– Global coverage is CRITICAL to SUCCESS

• Outcome – Learning, Improving, Faster

webinar


Manualwebinar

© Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 9

Automation

Smaller coverage, More money burnt & time wasted, Error-prone

Large coverage, quickly

completed, Less money & time wasted, Exact

results.

10

Improving Time-to-Marketwebinar


Com

mit

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Users / Consumers

Rele

ase

11

Tips to Improve the App Quality• Automate generic things as much as you can• During the development your app changes –

the same must apply for testing!• Carefully select (testing) technology & partner• Use all 24 hours per day with test automation• Cloud-based platform is the only solution to

get you quickly covered in the global markets

webinar


12

Test Automation Will Deliver You…• App Compatibility & Global coverage

• Reusability• Test efficiency & effectiveness• Faster time-to-market

webinar


Test Automation during the Mobile App Development Process

Test

requ

irem

ents

/ P

lann

ing

Smoke testing

Regression testing Regression testing Regression testing Regression testing

Functional & Interoperability testing



Performance & Stress testing Performance & Stress testing

Compatibility testing

13



webinar


14

Example: Cost of Manual Testing• Test setup for each device – 5 minutes • 100 test cases, each take about 15 seconds to execute

– 25 minutes• 4 different application variants: tablet, phone and 2

localized versions (English & German)• Avg. 2 bugs (of 100 test cases) are found on each test

run – 20 minutes to report bugs• 15 devices will be used for testingTIME FOR TESTING: 3000 min (50 hours)COST FOR TESTING: $60/h x 50 = $3,000 (x4) = $12K

webinar


15

Example: Cost of Test Automation• Creation of test cases/scripts – 100 test cases running 15 minutes

per each = 1500 minutes (25h) COST FOR TEST CREATION: 25 hours x $60/h = $1,500

• Tools for test automation with the following setup:– Unlimited devices (250+ Android devices)– Unlimited execution times

TESTDROID CLOUD: $1,500/month

TOTAL COSTS FOR TEST AUTOMATION: $3,000/monthDirect savings compared to manual: $9,000ROI ratio: (Gains-Investm.) / Investm.: $9K-$3K/$3K = 2.00

webinar


16

Benefits with Testdroid Cloud• Instant availability of hardware and no need to

buy any device by yourself• Testing gets done much faster – typically the

same outcome can be achieved in 10% of time• Test cases are reusable for your next project• Immediate feedback – all reports as logs,

screenshots, performance stats etc.• QA can focus on quality – not pressing keys!

webinar


17

Example: On-Premise Setup• Creation of test cases/scripts – 100 test cases running 15

minutes per each = 1500 minutes (25h)COST FOR TEST CREATION: 25 hours x $60/h = $1,500

TESTDROID ENTERPRISE: $2,000/month (15 devices)

TOTAL COSTS FOR TEST AUTOMATION: $3,500/monthDirect savings compared to manual: $8,500ROI ratio:

(Gains-Investm.) / Investm.: $8.5K-$3.5K/$3.5K = 1.43

webinar


18



webinar


19

Mobile Apps for Different Verticalswebinar


Mobile

Games

Utilities & Tools

Video Streaming & Multimedia

Banking & Payment

Retail & Travel

• Testdroid has helped thousands of app developers in these subverticals!

• What are the critical elements in each these verticals?

20



Mobile

Games

Utilities & Tools


Banking & Payment

Retail & Travel

• User Experience!• Resource consumption &

validation (CPU, Mem…)• Fully utilizing hardware –

e.g. Touch screen• Mobile games require

access to graphics APIs (e.g. OpenGL ES)

21



Mobile

Games

Utilities & Tools


Banking & Payment

Retail & Travel

• Usability!• Functional and

Behavioral testing • Metrics analysis for all

captured data (e.g. logs, screenshots, perf stats)

• Connection with Social Media + other apps

22



Mobile

Games

Utilities & Tools


Banking & Payment

Retail & Travel

• Performance!• Connectivity, robustness

and durability• Screen orientation

(portrait vs. landscape)• Graphics quality,

streaming capabilities• User profiles

23



Mobile

Games

Utilities & Tools


Banking & Payment

Retail & Travel

• Security!• Secure connectivity with

back-end systems• Top Quality – extremely

brand sensitive vertical• Compliances and

verification between real devices and infrastructure

24



Mobile

Games

Utilities & Tools


Banking & Payment

Retail & Travel

• Data!• Connectivity and data

connection with back-end systems

• Configurability of the app• Bad quality can hurt the

brand and leave you without customers

25



webinar


27

Tip #1: Test early, Test oftenwebinar


28

Tip #2: Plan What to Automatewebinar


People

Infrastructure

Tools

Training

Time

MAN

UAL

AUTO

MATED

29

Tip #3: Use only Real Deviceswebinar


30

Tip #4: Use Atomic Test Unitswebinar


Test iteration #1 Test iteration #2

31

Tip #5: Create Hermetic Testswebinar


32

Tip #6: Use All Possible Deviceswebinar


Testdroid Cloud’s 250+ Android devices = 93-95% global Android volumes!

33

Tip #7: Separate Apps and Testswebinar


Your App

34

Tip #8: Output Everything to Logswebinar


35

Tip #9: End-to-End Testingwebinar


36

Tip #10: Integrate Dev & Testingwebinar


Com

mit

Developers

Source code

Repository

Build failed

Polling

Notifying

Successful build

QA & Testers

Building

Error reporting

Test Cases

Application

Rele

ase

Users / Consumers

37



webinar


38

Testdroid Productswebinar


Complete Solution for Mobile Apps/Games Testing

39

webinar


Testdroid – Because it is important to know what app testing costs!

40



webinar
