continuous testing in practice - amazon web services · 40% manual testers. the continuous testing...

Continuous Testing In Practice

Feb 2019

Speaker

2

Guy Arieli

CTO

Agenda

3

What Is Continuous Testing?01 10 min

Demo03 20 min

02 Case Study 10 min

Q&A04 10 min

What is Continuous Testing?

Executing automated tests as part of the Continuous Integration pipeline for immediate feedback on the risks of a release candidate

4 Developer A

master branch

CI Server

Build

Test

Evaluate

Feature branch

Co

mm

it

Pu

ll

req

ue

st

Developer B

Feature branch

Co

mm

it

Pu

ll

req

ue

st

Agenda

5

What Is Continuous Testing?01 10 min

Demo03 20 min

02 Case Study 10 min

Q&A04 10 min

6

Leading North American transportation company

Case study background

• 3 apps

• R&D organization:

~ 200 developers in the US

~100 QA - offshore in India with SI

• Use Appium + Selenium

• Process started 2 years ago

Background

• Test automation does not identify many bugs

• Automation stability issues

• QA Environment issues – availability, low use case coverage

• Disconnect with R&D on process

• Feedback loop is 1-2 weeks

• Automation has low business impact

Challenges

Slow R&D process, low quality, negative business impact

➔Speed up R&D progress

➔Improve quality

➔Shorten time to market

Objectives

7

Organizational structure and communication flow

8

Siloed structure DevOps structure

OpsDev Engineer TesterOpsDev Engineer

Tester

OpsDev Engineer

Tester

OpsDev Engineer

Tester

9

Case study: North American financial institute

Drive for more automation requires workforce transformation

Performance & Volume

5%

Environment and Data10%

25%

, 60%

Automation Testers & Software engineers in test

Exploratory testers

Performance & Volume10%

Environment and Data20%

Automation Testers30%

Manual testers40%

The Continuous testing dev-test process

Epic Story thinning Acceptance

Criteria (user story scenarios)

Acceptance Criteria

(feature files)

Code incl. White box Test

(step definitions)Build & Deploy

Automated testing

Acceptance testing

Showcase / Demo Sign-off

Dev Engineer

Dev & Test EngineerDevOps EngineerDev & Test Engineer

Functional Spec

Product Owner

Analyst Product Owner

Analyst Owner / analysts

Tester

TesterProduct Owner

CI/CD cycle

Dev Engineer

Tester

Development Engineer

Fast feedback with continuous testing

Commit step definitions + feature files

Trigger test execution

Load app to cloud

Tester + Test

Engineer

Pull request to the main branch

Consolidate test results

Continuous Integration

reporttest

build Pull request Approved?

no yes

Refuse pull requestDebug / update test suite

1Pull app source2

Build app3

Pull test source4

Build test project5

6 7 8

9

10

Git Hook

Merge to master branch

Distributed Digital Assurance Lab

Single unified UI externalizing different geographical locations

12

60 devices & Browser

40 devices & Browsers

Agenda

13

What Is Continuous Testing?01 10 min

Demo03 20 min

02 Case Study 10 min

Q&A04 10 min

14

• Jenkins

• Git

• Digital Assurance Lab

• Pre-prepared tests

Tools used

15

Demo

16

Process outcome

• Regression feedback within minutes

• New feature feedback within 24 hours

• Environment issue handling down from days to hours

Process

• Story points produced by R&D *3

• Quality improvement

• improved app store rating

• Higher customer satisfaction

• Market feedback

Organizational impact

17

Fast feedback vs. Coverage mode

Best Practices for test creating and execution

• Usually used by developers

• Purpose is quick validation

• Execute immediately on any available device

• Automatically Triggered by pull requests

• Usually used by automation team

• Purpose is ensuring app works across devices

• Scheduled nightly executions

• Run tests on all specified devices

Fast Feedback Coverage Mode

Agenda

18

What Is Continuous Testing?01 10 min

Demo03 20 min

02 Case Study 10 min

Q&A04 10 min

• Short / small tests

• Limit the test selection by time (5 minutes). Grow in device number if needed

• Select tests that ‘touch’ the entire system

• Remove naive tests

• Ignore device matrix and execute in fast feedback mode

Continuous Testing Best Practices

Continuous testing ≠ test automation

19

Questions?▪Schedule a meeting to see how Experitest can help:

▪sales@experitest.com

21