bcs - sigist - test automation in the hybrid cloud

Jonathon Wright ([email protected])

http://www.linkedin.com/in/automation

BCS SIGIST Winter 2013 Conference (Thursday @ 2:45 PM, 5th December 2013)

Test Automation in the (Hybrid) Cloud

@jonathon_wright#BCS_SIGIST

www.sigist.org.uk

My starting point: Founder of the (Hybrid) Approach Pattern (2003)



The current economic climate is making companies

review their approach to IT even more closely

“We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant

scalability, flexibility, and availability for testing-on-demand. - But how well does this work in practice?”

• Why (Business Benefits & Value)

• What (Cloud Test Automation)

• When (Timeline)

• Agile Portfolio Management (Scope)

• Transition (AMMi)

• Dependencies (Cloud Type)

• Design (First Day Testing)

• Design (Manual’mation)

• Implementation (Agnostic)

• Cloud Platform (IaaS)

• High Volume Automation (HVAT)

• Continuous Build & Delivery

• Actionable Insight (Funnel Virtualisation)

• Dashboards (Real-time)

TESTINGAS A SERVICE

(TaaS)

INFRASTRUCTURE AS A SERVICE

(IaaS)

SOFTWARE AS A SERVICE(SaaS)

REAL-TIME VISIBILITYAND CONTROL

INSTANT SCALABILITYSCALE UP AND DOWN

HIGH AVAILABILITY

AMMI

PLATFORM AS A SERVICE(PaaS)

ISO 29119

PERFORMANCE TESTINGAS A SERVICE

(PTaaS)

PAY AS YOU USE(PAYU)

SLA’s

Background – The Journey to the Cloud

TEST AUTOMATIONAS A SERVICE

(TAaaS)

Background – Three Decades in Automation – Mid-Late 90s

@jonathon_wright#BCS_SIGIST Product Test Engineer

Gen Zero

(Linear)AUTSUT

Budget in pounds, company in thousands

MainframeLocations:2Teams:5Tool(s): 1

License x1

Background – Three Decades in Automation – Late 90s

@jonathon_wright#BCS_SIGIST Product Test Engineer

1st Gen

(Modular)AUTSUT

Locations:5Teams:10 Client ServerTool(s): 3

Licenses 10+

http://www.google.co.in/patents/US7526498

Budget in thousands, company in millions

Automation Framework ArchitectCore Hours 1am to 10am (EST)

Graduate TestersCore Hours 9am to 5pm (EST)

Automation EngineersCore Hours 9am to 5pm (GMT)

Background – Three Decades in Automation – Early 00s

@jonathon_wright#BCS_SIGIST Automation Consultant

2nd Gen

(Data)AUTSUT

Locations:5Teams:15 Client ServerTool(s): 5

Budget in hundreds of thousands

Licenses x25

Automation Consultants (UK)Core Hours 9am to 5pm (GMT)

Development (2 week sprints)Core Hours 9am to 5pm (GMT+1)

Background – Three Decades in Automation – Late 00s


Automation Framework Architect

3rd Gen

(Keyword)SUT

Locations:20+Teams:100+Tool(s): 15+

Budget in millions, company in billions

Web

AUT


Automation Consultants (NY)Core Hours 9am to 5pm (EST)

Automation Consultants 9am to 5pm (UTC+8)

Development Capability (US)Core Hours 9am to 5pm

Development Capability (EUR)Core Hours 9am to 5pm

Licenses 100+

Background – Three Decades in Automation – Early 10s

@jonathon_wright#BCS_SIGIST Global Head of Automation

4th Gen

(Hybrid)

Locations:25+Teams:50+Tool(s): 20+

Budget in millions, company in billions

AUTSUT

Cloud




Development Capability (EUR)Core Hours 9am to 5pm

Licenses PAYU

Background – Three Decades in Automation – Today

@jonathon_wright#BCS_SIGIST Agile Portfolio Management

5th Gen

(TaaaS)

Locations:100sTeams:1k+Tool(s): 50+

Budget 78 million, company investment 4 billion

Cloud

SUT AUT



wA

gile

Team

s (A

sia)

Co

re H

ou

rs 9

am t

o 5

pm

(U

TC+8

)

wAgile Teams (US)Core Hours 9am to 5pm (EST)

wAgile Teams (EUR)Core Hours 9am to 5pm (CET)


Background: Where's testers? – BCS SIGIST Winter 2011 Conference

Time Zone 1 Time Zone 2 Time Zone 3

Community Tester - Platform: Hybrid Community Cloud (e.g. MyDb.com)

Beta Tester - Platform: Public Cloud (e.g. sdod.db.com)

Alpha Tester - Platform: Public/Private Cloud (e.g. dbras.db.com)

Cloud Tester - Platform: Private Cloud (e.g. vpn.internal.db.com)

Internal Tester - Platform: Private Cloud (e.g. dod.internal.db.com)

Testing Command Center (Domain Specific)

• Supporting Core Testing Activities (Environment Monitoring/Management)

•Defect Monitoring & Risk Assessment (Social Enterprise Activity Feeds)

•Testing Management (Test Assets & Resource Enablement)

Solution Command Center (Solution Specific)

• End Client Dashboard/Reporting (KPIs)

• Solution Management (Programme Level)

•Service Provider Management (SLAs/Who does what when)

Background – Testing as a Service – BCS SIGIST Winter 2011 Conference


“Existing technologies of virtualisation, business process modelling, cloud based test automation tools and

rapid and easy internet access allow for the development of approaches that allow companies to order testing

as a service and pay only for what they use.”

Background – Test Automation as a Service


“Over the three years we have been developing a azure cloud based solution to provide true

"Test Automaton as a Service" using a hybrid automation approach specifically designed with the Hybrid

Cloud (Private/Public) in mind to utilise the instant scalability, flexibility and availability of the Cloud.”

www.TaaaS.net

http://www.taaas.net/

www.sigist.org.uk@jonathon_wright#BCS_SIGIST


“The current implementation is a service-oriented architecture (SOA) based on utilising business process

modelling to create business level keywords written in natural language (domain specific languages)

validated against content sensitive validation to create reusable, jigsaw pieces representing the Solution

Under Test (SUT).”

www.slideshare.net/Jonathon_Wright

http://www.slideshare.net/Jonathon_Wright

platform, technology, client, browser, version, language, test type agnostic

Executing Cloud Tests

feat. VM Dispenser Technology

Cloud Test Environmentsnatural language validated against

content sensitive validation

Creating Cloud Tests


service-oriented architectureBorn in the Cloud


Relevant Clear traceability of the business value of Automation through the visualisation

of the tests via Business Process Modelling (BPMNv2.2 compliant)

Effective Self-validating test assets achieved using natural language with context sensitive

validation against business and testing rules, workflows and data

Maintainable Self-maintaining test asset loader/scraper

Efficient Reports on SUT health including ratings such as percentage availability since

build/release, reported errors over time and traffic to error ratio

Manageable Unified platform which non-domain experts can use a natural language to

represent business processes and user story acceptance criteria

Portable

Technology agonistic - Platform, client, component, browser, version &

language

Test type agnostic – smoke, regression, integration & performance

Reliable Fault tolerance is built in to report and continue on different levels of fuzzy

matching combined with the non-technology specific test definition language

Diagnosable

Actionable defects provided by environment under test (EUT) live pause-

playback supported by dynamic data adapters (DDA) for accelerated defect

investigation and resolution

Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)

Summary - Test Automation as a Service (TaaaS.net)


www.sigist.org.uk

RIP Traditional Automation

Frameworks (1993-2013)

Panel Discussion - Should software testers be able to code?


“Here lies beloved architects, engineers and developers who invested a disproportional amount of time and

effort searching for the holy grail of test automation. Careers ended suddenly after a portfolio test manager

asked them to provide the actual business value of what they were doing.”

www.sigist.org.uk

Q1) How can we move to ‘Test Automation in the Cloud’?


“Existing technologies of virtualisation, business process modelling, cloud based test automation solutions and

rapid and easy internet access allow for the development of approaches that allow companies

to order ‘Testing as a Service’ and pay only for what they use (PAYU).”

Automation Ready

Cloud Power

Test Automation in the Cloud

Where are we going next?@jonathon_wright#BCS_SIGIST

Example Automation Projects Automation Level Investment in Effort Business Benefit

Description AMMi Gen Implementation Planning Execution Accel. ROE

Client Specific Implementation - Hybrid Approach 4 5 24 63 17 6.9 112%

Client Specific Implementation - BDD Approach 3 4 30 83 22 5.2 85%

Client Specific Implementation - Traditional 1 1 120 250 67 1.7 26%

Testing As A Service Model – Manual’mation 3 1 28 5 25 4.6 200%

Testing As A Service Model - Hybrid Approach 4 5 12 31 19 6.1 186%

TaaaS - Cloud Based Implementation 5 5 5.6 25 7 16.0 306%

www.sigist.org.uk

Q2) Are we ready for ‘Test Automation in the Cloud’?

Automation Maturity Model index (AMMi®)Assessment to provide a current automation maturity level and the target maturity level:

* Source: AMMi®(http://www.AMMi.org.uk) + Test Automation Patterns (http://TestAutomationPatterns.wikispaces.com)

AMMi Phase Target Criteria

Level 0 Accelerating Automated Test Lifecycle Methodology (ATLM)

Level 1 Traditional Framework Driven (Gen 1-5)

Level 2 Managed Processes are planned, performed, measured, and controlled

Level 3 Sustaining Automation process defined and validated against international

standards (ISO-29119)

Level 4 Quantified Predictability of automation process performance

Level 5 Optimising Automation process variation and statistical predictability


“Sometimes half the battle just starting companies on their automation journey by taking them one step

closer to becoming ready for automation.” AMMi Foundation 2003

Q3) – Which ‘Automation Tool Platform’


“My current role at one of the worlds largest investment banks. Their landscape consisted of over 300+

solutions under test (SUT), representing 2,700+ applications under test (AUT),

across 50+ global locations with over 300 automation practitioners worldwide.”

http://www.jetbrains.com/teamcity/

http://www.jetbrains.com/teamcity/

http://www.jetbrains.com/dotcover/

http://www.jetbrains.com/dotcover/

https://github.com/oneclick/rubyinstaller/wiki/Development-Kit

https://github.com/oneclick/rubyinstaller/wiki/Development-Kit

http://pm.versionone.com/

http://pm.versionone.com/

http://www8.hp.com/us/en/software-solutions/software.html?compURI=1172957

http://www8.hp.com/us/en/software-solutions/software.html?compURI=1172957

http://www.codeproject.com/Articles/289028/White-An-UI-Automation-tool-for-windows-applicatio

http://www.codeproject.com/Articles/289028/White-An-UI-Automation-tool-for-windows-applicatio

http://www.testautomationfx.com/

http://www.testautomationfx.com/

http://rubygems.org/gems/cucumber

http://rubygems.org/gems/cucumber

http://www.greenpeppersoftware.com/confluence/display/GPW/Home/

http://www.greenpeppersoftware.com/confluence/display/GPW/Home/

http://jbehave.org/

http://jbehave.org/

http://www.thoughtworks-studios.com/twist-agile-testing

http://www.thoughtworks-studios.com/twist-agile-testing

http://www.nunit.org/index.php?p=home

http://www.nunit.org/index.php?p=home

http://www.facilita.co.uk/

http://www.facilita.co.uk/

http://www.sikuli.org/download.html

http://www.sikuli.org/download.html

http://www.testplant.com/dlds/eggplant-functional/

http://www.testplant.com/dlds/eggplant-functional/

http://docs.seleniumhq.org/

http://docs.seleniumhq.org/

http://www.concordion.org/Tutorial.html

http://www.concordion.org/Tutorial.html

http://www.tricentis.com/en/

http://www.tricentis.com/en/

http://www-01.ibm.com/software/rational/welcome/greenhat/

http://www-01.ibm.com/software/rational/welcome/greenhat/

http://msbuildextensionpack.codeplex.com/

http://msbuildextensionpack.codeplex.com/

http://www.telerik.com/automated-testing-tools/

http://www.telerik.com/automated-testing-tools/

http://www.microsoft.com/visualstudio/eng/2013-downloads

http://www.microsoft.com/visualstudio/eng/2013-downloads

http://tfs.visualstudio.com/

http://tfs.visualstudio.com/

http://www.perfectomobile.com/

http://www.perfectomobile.com/

Q3) – Which ‘Automation Platform’ Axe 4 (Beta2)


A1

Login.Process

Request.VM

Access.VM

Manage.VM

Logout.Process

A3

A2

C1C3

D2C2D1

C4

E2D4 E1

E3 E4

D3

Business Process

Scenarios (BPS)

Business Process

Data (BPD)

Business Process

Modelling (BPM)

Business Process

Test (BPT)

Quick Overview - ‘First Day Automation’ Approach

@jonathon_wright#BCS_SIGIST www.sigist.org.uk

Moment in time

http://www.dorothygraham.co.uk/automationExperiences/index.html

http://www.dorothygraham.co.uk/automationExperiences/index.html

Start Cloud Tester

Modelling Tool (BizAgi or Visio)

Static Source

Dynamic Source

Cloud Source

Direct Source (xPDL or XAML)

Test Asset Loader(Test Asset Cube)

Business Process Modelling

BPT Tool(UFT or Specification)

Business Process Tests

Test Scheduler (WCF or REST)

TaaaS.net(Test Designer)

Q4) How do we become ‘First Day Automation’ Ready?

Agile Portfolio Management – BPM Export (Visio)


Agile Portfolio Management – Business Process Modelling


Solution Lifecycle Management - Business Process Scenarios


BPS Tool

(BizAgi or ALM/QC)

Direct Source (Workflow or XML)

Business Process Scenarios

Hyper-Test.com(Test Asset Journals)

www.eurostarconferences.com

Solution Lifecycle Management – Business Process Scenarios

HP Quality Centre (10+) / Application Lifecycle Management (11.5)@jonathon_wright#BCS_SIGIST

Solution Lifecycle Management – Business Process Testing


http://www.hyper-test.com/

http://www.hyper-test.com/

“Everyday business and consumer life creates 2.5 quintillion bytes of data per day and 90% of the data in the

world today has been created in the last two years?” with 1.5 petabytes of data across the current landscape.

Development SD-PM SD-TEST SD-AUTOMATIONVendors

Tier 3 - Sub-Domains – Application Level (x2,738)

Tier 2 - Domains – Solution Level (x330)

Domain Leads Domain StakeholdersDomain PMDomain Test Managers

Tier 0 – Programme Level (x4)

Heads of TestingHead of Automation Global Technology Office

Tier 1 - Workstream – Portfolio Level (x8)

Portfolio Test Managers Workstream StakeholdersWorkstream LeadsAPM Products Organisational

Agile

Portfolio

ReportingHybrid Cloud

Distributed AgileProject ArtefactsALM ProductsLocal Network

APM Products Global-widePublic Cloud Executive (Actionable Insight)

SLM Products Enterprise

AgilePrivate Cloud Enterprise

Reporting

Ch

art

er

Q5) What is ‘Agile Portfolio Management’?

Blu

ep

rin

tK

ey

Op

era

tin

g

Pro

ced

ure

s

Dyn

am

ic

Ag

ile

Pro

cess

es

Q6) How to provide ‘Actionable Insight / Automation Value’

Testing as a Service (TaaS) Dashboard (Realtime)

Risk Dashboard(Presentation Layer)

Benefit Analysis(Presentation Layer)

Story Board(Presentation Layer)

Business Risk

Project Risk

Technical Risk

Business Domain Language

Test Domain Language

Business Stories

Business Workflow

Business Rules

Business Process Design (BPD)(Visio Design Documentation)

Non-Functional Requirements (NFR)

(Dynamic Data Source (XML))

Detailed Session Definition (DSD)

(Dynamic Data Source (XAML))

Detailed Profile Definition (DPD)


Detailed Test Definition (DTD)


Business Components & Interfaces


Business Process Modelling (BPM)

(BPMNv2)

Business Process Scenarios (BPS)

(eXensible Process Definition Language)

Business Process Tests (BPT)


Business/Test Transactions Layer

Solution Under Test (SUT) Layer

Dynamic Data Adapters Dynamic Test Adapters

Business Questions

@jonathon_wright#BCS_SIGIST www.sigist.org.uk

Dashboards (Real-time) providing ‘Actionable Insight’

Theme: Travel@jonathon_wright#BCS_SIGIST

Automation Ready

Cloud Power


“To the cloud”@jonathon_wright#BCS_SIGIST

www.sigist.org.uk

Q7) Should we move to the cloud?

New Challenges & Opportunities ? and/or

A match made in heaven?



Co

nn

ecti

on

Systemic Failure

(Network or DNS)

Alternative Service Provider?

New Challenges

What happens when these services FAIL?

Unique Services

(Such as Banks)

"There's normally a role that technology plays in

the outage, but when I look at the root cause, by

far the greatest cause is people and processes."

Damian Saunders, a cloud networking director

“For decades, RBS failed to invest properly in its

systems” Ross McEwan, Chief Executive, RBS, December 3rd 2013

Diagram Source - The Federal Reserve Bank of New York, Shadow Banking System, 2012 (http://www.newyorkfed.org/research/staff_reports/sr458.pdf)

“Don’t assume the ecosystems (commercial or community) and

other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)

http://www.newyorkfed.org/research/staff_reports/sr458.pdf

Consumer Freedom


“Enable sophisticated modelling of the components of cloud deployments,

including servers, storage, databases and data.

For the consumption of cloud resources” - PlanForCloud.com (2012)

CompareTesting.com

Instant scalability

Flexible Test Execution

Flexible Test Assets Environments on-demand

Geo-based execution

Instant Accessibility

99.9% Availability (100% Web)

Instant Portability

Best of Breed (BoB) tools on demand

No upfront investment

Pay As You Use (PAYU)

New Opportunities

www.sigist.org.uk


PrivateCombination

Public

SUT SUTSUT

Community

Clouds

Public

Clouds

Private

Clouds

TaaaS.net TaaaS.net

“... the prediction that 60% of the average enterprise will have 60% of its applications in the cloud.”

- Karl Flinders, Computer Weekly (2011)

“IBM are working with other cloud suppliers to establish the standards for cloud service provider compliance.”

- Murat Gunenc, IBM Cloud Representative @Rhodes W1 (2011)

Q8) Cloud Type

Automation Ready

Cloud Power


The Final Frontier@jonathon_wright#BCS_SIGIST

HybridCloud

PrivateCloud

Storage

Networking

Servers

Virtualization

Applications

Infrastructure(as a Service)

Runtimes

Databases

Continuous Integration,

Build & Deployment

PublicCloud

CommunityCloud

Q9) How can we move to ‘Test Automation in the Cloud’?

Lab Management

Test Controller(s)

Test Agent(s)

High Volume

Automated Testing (HVAT)


Test Agent(s) – Telerik Art of Testing Framework


High Volume Mobile Automated Testing (HVMAT)


High Volume Mobile

Automated Testing (HVMAT)

VMM Library

Data Protection

VM Factory

Data Masking

Data

Co

ntr

oller

–(C

om

plian

ce w

ith

DPA

)

Data

Pro

cess

or

–(W

ritt

en

leg

al co

ntr

act

)

Data

Pro

vid

er

–(E

uro

pean

Eco

no

mic

Are

a)

Feature(s) Work ID

Code

Team Foundation Service

TFS - Gated Check-in!

CancelOK

Build Server

Backlog

VS2013 x64 Cross Tools Command

Microsoft Windows [Version 8.1.9431]C:\Automation\>mstest /testlist:MasterDriver/testmetadata:”TestProject.vsmdi” /testsettings:”MobileDevices.runsettings”

Unit Tests

Feature-Driven Exploratory Test Case(s)

Issue(s) Action(s) Log ScreenCaptureIntelliTrace

Traffic Proxy

Continuous Deployment Test Controller(s) Test Agent(s) Browser(s)

Client(s)Load Agent(s)

Interface(s)

Solution Under Test

Application Under Test

ASP.net | DynoForms (Ajax/Json/Spoc)

Application Server(s)

Database Server(s) Shim/Stub(s)

Shim/Stub(s)

Business Process Scenarios (BPS)Business Process Tests (BPT) Business Process Model (BPM)

Business Process Data (BPD)Business Process Reporting (BPR)

Q10) How can we execute ‘Test Automation in the Cloud’?

Solution Lifecycle Management – Business Process Tests

“verbs and nouns in the test specific language are written in natural language which using context sensitive validation.”

Logi

n.P

roce

ss

Action Object Event

Enter Username Text

Enter Password Text

Press Login Button

Domain Specific Language

Act

ion

s

Ob

ject

s

Even

ts

Press | Login | Button

Nouns

Verbs

Test Specific Language (TSL)


“The test editor is designed to be low waste, it will only offer nouns and verbs based on discovered artefacts not programming objects.”

Agile Portfolio Management – Business Process Scenarios


Agile Portfolio Management – Business Process Modelling


Agile Portfolio Management – Agile Feature(s)

Estimate Effort?@jonathon_wright#BCS_SIGIST

Agile Portfolio Management – Product Backlog

Microsoft Project 2013 (Connected to TFS)Team Foundation Service (Product Backlog)


Agile Portfolio Management – Work Items > Tasks

Team Foundation Service TFS Workbench

Digital Analogue


Agile Portfolio Management – Feature-Driven

@jonathon_wright#BCS_SIGIST www.eurostarconferences.com

Agile Portfolio Management – Exploratory Testing

@jonathon_wright#BCS_SIGIST www.eurostarconferences.com

@jonathon_wright#esconfs

Agile Portfolio Management – Test Case(s)

Application Lifecycle Management – Unit Test(s)


Solution Lifecycle Management – Code



Solution Lifecycle Management – Issue(s)

Test Case Migrator Plus (TFS)


Application Lifecycle Management – IntelliTrace


@jonathon_wright#esconfs

Application Lifecycle Management – Action(s) Log Application Lifecycle Management – Screen Capture(s)


Solution Under Test – Traffic Proxy


Solution Lifecycle Management - Interfaces


Application Lifecycle Management – Client(s)

Real Mobile Devices (PerfectoMobile) Test Studio for iOS (Telerik)


Solution Lifecycle Management – Build Server


Solution Lifecycle Management – Continuous Deployment


Solution Lifecycle Management – Test Controller(s)

www.eurostarconferences.com@jonathon_wright#BCS_SIGIST

Load Agent(s) – Visual Studio 2013 – Load Test Manager



Solution Lifecycle Management – Test Agent(s)


Cross Browser Testing – Telerik (Art of Testing - Framework)


Agile Portfolio Management – Business Process Data

ALTOVA XML Spy 2013 (R2 SP1)



Agile Portfolio Management – Reporting - Burndown

TFS Workbench

Monitor velocity & capacity


Agile Portfolio Management – Reporting - Test Execution

Microsoft Test Manager 2012.R3@jonathon_wright#BCS_SIGIST


“The best aspects of these proven test approaches

demonstrate how they have evolved over the past

decade, and this echoes some of the changes towards

more lean and agile business methodologies. They are in

a constant state of evolution – just as the underpinning

technology evolves over time.”

My section in the ‘Experiences in Test Automation’ book

(Dorothy Graham & Mark Fewster) 2012.

Final Thoughts - Test Automation in the Cloud

www.sigist.org.uk

Contact Details – Jonathon-Wright.com

Testing Blog (www.Right-Testing.com)

Automation Blog (www.TaaaS.net)


LinkedIn + Email ([email protected])

www.linkedin.com/in/automation

Testing-Alliance.com

http://www.right-testing.com/








http://www.testing-alliance.com/