eurostar - test automation in the cloud - webinar

Today’s webinar is presented by Jonathan Wright and he will discuss Test Automation in the Cloud.

Welcome to the EuroSTAR July Webinar Series

www.eurostarconferences.com

This webinar is due to start at 2pm. Make sure you stick around at the end for the Q&A session and continue the conversation with the speaker on Twitter after the show!

@esconfs#esconfs

Test Automation in the Cloud

Jonathon Wright, Founder, Automated Development Services, UK

Jonathon Wright has over 13 years of commercial automation experience with a number of international organisations. Currently working on providing ‘Test Automation as a Service’ to

a number of global clients. Jonathon also contributed to the recent "Experiences of Test Automation: Case Studies of Software Test Automation (2012)" and a number of upcoming

books on test automation in the cloud. He is an active blogger on "Test Automation as a Service" (TaaaS.net) as well as presenting at various international testing conferences

(StarWEST(California)/Fusion(Sydney)/ANZTB(Melbourne)/BCS-SIGIST(London)).

www.eurostarconferences.com@esconfs#esconfs

http://www.linkedin.com/in/automation


Q&A


If you have a question, the control panel on the right has a Questions box. All questions will be answered at the end of the presentation.

The GoToWebinar attendee interface is made up of two parts. The Viewer Window shows the presenter’s screen. The Control Panel is where attendees can interact and ask questions.

The webinar will be recorded and you will be notified via email when it is available.

@esconfs#esconfs

How can I get the slides?


Slides and all materials will be posted on the EuroSTAR website as well as emailed to you.

@esconfs#esconfs

Join the conversation on Twitter

#esconfs

@esconfs

@Jonathon_Wright


Continue the conversation with the speaker on Twitter after the show!

@esconfs#esconfs


Jonathon Wright ([email protected])

www.linkedin.com/in/automation

EuroSTAR (Tuesday@ 2:00 PM, 23rd July 2013)


@jonathon_wright#esconfs


The current economic climate is making companies

review their approach to IT even more closely

“We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant

scalability, flexibility, and availability for testing-on-demand. - But how well does this work in practice?”

• Why (Business Benefits & Value)

• What (Cloud Test Automation)

• When (Timeline)

• Agile Portfolio Management (Scope)

• Transition (AMMi)

• Dependencies (Cloud Type)

• Design (First Day Testing)

• Design (Manual’mation)

• Implementation (Agnostic)

• Cloud Platform (IaaS)

• High Volume Automation (HVAT)

• Continuous Build & Delivery

• Actionable Insight (Funnel Virtualisation)

• Dashboards (Real-time)

Background – Test Automation as a Service (TaaaS.net)


“Over the three years we have been developing a azure cloud based solution to provide true

"Test Automaton as a Service" using a hybrid automation approach specifically designed with the Hybrid

Cloud (Private/Public) in mind to utilise the instant scalability, flexibility and availability of the Cloud.”

www.TaaaS.net

http://www.taaas.net/

www.eurostarconferences.com@jonathon_wright#esconfs


“The current implementation is a service-oriented architecture (SOA) based on utilising business process

modelling to create business level keywords written in natural language (domain specific languages)

validated against content sensitive validation to create reusable, jigsaw pieces representing the Solution

Under Test (SUT).”

www.slideshare.net/Jonathon_Wright

http://www.slideshare.net/Jonathon_Wright

platform, technology, client, browser, version, language, test type agnostic

Executing Cloud Tests

feat. VM Dispenser Technology

Cloud Test Environmentsnatural language validated against

content sensitive validation

Creating Cloud Tests


service-oriented architectureBorn in the Cloud


Relevant Clear traceability of the business value of Automation through the visualisation

of the tests via Business Process Modelling (BPMNv2.2 compliant)

Effective Self-validating test assets achieved using natural language with context sensitive

validation against business and testing rules, workflows and data

Maintainable Self-maintaining test asset loader/scraper

Efficient Reports on SUT health including ratings such as percentage availability since

build/release, reported errors over time and traffic to error ratio

Manageable Unified platform which non-domain experts can use a natural language to

represent business processes and user story acceptance criteria

Portable

Technology agonistic - Platform, client, component, browser, version &

language

Test type agnostic – smoke, regression, integration & performance

Reliable Fault tolerance is built in to report and continue on different levels of fuzzy

matching combined with the non-technology specific test definition language

Diagnosable

Actionable defects provided by environment under test (EUT) live pause-

playback supported by dynamic data adapters (DDA) for accelerated defect

investigation and resolution

Source: The Big Picture of Test Automation: Test Trustworthiness – Alan Page, Microsoft (2012)

Summary - Test Automation as a Service (TaaaS.net)



RIP Automation Frameworks

(1993-2013)

This years EuroSTAR topic ‘Questioning Testing’

Q1) Do you really need an office?

(Does your organisation support VPN, video conferencing, online collaboration tools)

Q2) Do you really need an IT department?

(Does your organisation have access communication tools (e.g. email/IM) business

platforms (e.g. SalesForce) or internal applications (e.g. Citrix)

Q3) Do you really need an internal Test/QA department?

(Does your organisation need full-time testing specialists. (e.g. security, penetration,

performance, compliance, usability & automation engineers)

Q4) Do you really need a bespoke Automation Framework?

(What is the core purpose / goals of your organisation? i.e. do they sell food products?

are they a bank? or should they be specialists in creating bespoke test harnesses)


“Here lies beloved architects, engineers and developers who invested a disproportional amount of time and

effort searching for the holy grail of test automation. Careers ended suddenly after a portfolio test manager

asked them to provide the actual business value of what they were doing.”


Q1) How can we move to ‘Test Automation in the Cloud’?


“Existing technologies of virtualisation, business process modelling, cloud based test automation solutions and

rapid and easy internet access allow for the development of approaches that allow companies

to order ‘Testing as a Service’ and pay only for what they use (PAYU).”

Automation Ready

Cloud Power


Where are we going next?@jonathon_wright#esconfs

Example Automation Projects Automation Level Investment in Effort Business Benefit

Description AMMi Gen Implementation Planning Execution Accel. ROI

Client Specific Implementation - Hybrid Approach 4 5 24 63 17 6.9 112%

Client Specific Implementation - BDD Approach 3 4 30 83 22 5.2 85%

Client Specific Implementation - Traditional 1 1 120 250 67 1.7 26%

Testing As A Service Model – Manual’mation 3 1 28 5 25 4.6 200%

Testing As A Service Model - Hybrid Approach 4 5 12 31 19 6.1 186%

TaaaS - Cloud Based Implementation 5 5 5.6 25 7 16.0 306%


Q2) Are we ready for ‘Test Automation in the Cloud’?

Automation Maturity Model index (AMMi®)Assessment to provide a current automation maturity level and the target maturity level:

* Source: AMMi®(http://www.AMMi.org.uk) + Test Automation Patterns (http://TestAutomationPatterns.wikispaces.com)

AMMi Phase Target Criteria

Level 0 Accelerating Automated Test Lifecycle Methodology (ATLM)

Level 1 Traditional Framework Driven (Gen 1-5)

Level 2 Managed Processes are planned, performed, measured, and controlled

Level 3 Sustaining Automation process defined and validated against international

standards (ISO-29119)

Level 4 Quantified Predictability of automation process performance

Level 5 Optimising Automation process variation and statistical predictability


“Sometimes half the battle just starting companies on their automation journey by taking them one step

closer to becoming ready for automation.” AMMi Foundation 2003

Q3) – Which ‘Automation Tool’


“I was recently discussing roles at two of the worlds largest investment banks. Their landscape consisted of over

100+ solutions under test (SUT), representing 500+ applications under test (AUT), across 50+ global locations

with over 400 automation practitioners worldwide. They asked me how to measure the value of automation?”

http://www.jetbrains.com/teamcity/


http://www.jetbrains.com/dotcover/

http://www.jetbrains.com/dotcover/

https://github.com/oneclick/rubyinstaller/wiki/Development-Kit

https://github.com/oneclick/rubyinstaller/wiki/Development-Kit

http://pm.versionone.com/

http://pm.versionone.com/

http://www8.hp.com/us/en/software-solutions/software.html?compURI=1172957

http://www8.hp.com/us/en/software-solutions/software.html?compURI=1172957

http://www.codeproject.com/Articles/289028/White-An-UI-Automation-tool-for-windows-applicatio

http://www.codeproject.com/Articles/289028/White-An-UI-Automation-tool-for-windows-applicatio

http://www.testautomationfx.com/

http://www.testautomationfx.com/

http://rubygems.org/gems/cucumber

http://rubygems.org/gems/cucumber

http://www.greenpeppersoftware.com/confluence/display/GPW/Home/

http://www.greenpeppersoftware.com/confluence/display/GPW/Home/

http://jbehave.org/

http://jbehave.org/

http://www.thoughtworks-studios.com/twist-agile-testing

http://www.thoughtworks-studios.com/twist-agile-testing

http://www.nunit.org/index.php?p=home


http://www.facilita.co.uk/

http://www.facilita.co.uk/

http://www.sikuli.org/download.html

http://www.sikuli.org/download.html

http://www.testplant.com/dlds/eggplant-functional/

http://www.testplant.com/dlds/eggplant-functional/

http://docs.seleniumhq.org/

http://docs.seleniumhq.org/

http://www.concordion.org/Tutorial.html

http://www.concordion.org/Tutorial.html

http://storytestiq.solutionsiq.com/wiki/Getting_Started

http://storytestiq.solutionsiq.com/wiki/Getting_Started

http://www.tricentis.com/en/

http://www.tricentis.com/en/

http://www-01.ibm.com/software/rational/welcome/greenhat/

http://www-01.ibm.com/software/rational/welcome/greenhat/

http://msbuildextensionpack.codeplex.com/

http://msbuildextensionpack.codeplex.com/

http://www.telerik.com/automated-testing-tools/


http://www.microsoft.com/visualstudio/eng/2013-downloads


http://tfs.visualstudio.com/


http://www.perfectomobile.com/


A1

Login.Process

Request.VM

Access.VM

Manage.VM

Logout.Process

A3

A2

C1C3

D2C2D1

C4

E2D4 E1

E3 E4

D3

Business Process

Scenarios (BPS)

Business Process

Data (BPD)

Business Process

Modelling (BPM)

Business Process

Test (BPT)

Quick Overview - ‘First Day Automation’ Approach

@jonathon_wright#esconfs www.eurostarconferences.com

Moment in time

http://www.dorothygraham.co.uk/automationExperiences/index.html

http://www.dorothygraham.co.uk/automationExperiences/index.html

Start Cloud Tester

Modelling Tool (BizAgi or Visio)

Static Source

Dynamic Source

Cloud Source

Direct Source (xPDL or XAML)

Test Asset Loader(Test Asset Cube)

Business Process Modelling

BPT Tool(UFT or Specification)

Business Process Tests

Test Scheduler (WCF or REST)

TaaaS.net(Test Designer)

Q4) How do we become ‘First Day Automation’ Ready?


Agile Portfolio Management – BPM Export (Visio)


Agile Portfolio Management- Business Process Modelling



Agile Portfolio Management – Business Process Modelling


BPS Tool

(BizAgi or ALM/QC)

Direct Source (Workflow or XML)

Business Process Scenarios

Hyper-Test.com(Test Asset Journals)


Solution Lifecycle Management – Business Process Scenarios

HP Quality Centre (10+) / Application Lifecycle Management (11.5)




34 35

36 37

3839

http://www.hyper-test.com/


Feature(s) Work ID

Code

Team Foundation Service

TFS - Gated Check-in!

CancelOK

Build Server

Backlog

VS2013 x64 Cross Tools Command

Microsoft Windows [Version 7.1.7000]C:\Automation\>mstest /testlist:MasterDriver/testmetadata:”TestProject.vsmdi” /testsettings:”MobileDevices.runsettings”

Unit Tests

Feature-Driven Exploratory Test Case(s)

Issue(s) Action(s) Log ScreenCaptureIntelliTrace

Traffic Proxy

Continuous Deployment Test Controller(s) Test Agent(s) Browser(s)

Client(s)Load Agent(s)

Interface(s)

Solution Under Test

Application Under Test

ASP.net | DynoForms (Ajax/Json/Spoc)

Application Server(s)

Database Server(s) Shim/Stub(s)

Shim/Stub(s)

Business Process Scenarios (BPS)Business Process Tests (BPT) Business Process Model (BPM)

Business Process Data (BPD)Business Process Reporting (BPR)

Q5) What is the ‘Manual’mation’ Approach?


Agile Portfolio Management – Agile Feature(s)


Agile Portfolio Management – Product Backlog

Microsoft Project 2013 (Connected to TFS)Team Foundation Service (Product Backlog)


Agile Portfolio Management – Work Items > Tasks

Team Foundation Service TFS Workbench

Digital Analogue


Agile Portfolio Management – Feature-Driven


Agile Portfolio Management – Exploratory Testing



Agile Portfolio Management – Test Case(s)


Application Lifecycle Management – Screen Capture(s)


Application Lifecycle Management – Action(s) Log

Hyper-Test.com

Automation Migration Tool

40

39

41 42 43 44 45

4647



Feature(s) Work ID

Code



CancelOK

Build Server

Backlog



Unit Tests



Traffic Proxy



Interface(s)

Solution Under Test





Shim/Stub(s)











Digital Analogue













Solution Lifecycle Management - Interfaces



Solution Under Test – Traffic Proxy


Application Lifecycle Management – IntelliTrace


Solution Lifecycle Management – Issue(s)

Test Case Migrator Plus (TFS)

@jonathon_wright#esconfswww.eurostarconferences.com

Solution Lifecycle Management – Code


Application Lifecycle Management – Unit Test(s)



Agile Portfolio Management – Business Process Data

ALTOVA XML Spy 2013 (R2 SP1)


Agile Portfolio Management – Reporting - Burndown

TFS Workbench


Monitor velocity & capacity



Agile Portfolio Management – Reporting - Test Execution

Microsoft Test Manager 2012.R3

Q6) How to provide ‘Actionable Insight / Automation Value’


Solution Lifecycle Management – Business Process Tests

“verbs and nouns in the test specific language are written in natural language which using context sensitive validation.”

Login

.Pro

cess

Action Object Event

Enter Username Text

Enter Password Text

Press Login Button

Domain Specific Language

Actio

ns

Obje

cts

Even

ts

Press | Login | Button

Nouns

Verbs

Test Specific Language (TSL)

“The test editor is designed to be low waste, it will only offer nouns and verbs based on discovered artefacts not programming objects.”


Agile Portfolio Management – Business Process Scenarios





40 41 42 43 44 45

464748495051

5253

5455

565758

38

Dashboards (Real-time) providing ‘Actionable Insight’

Theme: Travel@jonathon_wright#esconfs

https://taaas.visualstudio.com/DefaultCollection/Testing.com

https://taaas.visualstudio.com/DefaultCollection/Testing.com

http://www.linkedin.com/pub/ray-arell/3/9a5/27


Automation Ready

Cloud Power


“To the cloud”@jonathon_wright#esconfs


Q7) Should we move to the cloud?

New Challenges & Opportunities ? and/or

A match made in heaven?


New Challenges

already tested?

Community Clouds


“Don’t assume the ecosystems (community or commercial) and

other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)



New Challenges & Opportunities

Consumer Freedom


“Enable sophisticated modelling of the components of cloud deployments,

including servers, storage, databases and data.

For the consumption of cloud resources” - PlanForCloud.com (2012)

CompareTesting.com

Instant scalability

Flexible Test Execution

Flexible Test Assets Environments on-demand

Geo-based execution

Instant Accessibility

99.9% Availability (100% Web)

Instant Portability

Best of Breed (BoB) tools on demand

No upfront investment

Pay As You Use (PAYU)

New Opportunities



PrivateCombination

Public

SUT SUTSUT

Community

Clouds

Public

Clouds

Private

Clouds

TaaaS.net TaaaS.net

“... the prediction that 60% of the average enterprise will have 60% of its applications in the cloud.”

- Karl Flinders, Computer Weekly (2011)

“IBM are working with other cloud suppliers to establish the standards for cloud service provider compliance.”

- Murat Gunenc, IBM Cloud Representative @Rhodes W1 (2011)

Q8) Cloud Type

Automation Ready

Cloud Power


The Final Frontier@jonathon_wright#esconfs

HybridCloud

PrivateCloud

Storage

Networking

Servers

Virtualization

Applications

Infrastructure(as a Service)

Runtimes

Databases

Continuous Integration,

Build & Deployment

PublicCloud

CommunityCloud

Q9) How can we move to ‘Test Automation in the Cloud’?

Lab Management

Test Controller(s)

Test Agent(s)



Test Agent(s) – Telerik Art of Testing Framework

VMM LibraryVM Factory

High Volume

Automated Testing (HVAT)

High Volume Mobile

Automated Testing (HVMAT)


High Volume Mobile Automated Testing (HVMAT)



Solution Lifecycle Management – VMM Library

59

60

61

http://www.incyclesoftware.com/inrelease/

http://www.incyclesoftware.com/inrelease/

Feature(s) Work ID

Code



CancelOK

Build Server

Backlog



Unit Tests



Traffic Proxy



Interface(s)

Solution Under Test





Shim/Stub(s)











Digital Analogue
















Solution Under Test – Traffic Proxy






@jonathon_wright#esconfswww.eurostarconferences.com










TFS Workbench










Login

.Pro

cess

Action Object Event

Enter Username Text

Enter Password Text

Press Login Button


Actio

ns

Obje

cts

Even

ts


Nouns

Verbs









Q10) How can we execute ‘Test Automation in the Cloud’?


Solution Lifecycle Management – Build Server



Solution Lifecycle Management – Continuous Deployment


Solution Lifecycle Management – Test Controller(s)

www.eurostarconferences.com@jonathon_wright#esconfs www.eurostarconferences.com

Solution Lifecycle Management – Test Agent(s)


Load Agent(s) – Visual Studio 2013 – Load Test Manager



Application Lifecycle Management – Client(s)

Real Mobile Devices (PerfectoMobile) Test Studio for iOS (Telerik)


Cross Browser Testing – Telerik (Art of Testing - Framework)

40 41 42 43 44 45

464748495051

5253

5455

565758

38

626364

65 66 67 68


“The best aspects of these proven test approaches

demonstrate how they have evolved over the past

decade, and this echoes some of the changes towards

more lean and agile business methodologies. They are in

a constant state of evolution – just as the underpinning

technology evolves over time.”

My section in the ‘Experiences in Test Automation’ book

(Dorothy Graham & Mark Fewster) 2012.

Final Thoughts - Test Automation in the Cloud


Contact Details – Jonathon-Wright.com

Testing Blog (www.Right-Testing.com)

Automation Blog (www.TaaaS.net)


LinkedIn + Email ([email protected])

www.linkedin.com/in/automation

Testing-Alliance.com

http://www.right-testing.com/








http://www.testing-alliance.com/


Demo - Test Automation in the Cloud

http://youtu.be/YLAfJ64vz0M

http://youtu.be/YLAfJ64vz0M




Agile Portfolio Management – BPM Export (Visio)


34

http://office.microsoft.com/en-gb/visio/visio-professional-2013-free-trial-flowchart-software-FX103791368.aspx

http://office.microsoft.com/en-gb/visio/visio-professional-2013-free-trial-flowchart-software-FX103791368.aspx



HP Quality Centre (10+) / Application Lifecycle Management (11.5)

35

http://www8.hp.com/uk/en/software-solutions/software.html?compURI=1215990

http://www8.hp.com/uk/en/software-solutions/software.html?compURI=1215990

Solution Lifecycle Management - Business Process Scenarios


36

http://msdn.microsoft.com/en-us/library/ms752059.aspx

http://msdn.microsoft.com/en-us/library/ms752059.aspx



Solution Lifecycle Management – Business Process Testing



37

http://www.altova.com/download/xmlspy.html





38

http://www.bizagi.com/

http://www.bizagi.com/

http://www.bpmn.org/

http://www.bpmn.org/

Solution Lifecycle Management – Hyper-Test.com



39



Estimate Effort?

40




41

http://office.microsoft.com/en-gb/project/

http://office.microsoft.com/en-gb/project/




Digital Analogue

42





43




44



45




46

http://msdn.microsoft.com/en-us/magazine/jj618301.aspx




http://www.apple.com/uk/ios/ios7/

http://www.apple.com/uk/ios/ios7/



47





48









49




50







51






52


Solution Lifecycle Management – Traffic Proxy

53

http://www.fiddler2.com/

http://www.fiddler2.com/




54




Logi

n.P

roce

ss

Action Object Event

Enter Username Text

Enter Password Text

Press Login Button


Act

ion

s

Ob

ject

s

Even

ts


Nouns

Verbs



55




56







57





TFS Workbench



58




Solution Lifecycle Management – VMM Library

59



Test Agent(s) – Telerik Art of Testing Framework

60

http://www.telerik.com/products/free-testing-framework/overview.aspx

http://www.telerik.com/products/free-testing-framework/overview.aspx


High Volume Mobile Automated Testing (HVMAT)


61




Application Lifecycle Management – Client(s)

Real Mobile Devices (PerfectoMobile) Test Studio for iOS (Telerik)

62



http://www.telerik.com/automated-testing-tools/ios-testing.aspx

http://www.telerik.com/automated-testing-tools/ios-testing.aspx

Load Agent(s) – Visual Studio 2013 – Load Test Manager


63




Solution Lifecycle Management – Build Server


64

http://windows.microsoft.com/en-GB/windows-8/preview

http://windows.microsoft.com/en-GB/windows-8/preview


Solution Lifecycle Management – Continuous Deployment


65

https://www.atlassian.com/software/confluence/overview/team-collaboration-software

https://www.atlassian.com/software/confluence/overview/team-collaboration-software

http://jenkins-ci.org/

http://jenkins-ci.org/



Solution Lifecycle Management – Test Controller(s)


66


Solution Lifecycle Management – Test Agent(s)


67


Cross Browser Testing – Telerik (Art of Testing - Framework)

68

http://youtu.be/hPTk-B8Hd1I

http://youtu.be/hPTk-B8Hd1I



Performance Testing As A Service – PTaaS.net


69

www.PTaaS.net

http://www.ptaas.net/


Questions?

@esconfs#esconfs

EuroSTAR Webinar Archive

Access almost 70 on-demand software testing webinars by the world's leading testing experts. Topics include….


Agile, Automation, Test Management, Process Improvements, Risk-based Testing, Performance Testing, Test Documentation, Cloud Testing, Session-based Testing, Weekend Testers

Visit the archive!

http://www.eurostarconferences.com/community/member/webinar-archive

@esconfs#esconfs

The EuroSTAR Community.Are you a member?

Did you know we have an online resource area with almost …


600 software testing presentations, 70 topical webinars, over 40 videos and podcasts and 25 expert eBooks

Do you know that membership is completely free and you can access these premium

testing resources anytime?

Join today! http://www.eurostarconferences.com/community/

member

@esconfs#esconfs


Are you coming to Gothenburg?

Join us in the Swedish Exhibition & Congress Centre, Gothenburg,

Sweden from the 04-07 November for the 21st annual EuroSTAR

Conference.

Visit www.eurostarconferences.comfor more information on what you

can expect to see at Europe’s largest software testing event!

@esconfs#esconfs

Book before Sept

http://www.eurostarconferences.com/

Have you seen the latest competition?

You and 3 of your team could be joining us at the 2013 EuroSTAR Conference!Entering is easy! Just send us a short video (2 minutes approx.) demonstrating how your team works together to meet

goals or solve problems.

Deadline for entries is Wednesday, August 14th! More information here: http://www.eurostarconferences.com/content/teamstar-2013@esconfs

#esconfs

http://www.eurostarconferences.com/content/teamstar-2013

Don’t miss these great savings, book your ticket today!

Plus Make even bigger savings when you register for the Group discount where every 5th attendee goes free!

For more information on the Conference Programme and pricing options go to www.eurostarconferences.com@esconfs

#esconfs

eurostar - test automation in the cloud - webinar

Technology

cloud test automation

cloud jonathon wright

test sut

business value of automation

azure cloud

cloud computing

true test automaton

uk jonathon wright