Hybrid Keyword Data Driven Hybrid Keyword Data Driven

Automation FrameworksAutomation Frameworks

Test Automation in the CloudTest Automation in the Cloud

Real world case studyReal world case study

Jonathon Lee WrightJonathon Lee Wright

jlwright@iee.orgjlwright@iee.org

www.automation.org.ukwww.automation.org.uk

www.linkedin.comwww.linkedin.com/in/automation /in/automation

11th November 201011th November 2010

• Complete “Automation Lifecycle Management ” (ALM) solution;

• Providing “Automated Testing as a service ” in the Cloud ;

• “Pay As You Go ” flexible billing (no expensive tool licensing costs);

• Distributed continuous build integration with “Application Under Test ”;

• Shorter market deployment times through quicker Automated Testing ;

• Shorter environment deployment times through Automated Health Check ;

• Better quality testing freeing specialist test resource to concentrate on non-repetitive test tasks;

• Easy interface for non-technical testers to create their own “user stories ”;

• Capability Maturity Model Integration (CMMI) through Business Process Modelling (BPM) comprising of Business Level Keywords (BLK);

• Introduction of “Full ” Agile automation testing and “First ” day automation.

What value did this case study deliver?

What approach?

• Manual Approach;

• Traditional Automated Approach:

• Hybrid Keyword Data Driven Framework Approach.

Vendor Tool(Record & Playback,

Adapt Scripts)Limited requirements Limited output

Framework Approach

Hybridutilising the best technologies and resourcesHybrid

Keyword

Data

Driven (Modularity)

Keywordbusiness level keywords represent user stories

Datadynamic use of sanitised business data

Driven (Modularity)modularity of test procedures to provide reliable processing of the framework components

Hybrid Approach

.NET 4.0 Development (Visual Studio 2010)

Cloud (Windows Azure/SQL/AppFabric)

Automation (Ranorex/Coded UI APIs))●●●●●

Hybrid

Vendor Tool (Ranorex IDE)●○○○○Traditional

ToolsModularityDataKeywordBPTBPMApproach

○ - Limited Support

● - Full Support

Single Technology Stack (SPOF)

Tool Vendor Dependency

C# Project incompatible with

Visual Studio

Technology Blocks

(Custom Flex Objects/Classes)

Limited non-technical tester

interface

Vendor Support

(maintenance subscription)

Object Repository

(object-based Record / Replay)

ConstraintsAdvantages

Traditional (Vendor Tool)

In-house framework development

& maintenance effort.

Additional skills / knowledge

resources

Azure Platform Subscription

Object Repository

(limited Record / Replay support)

Multiple Technology Stack

(VS2010 / C# / VB.NET / ASP.NET / TFS

Windows Azure / (WCF) Web Services /

DLL/ XML/ Azure SQL/AppFabric

Coded UI / Ranorex / AIR / Flex / XPATH )

Integration with development

(Visual Studio 2010)

Integration with deployment

(TeamCity v5.1)

Integration with testing

(Quality Center v11)

ConstraintsAdvantages

Hybrid (.NET Development)

Return on Investment

1 2 3 4 5 6 7

Number of Excecutions

Inve

stm

ent i

n E

ffort

MarketConfigurationTesting + HealthCheck

MarketConfigurationTesting

Traditional

Manual

MCT = 3 MCT+ HC = 3.4 Traditional = 3.7

Initial Investment

LeverageHybrid (Framework)

Vendor Tool

Rn = Aa / Am = (Va / n + n*Da) / (Vm / n + n*Dm)

Automation Goals

Reusablemodularity of

test procedures & functions

Manageableeffective test design,

execution and traceability

Maintainableto reduce the amount

of test maintenance effort

Robustproviding

object/event/errorhanding & recovery

Flexibleindependent of

test environment and be scalable

Measurablecustomisable reporting

& test execution progress

Business Process Testing (Process)

Login.Process (Data Stage)

Test Management Tool Team Foundation Server

User Stories

Business Level Keywords

Business Process Modelling (BPM)

Business Process Testing (BPT)

Business Level Data

Modularity

Keyword

Data

Driven

Hybrid

Login.Process (Keyword Stage)

Action

Set

Object

Username

Data

#Username

Parameters

Data

#Username

Value

Shop4Test

Object Repository

Object

Username

Class

EditBox

Framework

Action(s) Objects(s) Data

Object

Handler

Data

Handler

Event

Handler

Business Process Modelling (Example)

LoginFrmLoginChangePassword

FrmA0 A1 A2

MyShopHome

Frm

B0

EditProfileFrmB3

ShopHome

ShopLogin

CreateNewReport

CreateNewReport

Frm

C0

ReportType

FrmC1

ProductGroups

FrmC2

RunReport

FrmC7

TimePeriods

FrmC3

ProfileFrmC4

C8

C9

Geographies

FrmC5 C10

ReportSche

duleFrmC12

TermsConditions

Frm

D0

B1

B2

ShopperPan

elsFrmC6 C11

CurrentReports

Frm

D0

CurrentReports

CompletedReports

Frm

R0

CompletedReports

Flex Reports Excel

F0 E0

MyGroupFrm

RecurringReports

Frm

S0

RecurringReports

MyGroups

G0

ModifyGroupFrmG1

OrderDetailsFrmR1

Index- Complete- In Complete- Supported - Out of Scope

Business Level Keywords (Testing)

Translates

Automation Lifecycle Management (Process)

Technology Stack

Framework Architecture (Cloud Testing)

Framework Architecture (Deployment)

Shared Persistent Message Queue Server

Customer Facing Web Services Per Customer Database Resources

Customer Deployment

Per Customer On demand VM Test Lab EnvironmentRule Store

Schedule

History

Real time progress

Test runner

Dashboard

Control and Reporting Dashboard Scheduler

Scheduler

User Story Editor/Workbench

Results

Technology Adapter - Ranorex / Coded UI /

Watir / Selenium etc...

Generic Data Services Logging

Results Recorder

Cloud Resources

Billing

Credentials

Asset Loader Build Integration Quality Centre or other BPM Integration

Asset Store

Results Recorder

Results Recorder

Scheduler

System Management - Cloud/Application Tools

Configuration

Monitoring

Test Criteria - User Data, Load, Geo., Technology etc...

User Data Store

Test Adapter - Accessability / Load / Regression

/ Penetration / Browser/ Platform / Smoke

SEO etc...

Framework Architecture (High Level)

Automation Lifecycle Management (ALM)

Requirements - Business Process Data

AUT Database

Quality Center

Test Automation Database

Mock Live Data (Sanitised)

Team Foundation Server

Test Planning - MasterDriver (Excel)

Object Repository

Test Repository

Test Planning - XML Schema

Test Execution - Testing in the Cloud

WebServices (WCF)

<< Realtim

e Test Results >>

<< Realtim

e Error Reporting >>

<< Check Test Schedule >>

Test Execution - Automation Scheduler

Selectable list of user case stories (Test Sets)

Available machines

Global Stop Buttons (Stops all tests on all running machines)

Global Pause button

Global Play button

Web Services (WCF)

XML

Communication - Message Definition (WCF)

Windows Service (Lifecycle)

Event / Object Handling

Supported events:

� button.click

� text.parmsetcheck

� text.validate

� link.click

� reporttype.selecttreeitem

� reporttype.opentreenode

� list.parmsetcheck

� checkbox.select

� tree.validate

� tree.parmsetcheck

� radio.parmsetcheck

� checkbox.parmsetcheck

� checkbox.validate

Business Level Keywords (Process)

Business Level Keywords represent a number of logical test steps that make up a business process i.e. “Login.Process”

User Stories are represented by a number of Business Level Keywords:

Each User Story represents client Market Configuration Testing mappings:

LoginFrm.Logo.Image

LoginFrm.UserName.Label

LoginFrm.Password.Label

LoginFrm.LoginBoxHeader.Label

LoginFrm.LoginButton.Label

LoginFrm.ChangePWButton.Label

LoginFrm.ForgotPWLink.Label

LoginFrm.ServiceDeskTel.Label

LoginFrm.ServiceDeskMail.Label

LoginFrm.Copyright.Label

ABCDE

FGHIJ

F

G

H I J

A

B

C

D

E

Market Configuration Testing (Process)

Process the first test step configuration data value located in the “Test Repository” i.e. #dh.MyServices_Login_UserNameLabel

Using the data value to confirm that the “Market Requirement” exists for the target market (<Client> + <WorkStream>)

Once confirmed the configuration value can be extracted from the “Market Specification” by selecting the target market (<Language>)

Test Step 1 – Using the “Configuration” value

to verify that market configuration is correct:

UserNameLabel = “User Name:”Result = FAIL

Object Recognition (Smart Identification)

Get the test step object name from the Test Repository (MasterDriver) i.e. LoginFrm.Password.Input

Using the object name extract the object class & associated value from the Object Repository (MasterParameters) i.e. Html.EditBox & “password”

Using the object class extract the primary & secondary class tag identifiers from the Class Definition (MasterParameters) i.e. “InnerText” & “Type”

Attempt #1 – Using the “WebElement” Class

Describe the Object Physical Description:

WebElement Object= [@InnerText = “password”]

Attempt #2 – Using the “InputTag” Class

Describe the Object Physical Description:

InputTag Object= [@Type = “password”]Physical Object

Physical Description

Object Recognition (Fuzzy Logic)

FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']";FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']";

IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>();IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>();

Get all the screen objects on the flex page.

Retrieve all the Buttons on the page.

For ( int I = 1; >= flexbuttons.Count; i++)

{

// get the xpath for each element.

RxPath me = flexButtons.ElementAt<Ranorex.Button>(i).GetPath();

// convert the xpath to a string

string[] str = me.ToString().split(‘/’);

// convert xpath to a split of strings

string[] str1 = xPath.Split(‘/’);

// compare both string paths until a common tree route is found (this will be your button)

CompareTreePaths( str, str1 )

}

When you have identified a tree path, you can then use it in the way described in the MasterDriver file (e.g. flexButton.Click).

Unified Test Reporting (UTR)

Unified Test Reporting – Real-time Log

TestRunnerInput (XML) Output (WebService)

.NET Object WCF

Error/Event Handling

Logging (Debug)

Results (XML)

Process Improvement

� Agile Planning – Team Foundation Server linked to Task Board (SfTS v2);

� Coding Standards – Code reviews (internal/external to team), coding tools, unit test standards, agile hints, logging etc etc;

� Testing - Internal peer reviews, unit tests developed, exploratory testing for each story as a task, Integration test story in each sprint;

� Wiki – Knowledge capture including online framework documentation (screenshot)

� Technical Notes /Debt - on project wiki (intranet SharePoint).

Key Achievements

• Centralised - management of test assets to provide open connectivity as a generic source of test scenario data;

• Dynamic - generation of sanitised input test data validated against business rules along with an object repository based on a combination of descriptive programming containing XPATH regular expressions;

• Legacy - support for static data types plus full/partial fall back support for manual test execution;

• Unified reporting - using funnel virtualisation techniques to analyse the quality of builds, assisted by diagnostic data adapters;

• Generic reusable - test components, modules and libraries providedreliable processing of the entire test automation effort.

SSUMMARYUMMARY

•• HybridHybrid –– utilising the best technologies and resources to do the job;utilising the best technologies and resources to do the job;

•• Keyword Keyword –– creating simple and robust test scenarios written in business creating simple and robust test scenarios written in business

level keywords combined with natural language;level keywords combined with natural language;

•• Data Data –– effective use of dynamic business data to provide an input sourceffective use of dynamic business data to provide an input sourcee

•• Driven Driven –– processes the test component actions, objects and events processes the test component actions, objects and events

seamlessly;seamlessly;

•• Automation Automation –– that is collaborative, distributed and scalable;that is collaborative, distributed and scalable;

•• Frameworks Frameworks –– independent of application or environment under test;independent of application or environment under test;