Download - Hybrid Automation for the Hybrid Cloud
Hybrid Keyword Data Driven Hybrid Keyword Data Driven
Automation FrameworksAutomation Frameworks
Test Automation in the CloudTest Automation in the Cloud
Real world case studyReal world case study
Jonathon Lee WrightJonathon Lee Wright
[email protected]@iee.org
www.automation.org.ukwww.automation.org.uk
www.linkedin.comwww.linkedin.com/in/automation /in/automation
11th November 201011th November 2010
• Complete “Automation Lifecycle Management ” (ALM) solution;
• Providing “Automated Testing as a service ” in the Cloud ;
• “Pay As You Go ” flexible billing (no expensive tool licensing costs);
• Distributed continuous build integration with “Application Under Test ”;
• Shorter market deployment times through quicker Automated Testing ;
• Shorter environment deployment times through Automated Health Check ;
• Better quality testing freeing specialist test resource to concentrate on non-repetitive test tasks;
• Easy interface for non-technical testers to create their own “user stories ”;
• Capability Maturity Model Integration (CMMI) through Business Process Modelling (BPM) comprising of Business Level Keywords (BLK);
• Introduction of “Full ” Agile automation testing and “First ” day automation.
What value did this case study deliver?
What approach?
• Manual Approach;
• Traditional Automated Approach:
• Hybrid Keyword Data Driven Framework Approach.
Vendor Tool(Record & Playback,
Adapt Scripts)Limited requirements Limited output
Framework Approach
Hybridutilising the best technologies and resourcesHybrid
Keyword
Data
Driven (Modularity)
Keywordbusiness level keywords represent user stories
Datadynamic use of sanitised business data
Driven (Modularity)modularity of test procedures to provide reliable processing of the framework components
Hybrid Approach
.NET 4.0 Development (Visual Studio 2010)
Cloud (Windows Azure/SQL/AppFabric)
Automation (Ranorex/Coded UI APIs))●●●●●
Hybrid
Vendor Tool (Ranorex IDE)●○○○○Traditional
ToolsModularityDataKeywordBPTBPMApproach
○ - Limited Support
● - Full Support
Single Technology Stack (SPOF)
Tool Vendor Dependency
C# Project incompatible with
Visual Studio
Technology Blocks
(Custom Flex Objects/Classes)
Limited non-technical tester
interface
Vendor Support
(maintenance subscription)
Object Repository
(object-based Record / Replay)
ConstraintsAdvantages
Traditional (Vendor Tool)
In-house framework development
& maintenance effort.
Additional skills / knowledge
resources
Azure Platform Subscription
Object Repository
(limited Record / Replay support)
Multiple Technology Stack
(VS2010 / C# / VB.NET / ASP.NET / TFS
Windows Azure / (WCF) Web Services /
DLL/ XML/ Azure SQL/AppFabric
Coded UI / Ranorex / AIR / Flex / XPATH )
Integration with development
(Visual Studio 2010)
Integration with deployment
(TeamCity v5.1)
Integration with testing
(Quality Center v11)
ConstraintsAdvantages
Hybrid (.NET Development)
Return on Investment
1 2 3 4 5 6 7
Number of Excecutions
Inve
stm
ent i
n E
ffort
MarketConfigurationTesting + HealthCheck
MarketConfigurationTesting
Traditional
Manual
MCT = 3 MCT+ HC = 3.4 Traditional = 3.7
Initial Investment
LeverageHybrid (Framework)
Vendor Tool
Rn = Aa / Am = (Va / n + n*Da) / (Vm / n + n*Dm)
Automation Goals
Reusablemodularity of
test procedures & functions
Manageableeffective test design,
execution and traceability
Maintainableto reduce the amount
of test maintenance effort
Robustproviding
object/event/errorhanding & recovery
Flexibleindependent of
test environment and be scalable
Measurablecustomisable reporting
& test execution progress
Business Process Testing (Process)
Login.Process (Data Stage)
Test Management Tool Team Foundation Server
User Stories
Business Level Keywords
Business Process Modelling (BPM)
Business Process Testing (BPT)
Business Level Data
Modularity
Keyword
Data
Driven
Hybrid
Login.Process (Keyword Stage)
Action
Set
Object
Username
Data
#Username
Parameters
Data
#Username
Value
Shop4Test
Object Repository
Object
Username
Class
EditBox
Framework
Action(s) Objects(s) Data
Object
Handler
Data
Handler
Event
Handler
Business Process Modelling (Example)
LoginFrmLoginChangePassword
FrmA0 A1 A2
MyShopHome
Frm
B0
EditProfileFrmB3
ShopHome
ShopLogin
CreateNewReport
CreateNewReport
Frm
C0
ReportType
FrmC1
ProductGroups
FrmC2
RunReport
FrmC7
TimePeriods
FrmC3
ProfileFrmC4
C8
C9
Geographies
FrmC5 C10
ReportSche
duleFrmC12
TermsConditions
Frm
D0
B1
B2
ShopperPan
elsFrmC6 C11
CurrentReports
Frm
D0
CurrentReports
CompletedReports
Frm
R0
CompletedReports
Flex Reports Excel
F0 E0
MyGroupFrm
RecurringReports
Frm
S0
RecurringReports
MyGroups
G0
ModifyGroupFrmG1
OrderDetailsFrmR1
Index- Complete- In Complete- Supported - Out of Scope
Framework Architecture (Deployment)
Shared Persistent Message Queue Server
Customer Facing Web Services Per Customer Database Resources
Customer Deployment
Per Customer On demand VM Test Lab EnvironmentRule Store
Schedule
History
Real time progress
Test runner
Dashboard
Control and Reporting Dashboard Scheduler
Scheduler
User Story Editor/Workbench
Results
Technology Adapter - Ranorex / Coded UI /
Watir / Selenium etc...
Generic Data Services Logging
Results Recorder
Cloud Resources
Billing
Credentials
Asset Loader Build Integration Quality Centre or other BPM Integration
Asset Store
Results Recorder
Results Recorder
Scheduler
System Management - Cloud/Application Tools
Configuration
Monitoring
Test Criteria - User Data, Load, Geo., Technology etc...
User Data Store
Test Adapter - Accessability / Load / Regression
/ Penetration / Browser/ Platform / Smoke
SEO etc...
Requirements - Business Process Data
AUT Database
Quality Center
Test Automation Database
Mock Live Data (Sanitised)
Team Foundation Server
Test Execution - Testing in the Cloud
WebServices (WCF)
<< Realtim
e Test Results >>
<< Realtim
e Error Reporting >>
<< Check Test Schedule >>
Test Execution - Automation Scheduler
Selectable list of user case stories (Test Sets)
Available machines
Global Stop Buttons (Stops all tests on all running machines)
Global Pause button
Global Play button
Event / Object Handling
Supported events:
� button.click
� text.parmsetcheck
� text.validate
� link.click
� reporttype.selecttreeitem
� reporttype.opentreenode
� list.parmsetcheck
� checkbox.select
� tree.validate
� tree.parmsetcheck
� radio.parmsetcheck
� checkbox.parmsetcheck
� checkbox.validate
Business Level Keywords (Process)
Business Level Keywords represent a number of logical test steps that make up a business process i.e. “Login.Process”
User Stories are represented by a number of Business Level Keywords:
Each User Story represents client Market Configuration Testing mappings:
LoginFrm.Logo.Image
LoginFrm.UserName.Label
LoginFrm.Password.Label
LoginFrm.LoginBoxHeader.Label
LoginFrm.LoginButton.Label
LoginFrm.ChangePWButton.Label
LoginFrm.ForgotPWLink.Label
LoginFrm.ServiceDeskTel.Label
LoginFrm.ServiceDeskMail.Label
LoginFrm.Copyright.Label
ABCDE
FGHIJ
F
G
H I J
A
B
C
D
E
Market Configuration Testing (Process)
Process the first test step configuration data value located in the “Test Repository” i.e. #dh.MyServices_Login_UserNameLabel
Using the data value to confirm that the “Market Requirement” exists for the target market (<Client> + <WorkStream>)
Once confirmed the configuration value can be extracted from the “Market Specification” by selecting the target market (<Language>)
Test Step 1 – Using the “Configuration” value
to verify that market configuration is correct:
UserNameLabel = “User Name:”Result = FAIL
Object Recognition (Smart Identification)
Get the test step object name from the Test Repository (MasterDriver) i.e. LoginFrm.Password.Input
Using the object name extract the object class & associated value from the Object Repository (MasterParameters) i.e. Html.EditBox & “password”
Using the object class extract the primary & secondary class tag identifiers from the Class Definition (MasterParameters) i.e. “InnerText” & “Type”
Attempt #1 – Using the “WebElement” Class
Describe the Object Physical Description:
WebElement Object= [@InnerText = “password”]
Attempt #2 – Using the “InputTag” Class
Describe the Object Physical Description:
InputTag Object= [@Type = “password”]Physical Object
Physical Description
Object Recognition (Fuzzy Logic)
FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']";FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']";
IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>();IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>();
Get all the screen objects on the flex page.
Retrieve all the Buttons on the page.
For ( int I = 1; >= flexbuttons.Count; i++)
{
// get the xpath for each element.
RxPath me = flexButtons.ElementAt<Ranorex.Button>(i).GetPath();
// convert the xpath to a string
string[] str = me.ToString().split(‘/’);
// convert xpath to a split of strings
string[] str1 = xPath.Split(‘/’);
// compare both string paths until a common tree route is found (this will be your button)
CompareTreePaths( str, str1 )
}
When you have identified a tree path, you can then use it in the way described in the MasterDriver file (e.g. flexButton.Click).
Unified Test Reporting – Real-time Log
TestRunnerInput (XML) Output (WebService)
.NET Object WCF
Error/Event Handling
Logging (Debug)
Results (XML)
Process Improvement
� Agile Planning – Team Foundation Server linked to Task Board (SfTS v2);
� Coding Standards – Code reviews (internal/external to team), coding tools, unit test standards, agile hints, logging etc etc;
� Testing - Internal peer reviews, unit tests developed, exploratory testing for each story as a task, Integration test story in each sprint;
� Wiki – Knowledge capture including online framework documentation (screenshot)
� Technical Notes /Debt - on project wiki (intranet SharePoint).
Key Achievements
• Centralised - management of test assets to provide open connectivity as a generic source of test scenario data;
• Dynamic - generation of sanitised input test data validated against business rules along with an object repository based on a combination of descriptive programming containing XPATH regular expressions;
• Legacy - support for static data types plus full/partial fall back support for manual test execution;
• Unified reporting - using funnel virtualisation techniques to analyse the quality of builds, assisted by diagnostic data adapters;
• Generic reusable - test components, modules and libraries providedreliable processing of the entire test automation effort.
SSUMMARYUMMARY
•• HybridHybrid –– utilising the best technologies and resources to do the job;utilising the best technologies and resources to do the job;
•• Keyword Keyword –– creating simple and robust test scenarios written in business creating simple and robust test scenarios written in business
level keywords combined with natural language;level keywords combined with natural language;
•• Data Data –– effective use of dynamic business data to provide an input sourceffective use of dynamic business data to provide an input sourcee
•• Driven Driven –– processes the test component actions, objects and events processes the test component actions, objects and events
seamlessly;seamlessly;
•• Automation Automation –– that is collaborative, distributed and scalable;that is collaborative, distributed and scalable;
•• Frameworks Frameworks –– independent of application or environment under test;independent of application or environment under test;