automation healthcheck report

8/11/2019 Automation HealthCheck Report

1/26

SQS Group Limited

VodafoneAutomation HealthCheckReport

Darryl Kennedy6 th February 2013


2/26

Executive Summary

Background

Vodafone have an existing test automation solution created using QC 10 and QTP 11 The automated regression testing solution includes both ECC (GUI) and SRM (Portal) The existing solution has been in existence since its creation in December 2011 The existing solution is maintained in Pune by VISPL Automation Team

Test execution and reporting in carried out in Pune by VISPL Automation Team

Purpose of the Automation HealthCheck

In order to maximise the benefits of automation, SQS have been asked to undertake anindependent review (HealthCheck) of the current test automation solution to ensure that

Industry best practice has been employed

The solution is extensible and provides the necessary portability and robustness

The test assets created are re useable and easily maintainable

Tests execute unattended, results and evidence are clear and easily interpretable

SQS have been asked to provide any recommendations for improvement along withsupporting benefits of implementing any recommendations


3/26

Work Carried out

SQS provided Vodafone with a test automation specialist who Reviewed a cross section of the QC and QTP assets available

Assessed the approach to automation and the automation framework deployed

Assessed the versatility, robustness and extensibility of the automation framework

Assessed the approach to script development including test data Assessed error / exception triggers and recovery

Assessed automation synchronisation and execution performance

Assessed ability to execute unattended and the existing infrastructure

High Level HealthCheck Finding

The QC and QTP solution adheres to certain design principles necessary for amaintainable component driven framework but lacks necessary robustness

The existing solution performs reasonably but essential attributes are missing or poorlyimplemented which are necessary for enterprise strength automation

Important strategic changes must be implemented if the true benefits of test automation areto be realised before significantly extending the scale of the existing solution

Executive Summary


4/26

Automation Solution Summary A good technical basis for an automated testing solution incorporating some featuresprovided by the HP Tool set. Consideration has been made to ensure the framework isextensible and maintainable with minimal effort.Some fundamental characteristics of test automation have been introduced such as sharedobject repositories, data abstraction and the use of external function libraries to provide amodular design.

Web and SAP automation present a number of challenges and although innovation hasbeen applied in the framework to provide solutions above what QC and QTP provide out ofthe box some basic principles have not been adhered to:

Modular design of the solution and use of external data and function library files

Custom HTML reporting easily interpretable by non technical users

XML Configuration files for multiple environment execution & Country specific test dataLacks an overall framework engine with the appropriate flow controlSynchronisation and error handling needs a lot of work so able to run unattended

No validation included into the functions which make up the Test Scenarios

The VISPL team were knowledgeable of their subject and extremely collaborative.

Executive Summary


5/26

Recommendation Priority & Implementation ComplexityThe table below summarises all the recommendations made in this report into the priorityfor implementation.

The table below summarises all the recommendations made in this report into thecomplexity of the implementation.

Executive Summary

Implementation Priority Total Recommendations

High 36

Medium 13

Low 1

Implementation Complexity Total Recommendations

High 12

Medium 10

Low 28


6/26


7/26

The HealthCheck focussed upon

Adherence to test automation best practice

Ease and skill requirements of execution and maintenance

Future proofing

The HealthCheck did not

Test existing automation (any issues identified during the HealthCheck will be reported)

Review the automated scripts for test coverage, accuracy to reflect business processesor ability to reflect business testing needs

Review the scope or prioritisation of test scenarios being automated

Estimate the cost to run and maintain the scripts

Basis for the HealthCheck

The findings are based on virtual meetings that took place on 4 th and 5 th February

Thank you to the VISPL Automation Team for the help and support they provided

Raghavendar Bandaru, Ganesh Kaushik and Nagesh Battula

Automation HealthCheck Scope


8/26


9/26

FrameworkComponent Criteria Rating Comments

Audit Trail

NamingConventions

AmberNaming conventions exist for Test Scenarios and Functions whichare adhered to. Additional detail should be applied to make it easierfor end users to interpret these easier i.e. SAP T Code.

UniqueIdentifiers Amber

Test Scenarios and functions are distinctly named but do not containa unique identifier.

Mapping toRequirements

AmberNo requirements exist in QC. A matrix of requirements ismaintained by the functional test team using Excel. This makestraceability time consuming.

Mapping toManual Tests Red

Manual and technical assets are duplicated in QC so traceability isdifficult and time consuming. Extra actions must be performed bythe end user to see the manual steps. Manual and technical assetsmay get out of synchronisation with each other.

Coverage Requirements Red No requirements exist in QC so there is no technical solutionavailable to ascertain coverage of the technical assets.

Test Automation Score Card


10/26

Day 1

Initiation meeting with Tom Sewell to agree scope of the HeathCheck Review (SQS & VISPL) of the QC and QTP framework implementation Review (SQS & VISPL) of the Scenarios, Objects, Reporting and Error Handling First day wrap up meeting with Tom Sewell

Day 2

Review (SQS & VISPL) of the Data, Config. File, Reporting, Execution & Development Execution (SQS) of Portal and SAP Test Scenario Initial Report Creation Walkthrough of the findings report structure with Tom Sewell

Day 3

Report Finalised Assessment of the automation Recommendations and examples of findings Issues identified during the HealthCheck

Walkthrough of findings (Vodafone & SQS)

Schedule of Activities


11/26

In order to conduct the HealthCheck, SQS requested access to the following

Access to all test assets

Access to the system under test and existing manual test scripts Access to working end-to-end automation test scripts

Access to a sample of custom built drivers for the various automation tools

Access to sample reports generated

A demonstration of the test execution:

Against the applications under review

Using manual run

Via build triggered run if applicable

Availability of the VISPL Automation Team

Environment support during demonstration and test execution of the solution Reasonable turn around times on queries raised whether against the VISPL

Automation Team or external (for example a manual test or solution design query).

All of the above was provided as requested by the VISPL Automation Team.

Pre-requisites and Support Criteria


12/26


13/26

Recommendations

Recommendation Guidelines

Each of the recommendations made has a unique identifier

Each of the recommendations is clearly defined

Each of the recommendations is supported by a benefit

Each of the recommendations has an implementation priority assigned

Each of the recommendations has an implementation complexity assigned

Ordering of Recommendations in Table format

The unique identifier does not reflect an implementation order or importance

Many of the recommendations are linked and must be considered as a whole


14/26

Recommendations for Quality Center

ID Recommendation Benefit Priority Complexity

QC01 Implement the usage of the Detail fieldin QC for each of the assets storedProvides the end user with a clearoverview of the purpose of the test High Low

QC02 Implement the usage of the Status fieldin QC for each of the assets stored

Creates a development workflow andimproves visibility of the overallprogress

High Low

QC03Implement the usage of the DesignSteps in QC for each of the assetsstored

Ensures the manual tests assets andthe technical test assets do not vergeapart from each other. Provides thetechnical team a clear and easy way ofworking

High Low

QC04 Store the requirements in QC whichare currently tracked using Excel

Provides dynamic traceability ofprogress and enables tracking ofissues found

High Low

QC05

Move the Test Resources outside ofQC to enable the implementation of anenterprise strength version controlsolution (unless the upgraded QCversion control provides the necessaryfunctionality)

QC provides a basic level of versioncontrol but not sufficient fordevelopment of automated test suiteson an enterprise level with manypeople working concurrently

Medium High


15/26

Recommendations for QTP Scripts


SC01 Naming Convention with uniqueidentifiers applied to all assetsSimplifies the traceability of all assetsand improves the reporting in QC High Low

SC02 Detailed headers and script commentsto be implemented to all assetsImproves the ease of use of thetechnical assets for collaboration Medium Low

SC03Code repeated in each test whichshould be a framework function i.e. EnvVariables

Simplifies and speeds up maintenance

activities required on all testsMedium Low

SC04

All tests must be suitable forenvironment portability. Load XMLenvironment variables from a centralfile dynamically

Supersedes XML loaded manually toevery test which is a slow andcumbersome process prone to error

High Low

SC05 Start up and teardown functions to beapplied to all assets. Allows for quick changes to be madeglobally to all test assets High Low

SC06Break away from functions and createQC assets for each businesscomponent

Will introduce overall framework andflow control and ability to execute testsfrom a failure point

Medium High

SC07

Use the QTP object to dynamically loadrun-time setting at the beginning ofeach test. Screen prints Never/Alwaysand global object timeouts

Applies consistency to every test caserunning in the suite and removes theneed to execute VBS manually

High Low


16/26

Recommendations for QTP Objects


OR01 Remove version numbers from currentOR in useRemoves the requirement to manuallyassociate this with every test using it High Low

OR02 Changes the object description, dont just use the SAP generated nameMakes scripts readable, intuitive andallows other users to understand them High Low

OR03 Object Windows must be distinct,merge _2 and _3 as appropriateKeeps OR tidy and reduces thenumber of windows to maintain High Medium

OR04 Objects must be distinct, merge _2 and _3 as appropriate

Keeps OR tidy and reduces thenumber of objects to maintain

High Medium

OR05Object property and value pairs mustbe improved and regular expressionsapplied where appropriate

Clearer object definition and reducesfails during execution Medium Medium

OR06 ORs for SAP and Web should bebroken up into smaller filesReduces the change of corruption andallows concurrent development Medium Low

OR07 Portal OR cleaned up to remove manyinstances of Browser objectKeeps OR tidy and reduces thenumber of objects to maintain High Medium

OR08 Remove all local objects once mergedinto the shared ORPrevents tests failing on shared andlocal conflicts High Low

OR09 Store shared OR in a central locationand not on a C:\

Easily accessible by all developers High Low


17/26

Recommendations for QTP Functions


FL01

Large single files over 12,000 lines ofcode must be broken down into smallerlibraries of technical functions and QCassets for business components

Collaborative development isimpossible and contention exists.Using QC assets will improve thetraceability and maintainability

High High

FL02 Remove the version numbers from thenames of function librariesReduces the manual overhead ofcontinually associating to every test High Low

FL03Library files and individual functionsmust have detailed headers whichinclude modification history

Provides a level of self documentingsolution and improves collaboration High Low

FL04Naming conventions must be improvedand include SAP TCode whereappropriate

Improved script readability and ease ofdebugging when errors occur High Medium

FL05 A Version Control solution must beadopted, taking code out of QC ifnecessary

Improves concurrent development,

traceability, multiple environmentexecution and reversion wherenecessary

High High

FL06Create traceability with the manualassets stored in QC by creatingtechnical assets in the same QC entity

Improves the overall knowledge of theteam. Keeps manual and technicalassets in sync and improvestraceability

High High


18/26



FL07Dynamic synchronisation must beapplied to all business components andall hard coded waits removed

Currently there is limited or none. Thestandard QTP object synchronisationpoints are not robust enough forunattended execution and porting thescripts across different testenvironments

High Medium

FL08 Login functions must handle multipleOPCO logins for all authorisation levelsThese should be stored in an externalfile Medium Low

FL09

An overall flow control must beimplemented into every businesscomponent. Currently QTP simplyexecutes linear with no checks toensure events are happening asexpected to allow control to pass back

to the calling test for error reporting andhandling

This will prevent tests from failingunexpectedly and provide a muchgreater visibility of what caused thefailure in the QTP execution report

High High

FL10Evaluate all reporter message toensure failing steps are reportedcorrectly.

Found an example of code which failsbut report message contained a pass.

An If / Else statement was completelybypassed as SAP was on the incorrectscreen

High Low


19/26



FL11

Remove the dependency on the QTPRecovery Manager. This should beused as a catch all not an everyinstance solution. These should behandled in the code itself

Much greater control of the errorhandling and provides far more detailin the QTP execution report as to thecause of the failure. Can then alsoincorporate screen captures

High High

FL12 Validation and checkpoints must beapplied to all business components

This provides detailed feedback as to

why tests failed and allows tests to runin batches unattended without theneed for constant monitoring

High Medium

FL13Error detection, error reporting, errorhandling and recovery must be appliedto all business components.

QTP Run settings updated with a VBSfail to suggest Test Stops on error, thisis only part of the necessary errorhandling. Tests must also resetconditions i.e. close browser and then

continue to the next test cleanly

High High

FL14 An overarching function is missing tohandle and recover from failure

Inbuilt QTP setting used to capturescreenshot on failure. This is typicallyunreliable and should only beincorporated in the code where failureshave been anticipated to occur

High Medium


20/26



FL15Improve the robustness of the businesscomponents by adding additionalvalidation to the objects before use.

Creating overload functions in QTPcould include this and many otherchecks i.e. Enabled, Visible etc. Asthey are overloaded no change to theend user so scripts can still berecorded in QTP.

High High

FL16Function to create HTML reportrequires lots of additional code in thefunctions and should be included in theoverloaded functions.

This will simplify the script creation andensure consistency throughout overbusiness component

Medium High

FL17 HTML creation functions must alsoupload to QC RUN table. Reduce manual overheads High High

FL18HTML functions for automatedreporting must be included inoverarching function to handle errors

Ensures these are created anduploaded during test runs whichencounter errors

High Low


21/26

Recommendations for Test Data


TD01OPCO User Accounts are stored on acentral file governed by environmentand country

Easy for the end user to maintain andprovides versatility for multipleenvironment execution

Medium Low

TD02

Different tabs for OPCOs should beamalgamated to single sheet and QTPfunction created to select appropriate

row

Reduces overall maintenance as thereare less Excel sheets to manage Medium Medium

TD03 Needs to have a unique identifier foreach Excel sheet which map to the test

Improves visibility and traceability High Low

TD04 Additional iterations of data should beincorporated into the Excel data sheetsto test more data permutations

Incorporating additional data setsprovides greater coverage easily withapplying a unique identifier to the datarow and mapping to test name.Current constraint in the frameworkdoes not allow this

Medium Low


22/26

Recommendations for Test Execution


TE01

6 to 10 dedicated QTP servers hostedvirtually so accessible from anywhere

load balanced using QC host groupsinstead of the 4 machines currently inuse

Utilises more of the 20 QTP licensesavailable and provides greaterflexibility from the 4 machines usedcurrently which are also the day to daymachines of the team. Tests can thenbe developed and executedconcurrently

Removes the dependency that theuser must be present to monitor theexecution

Removed the dependency of theDeveloper running their own script

High Low

TE02 Hardcoded C:\ path in the HTMLreports must be removed

This must be addressed to allow manymachines to run unattended and seeresults from anywhere

High Low

TE03

HTML report is uploaded dynamicallyto QC in the same way manual testsdo, these should be uploaded to theRUN table / Execution history

Ensures results can be accessed fromanywhere and not just the machinethey were executed on. Provides clearand concise audit trail

High Medium


23/26

Recommendations for General Adoption


GA01 QTP Patching level should be broughtup to the latest versionIssues already identified by HP will beaddressed Low Low

GA02Development work items must bevisible to everyone so its clear who isworking on what changes

Greater visibility of the changesplanned and progress tracked againsteach. Allows work to be decided on apriority basis

Medium Low

GA03 Solution documentation must becreated for the framework

Allows up skilling of anyone whoneeds to develop, maintain or executetests

Medium Low

GA04Solid Version Control solution requiredwhich may include moving outside ofQC

Unable to develop and execute testsconcurrently so different versions ofQTP code base must be maintained.QTP Environments in General Optionsalong with dynamic library loading inQTP 11 allow the developer to quicklychange environments and versions

High High


24/26

Implementation Plan


25/26

Implementation Plan To be followed as a pilot phase

Implementation Area Detailed Description

Build a Core Framework

1) Object Identification Mechanism to be streamlined

2) Object Repository to be re-structured

3) Custom functions to be built to replace QTP inbuilt methods

4) Incorporate Error handling, Object Validation, Screen Capture for evidence, Object syncand Checkpoints in the scripts

5) Reduce the large Function Library by moving the "Application Specific" functions tomodular QTP scripts

Test Plan

1) Convert the Application Specific functions along with the new framework into individual,modular, robust QTP scripts

2) Include the design step for each T-code - for better readability and review by Non-Technical/ Business Users

Test Lab

1) Update the scenario details, description, Objective, Pre-requisite

2) Build the end-to-end scenario by calling the modular QTP scripts into Test Lab andsetting the execution flow

3) Incorporate custom reporting to generate a report at the end of each scenario execution

4) Datasheet consolidation

Scheduling/ Host Management

1) Get the physical assets in place (20 Concurrent user license for QTP available)

2) Prepare a scheduling strategy to run the scenarios overnight where possible to bettermanage the execution timelines


26/26

SQS Group Limited

Darryl KennedyPrincipal ConsultantEmail : [email protected] Tel: 07950 128 060

SQS Group Limited

82 King Street,Manchester, M2 4WQ

Internet: www.sqs-uk.com Internet: www.sqs-group.com
mailto:[email protected]://www.sqs-uk.com/http://www.sqs-group.com/http://www.sqs-group.com/http://www.sqs-group.com/http://www.sqs-group.com/http://www.sqs-uk.com/http://www.sqs-uk.com/http://www.sqs-uk.com/mailto:[email protected]

automation healthcheck report

Documents