automation and performance testing presentation
TRANSCRIPT
Automation & Performance Testing
1. Automation Overview
2. Project assessment
3. Approach to building automated tests
4. Identifying tasks for automation
5. How should time be costed?
6. Building a tool kit
7. Testing environments
8. Training, Development, Resourcing
9. AXA Wealth Basingstoke & Bristol, identifying Automation needs
10.What Exists
11.Percentage of Coverage
12.Road Map
13.Suggested Priority List
14.Questions
Agenda
• What is test automation?
• Record and Playback, easy right?
• Who should build Automated tests?
• Full time or Part time.
• Time, expense & resource.
Automation Overview
• Test approach covers a section on automation and performance.
• Our technical testers are consulted and a decision is made on the benefits of automation.
• Documented proof we have considered these options – great for audit!
Project assessment for automation & Performance
• Project lead approach.
o Project output should be a set of automated testso Working more closely with development teams:o Buddying automated testers with developers during projects.
Big advantage with Visual Studio to deliver A/T’s as a project output and to smooth the C# learning curve.
• Building automated regression tests to replace manual
o Being held accountable! o Stakeholders should be identified. o Benchmark targets set.o Weekly demos to stakeholders.o Stakeholder knowledge is captured.o No surprises on delivery.
Approach to building automated tests
• Project lead approach.
• Building automated regression tests to replace manual• Focus on high resource drain areas relieving frequent resource effort
and high risk areas, as identified by the test managers.
Identifying tasks for automation.What should drive the priority?
• Complex Functionality
• Frees up experience test resource to analyse
20%
• High Resource• Repetitive Tasks• Common
functionality
80%
• Project Level
• BAU
How should time be costed? breaking down barriers
• Functional Testing • VS2010 Coded UI (Preferred option)
• HP Quick test Pro (Embassy, Siebel)
• Non Functional Testing• VS2010 Performance & Load tests (Preferred option)
• HP Performance Center (Embassy, Siebel)
• Other• Link Checker
• Data generator tools
Building a tool kit
• A known state of test data that can easily be redeployed before tests are re-run.
• Example: doing a switch, that policy would always be available and not used by another tester or disinvested in LIVE and therefore no longer available.
• Embassy data migration work may be reusable to snapshot database.
• Performance Environments
• You know and understand everything that happens in the enviroment.
• Can corrupt databases rapidly.• Cut down LIVE environment, ring-fenced LAN, Benchmarked times and
performance.
• Visual Studio 2010 Lab Management can spin up virtual environments
• Sponsor?
Testing Environments (Automation & Performance)
• Structured professional training in all tool sets.
• Roll out automated tests to other project members, testers, dev’s.
• Mentor and support other test analysts.
• Evaluate recruitment requirement permanent / contractor balance.
• Are the contractors allocated to the new OD available?
• Microsoft time to realise our goal of making VS2010 our preferred tool set.
Training, Development, Resourcing
• What tech test have we got on the Elevate Platform (Automation & Performance)
• What is the strategy for Bristol?
• AXA Wealth website, Performance on Elevate, more….
AXA Wealth Basingstoke & Bristol. Identifying
Automation needs.
AreaTests Status
OLS Pension Quotes 380 WorkingOLS USP Quotes 30 WorkingOLS Bond Quotes 120 WorkingEmbassy Investment Bonds 18 WorkingEmbassy Direct / Debit (a/c sort validation)
33 Working
Total 581
What exists (Basingstoke)
Area Tests StatusOCC Tests 13 Effort required due to data changeSiebel 30 Effort required due to data changeTotal 13
Area Tests StatusEmbassy Add Policy 141 Nearly workingEmbassy USP 23 Nearly workingEmbassy Disinvestments 25 Nearly workingTotal 189
Area Tests StatusChameleon 171 Not Working (Winrunner)Embassy Segmented Bonds 22 Not Working (project was abandoned)Total 193
Grand Total 976
• Difficult to translate:
• Manual regression is written in business scenarios, automation deals with functionality areas.
• Automation covers more checks such as, drop down lists, images text on screen, although dilutes exploratory testing to a degree.
• Current Successes
Percentage of coverage
Project Name
Release Date
Test function Tools
Aon Hewitt's May Performance Test Visual Studio 2010
July Release July Automated Regression
Run QTP Regression Pack
September September Automated Regression
Run QTP Regression Pack
AWWIPS Nov AutomationPerformanceBrowser checkerLink Checker
VSCodedUI / QTP Visual Studio 2010BrowserCam / BrowseraLink Checker
Bond (Embassy Light)
Nov – Jan Automation QTP Automation / VS2010
Belfast December Functional Manual Fast Forward - VS2010
FNZ On-going Automation QTP Automation
Embassy Regression
On-going Automation QTP Automation
This years high-level road map
• AXA Wealth Website - 2 perm (training) - length of project, automation & Performance.
• FNZ Regression - 2 Contractors, 1 Perm (overseeing 25%) – 6 Months.
• Embassy Automation regression - 2 Contractors, 1 perm (overseeing 25%) - 6 months.
• Elevate Automation - Understand what's there
• Elevate Performance – We understand there are LIVE issues that need addressing
• Bond (Embassy Light) – Needs to be accessed
• Future projects (Automation & Performance) - To be accessed
Quick priority list
Questions?