test automation frameworks final
Post on 09-May-2015
1.909 Views
Preview:
DESCRIPTION
TRANSCRIPT
Advanced Test AutomationAdvanced Test AutomationPractical Application of Test Automation Practical Application of Test Automation
FrameworksFrameworks
Lee BarnesFounder & Chief Technology Officer
Utopia Solutions
Copyright 2007 Utopia SolutionsPage 2
AgendaAgenda
• Assumptions & Definitions• Introduction to Automation Frameworks• Overview of Common Frameworks• Selecting the Right Framework• Case Study• Q & A
Copyright 2007 Utopia SolutionsPage 3
AssumptionsAssumptions
• Audience has:– Exposure to automation best practices– Exposure to contemporary test automation tool
(e.g. WinRunner, QuickTest Pro, SilkTest, QA Run, Robot)
• Addressing technical issues such as custom interface objects not in scope
• Presentation is tool agnostic
Copyright 2007 Utopia SolutionsPage 4
DefinitionsDefinitions
• Data-driven• Business Process• Keyword• Test case• Test scenario (script)• Test engine
Introduction to Automation Introduction to Automation FrameworksFrameworks
Copyright 2007 Utopia SolutionsPage 6
What is an “Automation What is an “Automation Framework” ?Framework” ?
• Structured approach to automation utilizing data-driven techniques, reusable test code assets and standards/guidelines
Automation Standards & Guidelines
Test Scenario
Input
Test Scenario Results
Test EngineReusableScripts/Modules
UtilityFunctions
Copyright 2007 Utopia SolutionsPage 7
Why?Why?
• Promote the three key characteristics automated test suites must possess to be successful:– Reliable– Maintainable– Scalable
Copyright 2007 Utopia SolutionsPage 8
How?How?
• A properly designed framework will possess these characteristics if they incorporate three key attributes:– Modular data-driven design– Separation of test definition and test
execution– Error detection and recovery
Copyright 2007 Utopia SolutionsPage 9
Modular & Data-Driven Modular & Data-Driven DesignDesign
• Promotes maintainability through reuse and reduced test suite size
• Promotes scalability by abstracting test data from test code
Copyright 2007 Utopia SolutionsPage 10
Separation of Test Definition Separation of Test Definition and Executionand Execution
• Promotes scalability by allowing test creation to be performed by non-technical testers and SME’s without test code modification
• Allows testers/business analysts and automation engineers to focus on their strengths – separately
• Enables execution of single test scenario across multiple platforms
Copyright 2007 Utopia SolutionsPage 11
Separation of Test Definition Separation of Test Definition and Test Executionand Test Execution
Test Engine
Test Scenario
InputTest
Scenario Results
Software Testing and Functional Subject Matter Experts
Automation Experts
ReusableScripts/Modules
UtilityFunctions
Test DefinitionInterface
Copyright 2007 Utopia SolutionsPage 12
Error Detection and Error Detection and RecoveryRecovery
• Promotes reliability by enabling unattended execution
• Best Practices– Use tool capabilities for the simple stuff
• Trapping unexpected pop-ups• Handling/logging unexpected return values from
built-in tool functions (e.g. object recognition errors)
– Keep it simple and consistent– Carefully plan execution status into framework
Copyright 2007 Utopia SolutionsPage 13
Handling Execution Handling Execution StatusStatus
• Need to track progress and handle recovery after restart• Assess possible statuses - PASS, FAIL, INCOMPLETE, etc.• Build into test suite modules & input files…
Business Process Module
Initialize status to INCOMPLETE…Perform business process…If validation condition = TRUE
Set status = PASSElse
Set status = FAILLog execution status
Test Scenario Input File
Common Automation Common Automation FrameworksFrameworks
Business Process FrameworkKeyword Framework
Integrated Framework
Business Process (BP) Business Process (BP) FrameworkFramework
Copyright 2007 Utopia SolutionsPage 16
Business Process Business Process FrameworkFramework
• Main components of test engine parallel AUT functionality at a business process level
• Sample Components:
• Login• PickOrder• AdjustInventory• VerifyInventory• …
BPFunctions
• LogResults• LogError• OpenTestInput• ReadTestInput• …
SupportingFunctions
• Design/Coding Standards
• Test Definition Guide
• Maintenance & • Execution Guide
Standards &Guidelines
Copyright 2007 Utopia SolutionsPage 17
Test Engine
Business Process Business Process FrameworkFramework
Test Scenario
Files
Login, test_user_01, password_01
Create_Order, <ord1>,SKU10045,100,…
Create_Order, <ord2>,SKU10045,100,…
Ship_Order,<ord1>,…
Ship_Order,<ord2>,…
Verify_Inventory,SKU10045,…
Test Scenario Results
Business Process Scripts
Utility Functions
Copyright 2007 Utopia SolutionsPage 18
Business Process Business Process FrameworkFramework
• Relatively simple to implement
• Straight forward test definition interface for test creators
• Efficient execution
Advantages Disadvantages
• Requires automation expertise if business process steps change
• Requires automation expertise to support new business processes
• Not suitable for other types of testing
Copyright 2007 Utopia SolutionsPage 19
Business Process Business Process FrameworkFramework
• Ideal for testing that involves the consistent execution of a finite number of end-to-end business processes
Keyword (KW) FrameworkKeyword (KW) Framework
Copyright 2007 Utopia SolutionsPage 21
Keyword FrameworkKeyword Framework
• Main components of test engine parallel object/action level functionality required to perform test actions
• Sample Components:
• FieldSetText• FieldGetText• ListSelectItem• ButtonClick• …
Object/ActionFunctions
• LogResults• LogError• OpenTestInput• ReadTestInput• …
SupportingFunctions
• Design/Coding Standards
• Test Definition Guide
• Maintenance & • Execution Guide
Standards &Guidelines
Copyright 2007 Utopia SolutionsPage 22
Test Engine
Keyword-Driven ModelKeyword-Driven Model
Object/ Action
Functions
Utility Functions
VerifyState, login_page, EXISTS
SetText, user_id, testuser01
SetTextSecure, password, j4ghjs39
ClickButton, submit
VerifyState, main_menu, EXISTS
…
Test Scenario Results
Test Scenario
Files
Copyright 2007 Utopia SolutionsPage 23
Keyword FrameworkKeyword Framework
• Suitable for many types of testing
• Automation expertise not required to maintain or add business processes
Advantages Disadvantages
• More complex test definition interface
• Can result in large test scenario input files
• Can result in inefficient execution
Copyright 2007 Utopia SolutionsPage 24
Keyword FrameworkKeyword Framework
• Ideal framework for:– Verbatim conversion of manual test scripts– Detailed field-level testing– Teams with technical test creators– Serving as base for integrated frameworks (discussed next)
Integrated Business Process / Integrated Business Process / Keyword FrameworkKeyword Framework
Copyright 2007 Utopia SolutionsPage 26
Integrated FrameworkIntegrated Framework
• Business Process framework wrapped around a keyword framework
• Contains all Keyword framework components plus business process templates to allow test scenarios to be defined at the BP level
Copyright 2007 Utopia SolutionsPage 27
Integrated BP/KW Integrated BP/KW FrameworkFramework
Business Process
Templates
Test Scenario
File
VerifyState, login_page, EXISTS
SetText, user_id, <user_id>
SetTextSecure, password, <pw>
ClickButton, submit
Test Scenario Results
Test Engine
Object/ Action
Functions
Utility Functions
Login, test_user_01, password_01
Create_Order, <ord1>,SKU10045,100,…
Create_Order, <ord2>,SKU10045,100,…
…
Copyright 2007 Utopia SolutionsPage 28
Integrated FrameworkIntegrated Framework
s
• Incorporates the benefits of both BP and KW frameworks
Advantages Disadvantages
• Can be complex to implement and maintain
• Can muddy the functional/technical boundary
Copyright 2007 Utopia SolutionsPage 29
Integrated FrameworkIntegrated Framework
• Ideal for testing that requires both business process and keyword frameworks
Selecting the Right FrameworkSelecting the Right Framework
Copyright 2007 Utopia SolutionsPage 31
Aligning Automation Aligning Automation Framework with Test Framework with Test
ObjectivesObjectives
• Testing objectives come in different sizes and shapes
• Choose the framework that best aligns with your needs:– Test coverage– Type of test cases (GUI, BP, data validation, etc.)– Quantity of test cases
• “Distill” your test cases (discussed later)
Copyright 2007 Utopia SolutionsPage 32
Understanding Test Understanding Test CoverageCoverage
Test Script
• Login as user test01
• Select Create PO link and verify user has access
• Verify Create PO page is defaulted with the following data …
• Verify SKU field rejects non-numeric data
• Create PO using the following data …
• Verify …
Requirements
• Functional• Order Mgmt.• Inventory Mgmt.• Customer Mgmt.• …
• Security• User access• …
• User Interface• Screen
navigation• Input validation• …
Copyright 2007 Utopia SolutionsPage 33
What’s MissingWhat’s Missing
Test Script
• Login as user test01
• Select Create PO link and verify user has access
• Verify Create PO page is defaulted with the following data …
• Verify SKU field rejects non-numeric data
• Create PO using the following data …
• Verify …
Requirements
• Functional• Order Mgmt.• Inventory Mgmt.• Customer Mgmt.• …
• Security• User access• …
• User Interface• Screen
navigation• Input validation• …
?
Copyright 2007 Utopia SolutionsPage 34
Distilling Test CasesDistilling Test Cases
Test Cases
• Create PO• Create PO under
credit limit• Create PO over
credit limit• …
• Receive Order• Full receipt• Partial receipt• …
• Input Field Validation
• Valid values• Max length• …
Test Script
• Login as user test01
• Select Create PO link and verify user has access
• Verify Create PO page is defaulted with the following data …
• Verify SKU field rejects non-numeric data
• Create PO using the following data …
• Verify …
Requirements
• Functional• Order Mgmt.• Inventory Mgmt.• Customer Mgmt.• …
• Security• User access• …
• User Interface• Screen
navigation• Input validation• …
Copyright 2007 Utopia SolutionsPage 35
Why Distill?Why Distill?
• Login as user test01
• Select Create PO link and verify user has access
• Verify Create PO page is defaulted with the following data …
• Verify SKU field rejects non-numeric data
• Create PO using the following data …
• Verify Customer credit limit...
Manual Testing
User Interface
Input Validation
Business Process
User Interface
User InterfaceUser Interface
User Interface
User Interface
Input Validation
Input Validation
Input Validation
Input Validation
Input Validation
Business ProcessBusiness Process
Business ProcessBusiness Process
Business Process
Copyright 2007 Utopia SolutionsPage 36
Distilled Test Cases MappedDistilled Test Cases MappedTo FrameworksTo FrameworksAutomated Testing
Automated Test Suite
Business ProcessTesting Framework
User InterfaceTesting Framework
Input ValidationTesting Framework
Business Process
User Interface
Input Validation
Case StudyCase Study
Multi-Platform Warehouse Management System
Copyright 2007 Utopia SolutionsPage 38
SituationSituation
• Warehouse Management System– Web-based client used by office staff– Subset of functionality available to warehouse staff via
hand-held RF device– One-to-one mapping of screens & fields between platforms
• Existing manual regression test suite– Focused on BP-level testing– 35% of test scenarios executed on both platforms
• Objectives– Create automated test suite to include existing test
coverage– Single input source for multi-platform test scenarios
Copyright 2007 Utopia SolutionsPage 39
Multi-Platform FrameworkMulti-Platform FrameworkAutomation OptionsAutomation Options
1. BP framework with parallel modules for each business process requiring execution on both platforms
2. KW framework with parallel functions for each object/action requiring support on both platforms
3. KW framework with “intelligent” functions that “sense” the execution platform and execute appropriately
4. Integrated (BP wrapped around KW) frameworkWhat criteria would you use to decide?
Copyright 2007 Utopia SolutionsPage 40
Multi-Platform FrameworkMulti-Platform FrameworkConceptual DesignConceptual Design
Test Scenario
Input
Tes
t D
efin
itio
nIn
terf
ace
Test Engine
Test Scenario Results
Desktop ExecutionFramework
Hand-held ExecutionFramework
FunctionalSME’s
AutomationEngineers
Copyright 2007 Utopia SolutionsPage 41
Best Framework Choice?Best Framework Choice?
• Integrated framework to incorporate the advantages of both BP and KW frameworks
• Low-level “intelligent” object/action functions best support multi-platform execution
Copyright 2007 Utopia SolutionsPage 42
Integrated BP/KW Integrated BP/KW FrameworkFramework
Business Process
Templates
Test Scenario
File
VerifyState, login_page, EXISTS
SetText, user_id, <user_id>
SetTextSecure, password, <pw>
ClickButton, submit
Test Scenario Results
Test Engine
Object/ Action
Functions
Utility Functions
Login, test_user_01, password_01
Create_Order, <ord1>,SKU10045,100,…
Create_Order, <ord2>,SKU10045,100,…
…
Copyright 2007 Utopia SolutionsPage 43
Translate Manual Scenario Translate Manual Scenario to Input Fileto Input File
• Log into warehouse A3 as testuser01
• Set inventory levels for SKU1001/Bin 47 and SKU1010/Bin 49 to 150
• Pick order ORD0034
• Pick order ORD0046
• Verify inventory level for SKU1001/Bin 47 is 140
• Verify inventory level for SKU1010/Bin 49 is 135
Test Scenario Test Scenario Input File
Login, test01, test01, warehouseA3
Adjust_Inventory, SKU1001, BIN047, 150
Adjust_Inventory, SKU1010, BIN049, 150
Pick_Order, ORD0034
Pick_Order, ORD0046
Verify_Inventory, SKU1001, BIN047, 140
Verify_Inventory, SKU1010, BIN049, 135
Copyright 2007 Utopia SolutionsPage 44
Translate BP’s to Keyword Translate BP’s to Keyword StepsSteps
Test Scenario Input File
Business Process Template
Copyright 2007 Utopia SolutionsPage 45
Test Driver Test Driver
Get BP
Instruction
Login, test01, test01, warehouse3
Load BP
Template
Map BP
Data to BP
Steps
Execute BP
Steps
VerifyState, Login Page, EXISTS
SetText, User ID, <user_id>
SetText, Password, <pw>
SetText, Warehouse, <warehouse>
ClickButton, Submit
VerifyState, Login Page, EXISTS
SetText, User ID, test01
SetText, Password, test01
SetText, Warehouse, warehouse3
ClickButton, Submit
Copyright 2007 Utopia SolutionsPage 46
Required Keyword Functions Required Keyword Functions ??
• All object/action pairs required to execute multi-platform test scenarios
• Additional hybrid keywords for common step sequences (e.g. login, navigation, etc.)
• Review existing manual test scenarios and get input from your customers – test suite users
Copyright 2007 Utopia SolutionsPage 47
Multi-PlatformMulti-PlatformKeyword FunctionsKeyword Functions
Function Set_Text, Parameters Object, Value
Get execution_platform
Select Case execution_platform
Case BROWSER
status = Set web Object = Value
Case RF
status = Set RF Object = Value
Get debug level
If debug level = TRACE
log execution detail
Return status
Multi-Platform Code
Mange multi-platform differences at the tool level if possible
Copyright 2007 Utopia SolutionsPage 48
Supporting FunctionsSupporting Functions
• Reading input files• Results logging, progress tracking, etc.• Test suite metrics (pass/fail, component
usage, etc.)• Error handling• Custom object interaction• Custom synchronization
Copyright 2007 Utopia SolutionsPage 49
Supporting FunctionsSupporting FunctionsResults LoggingResults Logging
• Start with the end in mind– Data– Level of detail– Format– Medium
Copyright 2007 Utopia SolutionsPage 50
Sample Summary Result Sample Summary Result FileFile
ACCUMULATED TOTALS PERCENTAGE------------------------------------------STEPS EXECUTED: 14TEST CASES EXECUTED: 5TEST CASES PASSED: 3 60.0TEST CASES FAILED: 1 20.0TEST CASES INCOMPLETE: 1 20.0SCRIPT/APPLICATION ERRORS: 3
TEST CASE RESULT
---------------------------------------------
TC0001.........................PASS
TC0002.........................PASS
TC0003.........................PASS
TC0004.........................INCOMPLETE
TC0005.........................FAIL
Summarized Test Case Results
Accumulated Execution Metrics
Copyright 2007 Utopia SolutionsPage 51
Sample Detail Log FileSample Detail Log File
BEGIN STEP: BP0001
START TIME: 14:46:48
S0001 Verify_State.............OK
S0002 Set_Text.................OK
S0003 Set_Text.................OK
S0004 Set_Text.................OK
S0005 Click_Button.............OK
BP0001.........................OK
END STEP: BP0001
END TIME: 14:48:04
BEGIN STEP: BP0002
START TIME: 14:48:06
S0001 TSL error handler invoked
S0001 Err: -10011, function: set_window
S0001 Verify_State............FAIL
One or more script errors were detected
BP0002........................FAIL
END STEP: BP0002
END TIME: 14:50:21Page 1 Page 2
Copyright 2007 Utopia SolutionsPage 52
Results Logging/Metrics Results Logging/Metrics FunctionsFunctions
Test
Driver
Keyword
Functions
CreateLogFiles
StepBegin
StepEnd
Writes step start time to log files
Writes step status, end time to log files, input file, metric totals & other mediums
Creates log files
LogMetrics Write accumulated test execution metrics to log files
Writes detailed step execution status to log files
StepDetail
Logs component usageLogUsage
Copyright 2007 Utopia SolutionsPage 53
…and Answers
Please direct any future questions to: lee.barnes@utopiasolutions.com
Questions…Questions…
top related