Strategies for a Successful E2E Systems Integration Test
Fiona Charles Let’s Test
May 9, 2012
Strategies for a Successful E2E SIT 2
This session
Describes key project management strategies I have used to manage large-‐scale Systems Integration Tests Offers practical tips for implementing those strategies on your next SIT
The focus is primarily on large-‐scale integrations, but the strategies and tips can be scaled for any size of test.
Strategies for a Successful E2E SIT 3
1 – Project management 2 – Applications 3 – People 4 – Test readiness 5 – Information & communication 6 – Environments & data 7 – Questions & discussion
Strategies for a Successful E2E SIT 4
A Typical Systems Integration Project
Merchandising systems
Point of Sale registers & Controllers
Supply Chain systems
Management Information
systems
Accounting systems New POS
New Loyalty systems
Strategies for a Successful E2E SIT 5
The Test Manager’s Job
Develop a test strategy that addresses the major risks within the time available
Build the team you need for success
Build the right “controlled data” tests and plan
Manage testing and evaluation
Ensure acceptance by stakeholders
Sounds just like any other test -‐ right???
Strategies for a Successful E2E SIT 6
The test management challenge is at least as great as the testing challenge
A project in itself
Always highly visible, and often controversial
Requirements for success often misunderstood
Your job is to hold to the big picture
E2E Systems Integration Testing requires the skills of an experienced test project manager
Strategies for a Successful E2E SIT 7
1 – Project management 2 – Applications 3 – People 4 – Test readiness 5 – Information & communication 6 – Environments & data 7 – Questions & discussion
Strategies for a Successful E2E SIT 8
It’s a Test Project Management job
Much of the complexity in managing a Systems Integration Test comes from the sheer number of things involved:
Applications Platforms Environments Data Application readiness dates Business processes or functions People
Managing these is a logistical, rather than test, challenge.
Strategies for a Successful E2E SIT 9
The numbers can seem overwhelming!
Strategies for a Successful E2E SIT 10
But it’s a nightmare only if you don’t manage it Divide and conquer
Have a strategy for each type of thing, and a project plan that says when you are going to deal with it
Keep your priorities current
Don’t confuse the urgent with the merely important
Delegate where you can
Use project management tools and techniques
Strategies for a Successful E2E SIT 11
Be sure to level-‐set in your own head
And then in everyone else’s head
Large Vehicles Need More Room Large Vehicles
Need More Room
Strategies for a Successful E2E SIT 12
Make sure you are positioned correctly
Senior role, working on equal footing with: Project managers Architects
No barriers to information Pulse of the program When each project is delivering (really) Risks, issues, delays, individual test strategies, etc.
Authority SIT entry criteria Requirements to make SIT work
Reporting to the overall Project/Program Manager
Strategies for a Successful E2E SIT 13
1 – Project management 2 – Applications 3 – People 4 – Test readiness 5 – Information & communication 6 – Environments & data 7 – Questions & discussion
Strategies for a Successful E2E SIT 14
The applications will determine test scope
Start with the Architecture/integration diagrams Use whatever you can find If they don’t exist, draw your own Validate what you use
Keep an open mind about the test boundaries “Received belief” may be wrong Focus on “whole system” outcomes Identify all incoming data that could influence your results Ask what the end point should really be
What’s the last application or report the critical data impacts? Who needs to sign off?
You may need to include unchanged systems
Strategies for a Successful E2E SIT 15
Journal
System Context Diagram
Teller Application
Perform CSR Transactions: Retrieve Client ProfilePerform Client SearchWithdraw Funds,Receive Cash,Deposit Funds (PDA), Negotiate Cheque/Money Order, Transfer between accounts,Purchase Drafts/ Money Orders,Get a VISA cash advance,Make VISA payment,Reverse VISA PaymentReverse VISA Cash Advance,Purchase Foreign ExchangePurchase Traveler Cheques or gift cheques,Make Bill Payment, Authenticate Client,Deposit Funds (Current Account)Perform Close Day and Extended hour ProcessesBatch Off Switch WorkstationQuery Client SessionUpdate Bank book
Teller (Authenticated)
Check SECAF Entitlements
Get FeeGet Currency List
Get Denominations(XML Files from Web
Server)
VISA Cash Advance , Make VISA Payment,VISA Cash Advance Reversals,VISA Payment Reversal
Insert Client Session Detail,Query Client/Transaction Session,Retrieve end of day data by branch
Credit,Debit,DDA Inquiry,Bank book updatesAcct Details,Acct History
Authenticate Client (PIN Verification),Perform CIS AuthorizationPerform CIS Inquiry(2,4,5,6 or 7),Retrieve Client Information, CIS Limits and Account List and Balances (Uses CIS 11 as primary path if the user has a client card)
Negotiate/Accept System priced Deals
Post GL Entries(batch interface)
Retrieve Client ProfileSearch ClientRetrieve Account ListAccount Details
APP
APP
APP
APP
APP
APPAPP
Existing CTP Objects for: Retrieve Client ProfilePerform Client Search
Retrieve Account List and Balances (if no client card
or for a full account list)Retrieve & Update Offer &
Opportunities
APP
Current Account - Cash and merchant envelope transactions.(batch interface)
APP
APP
APP
APP
Get Traveler Cheque and Currency Txn
Details(batch interface)
Extract for EDW
APP
Retrieve Transit List
General Ledger
Strategies for a Successful E2E SIT 16
1 – Project management 2 – Applications 3 – People 4 – Test readiness 5 – Information & communication 6 – Environments & data 7 – Questions & discussion
Strategies for a Successful E2E SIT 17
You will be dealing with LOTS of people
Strategies for a Successful E2E SIT 18
The rest of the project management team Project or Program Manager Architects Project Control
People associated with each application, e.g., Project leads Project test leads and testers Programmers/Architects
Management People outside the project who will want information
QA, auditors, business sponsors, acceptors
Your own test team
Strategies for a Successful E2E SIT 19
Make sure you have a strategy for each group
The rest of the project management team
• Build relationships -‐ lunch • Status reports, regular status meetings
People associated with each application
• Test working group or forum – get input & buy-‐in for strategy • SIT progress broadcasts
Management
• 1 on 1’s • Occasional status/strategy presentations
QA, auditors, business sponsors, acceptors
• Approach them – know who they are and what they need
Strategies for a Successful E2E SIT 20
Success depends on building relationships
Be aware of cultural differences Different technology groups may have very different processes Different groups may have varied risk tolerances: e.g., front-‐line business vs. finance or audit Distributed teams in different countries
Understand what each group wants/needs from your tests
Work to get early buy-‐in for everything significant in your plan Develop your test strategy in a lightweight medium, and walk it around (“socialize” it) If you have to publish a detailed document, try to get consensus before you write it
Strategies for a Successful E2E SIT 21
Staff your own team appropriately
Strategies for a Successful E2E SIT 22
You won’t have time to be hands-‐on!
Strong second-‐in-‐command Day-‐to-‐day SIT test team management
Environment and data manager Work with all participating teams
Small team of seasoned testers Design tests; prepare test materials
Toolsmith(s) Any automation you can do
Strategies for a Successful E2E SIT 23
1 – Project management 2 – Applications 3 – People 4 – Test readiness 5 – Information & communication 6 – Environments & data 7 – Questions & discussion
Strategies for a Successful E2E SIT 24
Establish the ground rules (Entry & exit criteria)
Environments ready, connected and tested
Coordinated data
Project-‐level tests complete, e.g., System and usability test Performance Near-‐neighbor interfaces
All must-‐fix bugs, and all integration bugs of any severity, resolved
Teams assigned (and set up in defect tracking system)
Communicate requirements and get early buy-‐in
Strategies for a Successful E2E SIT 25
Track readiness
Your own team’s deliverables Test materials (test cases and scenarios) Environments Data Initial setup of defect tracking system
System readiness All participating systems Readiness to meet SIT entry criteria
Report on readiness weekly using an easy-‐to-‐understand graphic
Strategies for a Successful E2E SIT 26
A simple timeline for system readiness M
ar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
4 11 18 25 1 8 15 22 29 6 13 20 27 3 10 17 24 1 8 15 22 29 5 12 19 26 2 9 16 23 30 7 14 21 28 4 11 18 25 2 9 16 23 30 6 13 20 27
Assumptions1) Main App v0.9 will be full-featured and stable.2) Project 3 Apps 2 & 3 will be E2E-ready no later than 12-Oct.3) Starting E2E test without Project 3 Apps 4 & 5 will be feasible and will provide E2E test value.
Risks1) Required modifications to GL Journalling and back-end financial accounting have not yet been finalized or planned. (Change Request #29)
Main App v0.4 Main App v0.7 Main App v1.0 (final)
Program E2E SIT TestPartial E2E Test with Main App v0.4, then full test with v0.9 (complete)
E2E Regression Test with Main App v1.0 (final fixes)
Incremental integration & pre-test
Proj 1 App 2 R6
Proj 5 App 2 Corp Billing
Proj 1 App 1Proj 1 App 0
Late - new date not yet available Planned date Date at risk
Proj 4 all other Business Apps
Proj 6 App 1
Proj 6 App 6
Proj 6 App 4Proj 6 App 2 Proj 6 App 3
Proj 6 App 7
Proj 6 App 5
Proj 4 Enablement
Proj 4 Billing
Proj 5 App 1
Completed - Ready for E2ELegend:
Proj 1 App 2 R7
Not yet planned
GL AccountingNOT YET PLANNED
Proj 3 App 5
Proj 3 Special Infrastructure
Proj 1 App 3
Proj 2 App1
Proj 3 App 2
Proj 3 App 4
Proj 2 App 2
Proj 3 App 3
Proj 3 App 1
Proj 6 App 8
Main App v0.9
Strategies for a Successful E2E SIT 27
Verify readiness
Schedule a handover meeting with each team before scheduled entry to SIT
Ask to see evidence (give advance notice of what that should be)
Give teams a standard checklist to populate
Review the checklist and evidence with each team when you meet
Strategies for a Successful E2E SIT 28
Readiness Documentation Examples
Test requirement Criteria Sample Documentation
System/functional test Evidence of testing covering the functionality delivered in the release
• Test cases & traceability matrix • Business acceptor sign-‐off on test coverage • IDs & detail for outstanding bugs • Exploratory test documentation (charters, logs. Etc.) • IDs & detail for outstanding bugs
Regression test of systems currently in production and modified for this project
Completed regression test covering critical production functionality.
• Evidence that the required testing has been done (risk assessment, test completion checklist/report, etc.) • Sign-‐off by required IT and/or business acceptors.
Strategies for a Successful E2E SIT 29
Readiness Documentation Examples
Test requirement Criteria Documentation
UAT (or Agile Acceptance Test) Completion of (User) Acceptance Test covering business acceptance criteria .
• Business acceptor sign-‐off OR • Conditional acceptance by business acceptors, pending final acceptance at the end of SIT.
All project test levels Outstanding bugs: • No outstanding must-‐fix • Review outstanding bugs with impacts on downstream data or function
• Summary report showing current status of all bugs logged during testing • Detailed bug reports for outstanding bugs, with analysis of downstream impacts, documented workarounds, and resolution plans
Strategies for a Successful E2E SIT 30
1 – Project management 2 – Applications 3 – People 4 – Test readiness 5 – Information & communication 6 – Environments & data 7 – Questions & discussion
Strategies for a Successful E2E SIT 31
Keep yourself informed
Take advantage of all the communication channels
Make sure you are in the key program/project level status meeting
Ask regularly for updates on system readiness
Review the program/project risk and issue logs for anything that might affect your test
Chat to project managers and test leads
Strategies for a Successful E2E SIT 32
Build your network
Strategies for a Successful E2E SIT 33
Outward communication
Make regular presentations to all the stakeholder groups
SIT strategy Progress reports
Set up a “notice-‐board” to post daily status (you’ll need this during E2E SIT execution)
Intranet, etc.
Set up a “test question and answer” forum for your team’s questions and share it on the notice-‐board
Strategies for a Successful E2E SIT 34
1 – Project management 2 – Applications 3 – People 4 – Test readiness 5 – Information & communication 6 – Environments & data 7 – Questions & discussion
Strategies for a Successful E2E SIT 35
Strategies for a Successful E2E SIT 36
Some of your most difficult issues are here
Multiple platforms (hardware, OS, DBMS)
Other infrastructure, like batch schedulers and message transports
Excusive use for your test of adequate test environments
Data, co-‐ordinated across participating applications
Configuration management
Co-‐ordinated defect management
You need an Environment/Data Manager!
Strategies for a Successful E2E SIT 37
Environment/Data Manager
Inventory all applications and infrastructure elements
Work with you and SIT test team on data strategy
Work with each team to develop plans Environment plan Data plan
Develop configuration management process
Administer configuration management process during E2E test execution
Strategies for a Successful E2E SIT 38
Test Environments
Virtualize if you can – but you may not be able to
Physical environments may not exist for every application
There may not be connectivity where you need it
You may have to negotiate for exclusive use
If setup/purchase is required, there will be long lead times Start early! (Get that Environment Manager on it!)
Strategies for a Successful E2E SIT 39
Coordinated Test Bed Data
This can be very difficult to achieve
A showstopper if you don’t have it
You need a strategy and a plan
Start early! (Get that Environment Manager on it!)
Strategies for a Successful E2E SIT 40
Configuration Management Can be complex, with Production fixes and other projects going on at the same time as yours
You need an exact picture at all times
A system solution across platforms will probably not work
Design a simple “logical release” process: Start with an inventory of all the systems, platforms, infrastructure elements participating in the test
Names, versions, etc. Logical Release 1 is the starting inventory Any time ANYTHING changes: application for defect resolution, infrastructure upgrade, etc., update the inventory, and you’re now on Release 2
Strategies for a Successful E2E SIT 41
Defect Management
Essential to have one defect management system for the entire integration
Set it up so you can track by application /project/ vendor
Set up a SWAT team to handle defect diagnosis and assignment
Neutral lead, e.g., overall Project Architect Representative from each team
Make the Lead a regular attendee at your daily checkpoint during test execution
Strategies for a Successful E2E SIT 42
Use project management to go from this
Strategies for a Successful E2E SIT 43
To something like this
Strategies for a Successful E2E SIT 44
Questions & discussion
Fiona Charles fiona.charles@quality-‐intelligence.com
www.quality-‐intelligence.com
Twitter: @FionaCCharles
Images and text ©Fiona Charles 2012