how to suspend testing and still succeed – a true story graham thomas independent software testing...
TRANSCRIPT
How toSuspend Testingand Still Succeed
– A True Story
Graham ThomasIndependent Software
Testing Consultant
1
ABSTRACTThis is the story of a testing programme that was destined to fail, but ultimately succeeded. We will look at what went wrong, explain why testing had to be suspended, and discuss
how with no real hope of recovery the team managed to set and meet their resumption requirements, and ultimately complete their testing on time.
We will discuss where things started to go wrong, how this was identified, and what
measures were taken to ensure a successful resolution. We will also look at the detail of the challenges that the testing team, and the program,
were daily presented with when testing was suspended. And tell how innovation, ingenuity and perseverance, against all the odds, won the day.
This is a real ‘war story’, from the testing front line, with valuable hard won experience, and is told in the very real hope that it will benefit all who hear it.
2
AGENDA
Background & Set the Scene
Testing Strategy
Initial Progress
Suspension
Resumption
Further Demands
Conclusions
3
BACKGROUND The Program was scoped as delivery of the infrastructure
and processing of the new Faster Payments solution for one of the member banks
Faster Payments is a new (mid 2008) component of the UK’s Financial Services Infrastructureo First new payments system in UK for 20 yearso Centrally Regulatedo High Volume and High Availabilityo Near Real Time financial transactions between member banks for
phone, internet and standing order transactions.o Daily (5/7) settlement
4
BACKGROUND 2 Arrived as Program Test Manager just as;
o Phase 1 development was deliveringo The first phase of external Scheme testing commenced (Self Certification)o Development plans for subsequent releases were firming upo The Program Testing Strategy was issued for review
Role was to pull together existing testing threads of:o Unit Testing (offshore, external suppliers, and internal)o Systems Testingo Systems Integration Testingo External Scheme Testingo OATo UATo Business Procedure Testing
Providing oversight, control, and stakeholder management for testing
5
TESTING STRATEGY First task was to review the Draft Testing Strategy
Met with current development and testing approach Aligned with outsourcing partner Strengthened testing organisation with representation at Program level Weak in Systems Integration Weak test measurement Weak in presentation
• Document too long 50+pp• and not audience friendly, too much text, few diagrams
Outcome of review:o Testing Strategy ‘fit for purpose’o Systems Integration Risk accepted at Program level
• Mitigation too expensive and not viable within timeframe
6
INITIAL PROGRESS Systems Integration completed successfully (as scoped)
Initially testing hampered by slippage in development delivery;o Single phase delivery into testing became multiple drops of codeo Testing planned on a Just in Time basis
OAT planning delayed and execution moved back
Slow progress in UAT execution due to issues with the test automation infrastructure supplied by the solution vendor
Non-functional proving of infrastructure took 250% longer than planned and had a qualified exit (i.e. not complete)
7
SUSPENSION Take a view from the room?
o How many people have actually experienced testing being suspended?
o What if any where the suspension criteria?
o What was the impact?
o Who got the blame?
8
WHAT THE STANDARD SAYS IEEE829
o 9.2.7 (LTP Section 2.7) Suspension criteria and resumption requirementsSpecify the criteria to be used to suspend all or a portion of the testing activity on the test items associated with this plan. Specify the testing activities that must be repeated when testing is resumed.
Two Website References:o Know when to pause in a series of tests. o If the number or type of defects reaches a point where the follow on testing has no
value, it makes no sense to continue the test; you are just wasting resources. o Specify what constitutes stoppage for a test or series of tests and what is the acceptable
level of defects that will allow the testing to proceed past the defects.o Testing after a truly fatal error will generate conditions that may be identified as defects
but are in fact ghost errors caused by the earlier defects that were ignored.
A Document Reference:o Suspension of test execution will occur if the number or severity of incidents raised is so
great as to make further execution valuelesso Specify the testing activities that must be repeated, when testing is resumed
9
WHY WE STOPPED TESTING We had realised that we did not have enough test resource to
complete the task so had brought in external resource to augment UAT
At current rate of progress in UAT 8 weeks into a 12 week schedule it was going to take over a year to complete!
Many of issues (defects) were to do with automation infrastructure and product configurationo The Systems Integration Risk had matured!
Unless we did something radical we were going to have to declare to the Scheme that we were failingo This would mean losing our implementation slot, and cause significant
internal cost overruns and knock-on impacts on other programs of work
10
SO NOW WHAT? We had to swiftly identify what was wrong
o A quick series of workshops with all stakeholders• Developers, Testers, Suppliers, Customers, and Business
Once we had an idea of what was wrong we had to speedily plan a resolutiono A targeted Systems Integration Test, carried out in the UAT environments
• Our Systems Integration risk had maturedo Product configuration
• We had a default supplier provided configuration which needed to be revised to reflect the current software and business model
o Test infrastructure optimisation• Multi threaded and concurrent testing configuration
We had to execute the resolutiono Allocate highest non-production priorityo Extended working hours, Provision of support, reservation of key resource to supporto Close knit group working between all teams
And the program still had to deliver on timeo We had stopped, were going to lose time resolving our problems, but still had to deliver on the same
date!
11
RESUMPTION We also had to determine what would allow us to resume testing
o New UAT manager wanted all UAT tests run successfully before UAT could commence!
o Requested a set of viable Resumption Requirements that met the reasons why we suspended testing• System Integrated• System Configured• Test Automation met desired throughput rates• That we developed a set of re-usable System Integration Tests that could be
re-run• No Short-cuts, position or views – We were not going to kid ourselves
I tried in vain to get the organisation to use the term Resumption Requirements, but everyone kept on sayingResumption Criteria!
12
RE-PLANNING
It wouldn’t be a software development without re-planning!
We re-planned UAT Executiono Combined Future development releases
• They weren’t separate business releases• Saved re-testing time lost with resumption activities
Utilised contingency Accepted a scheme request to move the implementation
date back gaining 3 weeks Moved Business Procedure Testing out of UAT Implemented predictive test measures
13
RESTARTING TESTING Amazingly the 4 week suspension was completed on time
o We fixed outstanding integration issueso We configured the producto We resolved the problems with test automationo And created a repeatable set of System Integration tests
o And in 5 weeks UAT had re-planned from a 3 phase to a 1 phase test cycle.
It took a week to deploy the changeso Don’t forget there may be a lag in resuming execution
And within 3 days of test execution we had already completed 60% of the revised UAT scheduleo I don’t want to make it sound that easy, because some tests required multiple
cycles to be run, however, suddenly the end was in sight and achievable
14
FURTHER DEMANDS So all is looking good, testing has just finished a 4 week
suspension, and current throughput rates show that UAT will complete on time.
However this is not an easy programo It is the largest, costliest and most complex development that the
organisation has undertaken for nearly a decade, and it is in an area of new technologies that the organisation is not familiar with
More testing is identified for UATo We have 6 times as much to do as they thought!o OAT planning completed and shows that 9 months are required not the 3
that the program haso And External Scheme testing is about to commence, and the timescales
have been cut by over 50%!
15
ORGANISE FOR SUCCESS
16
We had been playing football like 11 year olds o We had all been chasing after the ball – Testing Suspension
We instituted a daily War Room involvingo Decision Makerso The doerso Program Office to record and account
We met daily at 13:00, or more frequently if required to manage issues
We made decisions and gave direction
THE WAR ROOM 1 physical dedicated Meeting Room
o Equipped with projectoro Reports printed out and placed on the
wallso Measures and metrics for all to seeo Dedicated conference number
Strict rules as to attendance and protocol
Handled all program issues, not just testing
Worked as a collective, with individual responsibility
17
THE END GAME We used all of the tools at our disposal
o Took a firm view on prioritisation of defectso Took a clear view of scope for current releaseo Compromised for success
• To fail to deliver would cost the organisation £1m per month, excluding reputational and other damage
o Exercised tight control on changeo Engaged with Business, IT and Change throughouto Used all contingency even within deployment windowo Limited go-live exposure with low initial volumeso Provided full implementation hand-holding
18
POSTSCRIPTWhat did Success look like?
The program delivered on-time There were zero defects in the first 3 months The development group gained the confidence to know that it
could deliver the remaining phases of the program
The bad practice lessons were not repeated for future phases
As a result of the suspension, senior management questioned the original Testing Strategy,o The Testing Strategy was retrospectively re-writteno And signed off shortly after the system went live!
19
SUMMARY Suspension and Resumption is more complicated than the standard leads us
to believe
You will probably not be able to precisely specify suspension criteria prior to the suspension of testing. You will only know these at or around the time you suspend testing. (The standard is not too helpful here!)
Resumption Requirements will be directly dependent upon the reasons for suspension and will have to be written and agreed very shortly after suspension. (You are not just repeating previous testing as per the standard.)
Suspending testing does not necessarily buy you more time to test, and in fact may actually give you less
Suspending testing is a very powerful tool. Don’t underestimate the amount of management attention you will receive, but do be aware of the benefits that fully met resumption requirements can bring
20
CONTACT DETAILS
Graham Thomas
Independent SoftwareTesting Consultant
[email protected] +44 7973 387 853 www.badgerscroft.com
21