testing within an agile environment - beyza sakir and chris gollop
TRANSCRIPT
![Page 1: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/1.jpg)
Testing within an agile environmentChris Gollop & Beyza Sakir
![Page 2: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/2.jpg)
Overview
History Changing Testing Concerns Adaptations Future Conclusion Further Reading
![Page 3: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/3.jpg)
History – what we do
Release every 2 weeks Trade $billions/day Average 0.4ms execution time for MTF members 10,000 orders second 1 tester per 5-6 developers Very low rate of production issues
![Page 4: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/4.jpg)
History - company
Conceived as an Agile company Agile has matured but we’re still adapting Agile testing:
Acceptance Tests Integration Tests Exploratory Testing
![Page 5: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/5.jpg)
History - company
• This has allowed us to experiment and adapt:• Time freed up• Quick feedback on changes• Historic data available to analyse
![Page 6: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/6.jpg)
History – test team
4 test specialists with different strengths complement each other as a 'team‘ Rotate through all teams• No Test Manager• Part of development team though test interests have a separate iteration
meeting• Testing is responsibility of everyone
![Page 7: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/7.jpg)
History – team structure
l
Team One Team ThreeTeam Two
Test Specialists
Business Specialists
Development Specialists
![Page 8: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/8.jpg)
History - innovate
Keep changing, even if something appears to work there may be something better
“If I had asked people what they wanted, they would have said faster horses. I gave them a car” (Henry Ford)
![Page 9: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/9.jpg)
History – measure and feedback
Measure To know if it is effective Many monitoring and measuring tools
Feedback Tying feedback to ideas Quicker this is the faster you can confidently adapt Can be numerical or social
![Page 10: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/10.jpg)
Changing Testing Concerns Over Time
Traditional Testing Maturity Models start at “ad-hoc” and move through structured testing before finishing at “Continuous Improvement”
Some of our test concerns (over time) Decrease Manual Testing & increase automated Acceptance Tests Performance DSL Speed of Feedback Increasing number of Edge Cases as system becomes more complex
![Page 11: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/11.jpg)
Changing Testing Concerns Over Time
10,000 Behavioural Tests 8,700 Acceptance Tests (UI, API) 1,300 Integration Tests (subsets of services using internal interfaces)
Acceptance Test would test a rejection Integration Tests would test the different rejection types
![Page 12: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/12.jpg)
Changing Testing Concerns Over Time
Was right at the time as needed Acceptance Tests to get good coverage quickly, trade-off is they are slower to run repeatedly.
Focus is now quicker feedback.
![Page 13: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/13.jpg)
Adaptations – Fitness Landscape
![Page 14: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/14.jpg)
Adaptations – Fitness Landscape
Each square is similar to it's neighbour Higher it is the better the solution – valleys are bad, mountaintops are good
You don't fully know where you are on the landscape You feel your way by trying out new ideas and recognising failure You may be half-way up the mountain rather than at the top And the mountains move!
Some peaks move faster than others: Google v McDonalds
Evolution isn't just biological it's everywhere
![Page 15: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/15.jpg)
Adaptations – Palchinsky Principles
Peter Palchinsky was a Russian engineer who advised on (against) Lenin Dam and Magnitogorsk – large projects which essentially failed due to being too big and unable to adapt.
Palchinsky Principles are shaped to encourage stronger innovation, better leadership and more effective policies: Variation - seek out new ideas and try new ideas Survivability - when trying something new do it on a scale where failure is
survivable Selection - seek out feedback and learn from mistakes as you go along, avoid an
instinctive reaction of denial
![Page 16: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/16.jpg)
Adaptations – Recognise failure quickly
Being able to recognize a failure means you can re-cast it into something more likely to succeed.
Accept something is failing, don’t deny or ignore it.
"Hard to see what changes would improve current process when they’re currently sweeping all problems under the ‘our context’ rug." (Anthony Green)
The essential first step is being willing to fail.
![Page 17: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/17.jpg)
Adaptations - Google v Apple
Both considered as innovative companies.
Apple innovates in things that are core to its business Google innovates on things that indirectly contribute to its business
Apple releases products that are polished and well-scoped Google releases many different products (Buzz, Wave, Docs, AppEngine) early in
their development and lets the community develop and adapt them.
Apple create new mountains on the fitness landscape Google is surfing along on, or near, the moving fitness landscape mountain-top by
adapting its strategy and offerings.
![Page 18: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/18.jpg)
Adaptations – a brief history of what we have tried
![Page 19: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/19.jpg)
Adaptations – what worked
Production Data Testing in Live Big Feedback Remove Intermittent Tests Tester Pairing
![Page 20: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/20.jpg)
Adaptations – Sanitised Production Data
Raw data (database) Patterns of use Data profile patterns
Migration (automated) Real world data Larger than we can create Many edge cases Try and reincorporate into our Acceptance Tests Performance Staging
Progression in what and how we use the data
![Page 21: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/21.jpg)
Adaptations – Testing in Live
Uses same hardware and code as it is just an isolated venue Runs a subset of the Acceptance Tests
When we expanded our Exchange by adding a multi-tenanted concept known as a venue we took advantage for testing
Verifies that the various components of the exchange are hooked up and configured correctly and that the deployment process worked correctly
Runs automatically throughout the day as one of our monitoring systems
![Page 22: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/22.jpg)
Adaptations – Big Feedback
![Page 23: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/23.jpg)
Adaptations – Big Feedback
Important CI pipeline builds at the top Red (failed) items rise to the top Continuously and easily expanded Useful features survive
Failure ownership/failures at a glance - AutoTrish Intermittent tests - Magic Eight Ball Auto Embargo Performance metrics
Build sheriff to keep check-in embargo
![Page 24: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/24.jpg)
Adaptations – Intermittent Tests
Tests that can fail even if feature is working Now have <1 intermittent tests per run (out of 8,200) Each about 1 test run in 20
Hides genuine failures Have previously missed a release as couldn't tell if genuine
Can result from Asynchronous messaging Tests changing global settings/lack of isolation End of day transitions Date/time
![Page 25: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/25.jpg)
Adaptations – Intermittent Tests
anon
anon
![Page 26: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/26.jpg)
Adaptations – Intermittent Tests
All Failures run twice Fail twice = failure Fail once, pass once = intermittent Accepted as something we need to deal with
Work to remove DSL improvement Fix a genuine issue (race condition) End of iteration is given over to removing intermittency Build Sheriff also works on this
![Page 27: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/27.jpg)
Adaptations – Intermittent Tests
![Page 28: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/28.jpg)
Adaptations – Intermittent Tests
![Page 29: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/29.jpg)
Adaptations – Tester Pairing
Each tester has their own strengths
Noticed that at rotation handover the new tester generally found a bug Issue with the testers (denial) Something we can harness (acceptance)
Already do development pairing (Dev & Dev and Dev & Tester)
When do we pair? At the right time for the story
Initial brainstorming when creating a mindmap When we feel we need to (pull) When someone has time (push)
![Page 30: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/30.jpg)
Adaptations – mindmap
![Page 31: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/31.jpg)
Adaptations – what didn't work
Not many absolute failures
Although many reasons it can fail
Even if something fails you learn something
Natural selection will continue
You can't stay still, unless you are improving you are losing ground as others will innovate or adapt
![Page 32: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/32.jpg)
Adaptations – facts
Of the top 25 industrial corporations in the United States in 1900, only two remained on that list at the start of the 1960s. And of the top 25 companies on the Fortune 500 in 1961, only six remain there today (IBM)
Average life expectancy of a Fortune 500 company has declined from around 75 years half a century ago to less than 15 years today, and heading towards 5 years (Deloitte)
![Page 33: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/33.jpg)
Future
Expanding the way we test
Speed up feedback Increasing integration test coverage Pushing UI to API – testing at the right layer Change AT run order
Improving feedback/communication Test team standups
Documentation What is just enough How to get people up to speed in a complex system
![Page 34: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/34.jpg)
Future
Is about the people
Team make-up is varied and we recruit to keep it this way
“’Testing is a strictly formal process that by its nature should not vary’ - this is a typical belief by people who know nothing about testing.” (James Bach).
![Page 35: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/35.jpg)
Conclusion
Need to experiment and adapt to stay ahead
Even if something works it does not mean there isn't something better
Make changes on a scale where failure can be survived
Do not be afraid of failure but know how to recognise it quickly
![Page 36: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/36.jpg)
Further Reading
• Tim Harford• Richard Wiseman• Malcolm Gladwell• Daniel Kahneman• Dan Ariely• Richard Thaler
• LMAX Staff blogs - http://blogs.lmax.com/staff-blogs/
![Page 37: Testing within an Agile Environment - Beyza Sakir and Chris Gollop](https://reader034.vdocuments.net/reader034/viewer/2022042819/55c7f7d4bb61ebfc638b4619/html5/thumbnails/37.jpg)