![Page 1: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/1.jpg)
Skeptical Self-Defense For The Serious Tester
Laurent BossavitTwitter:@Morendil http://institut-agile.fr/
![Page 2: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/2.jpg)
Programmer. (Homo sapiens geekus.)
@Morendil
![Page 3: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/3.jpg)
FINISHED FILES ARE THE RESULT OF YEARS OF SCIENTIFIC STUDY COMBINED WITHTHE EXPERIENCE OF YEARS.
How many F’s do you count?
![Page 4: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/4.jpg)
How we view testing
![Page 5: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/5.jpg)
How we view testing
• Execute test cases?
• Obsess over details?
• Look for bugs?
• Solve complex puzzles?
• Reveal (sometimes unpleasant) truth about the quality of a product?
![Page 6: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/6.jpg)
What we have in common...
![Page 7: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/7.jpg)
What we have in common...
![Page 8: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/8.jpg)
What we have in common...
![Page 9: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/9.jpg)
What we have in common...
![Page 10: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/10.jpg)
What we have in common...
![Page 11: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/11.jpg)
Bullying?
![Page 12: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/12.jpg)
Bullying?
• Telling me how to do my job, when I do not assess you as competent
![Page 13: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/13.jpg)
Bullying?
• Telling me how to do my job, when I do not assess you as competent
• Telling my manager how I should do my job
![Page 14: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/14.jpg)
Bullying?
• Telling me how to do my job, when I do not assess you as competent
• Telling my manager how I should do my job
• Belittling my or my community’s contributions to theory or practice
![Page 15: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/15.jpg)
Mainstay of the bully’s arsenal
![Page 16: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/16.jpg)
How to be a bully
• Use “studies show” or “research proves”
• Use citations liberally (Bossavit 2013)
• Rely on unsound “metrics”
• Claim large-scale surveys (“thousands of projects”), exaggerate sample sizes
• Claim large monetary impacts
![Page 17: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/17.jpg)
The claim:
"It can cost you up to 110 times more to correct a requirements defect found in production than if that same defect
were found during requirements development (Grady, 1999)"
-- EGB Consulting Web site
(“Cost of defects curve”)
![Page 18: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/18.jpg)
The problem:
• Justifies “Requirements-Based Testing”, any number of Software Lifecycles
• Supposedly supported by “decades of research”
• Actually very hard to find corroborating data, best available data contradicts claim
![Page 19: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/19.jpg)
The evidence:
![Page 20: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/20.jpg)
Relative cost to fix errors:Boehm (1976)
![Page 21: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/21.jpg)
Relative cost to fix errors:U.S. Fed.Hwy. (2007), after McConnell
![Page 22: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/22.jpg)
Hughes Aircraft (1998)
![Page 23: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/23.jpg)
Hughes Aircraft (1998)
![Page 24: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/24.jpg)
Hughes Aircraft (1998)
![Page 25: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/25.jpg)
"This study didn't accurately record the engineering times to fix the
defects, so we will use average times summarized from several other
studies to weight the defect origins"-- R.B. Grady, 1989
![Page 26: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/26.jpg)
"An important distinction that emerged was that the large effort
multiplier holds for severe defects; many defects with lesser impact will not cost appreciably more to change
after delivery than before."-- Shull et al.,What We Have Learned
About Fighting Defects
![Page 27: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/27.jpg)
The claim:
High ROI (Return on Investment) for IV&V (Independent Validation &
Verification)
(“Big money” claims)
![Page 28: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/28.jpg)
83 x IV&V budget= $1.6 billion!
NASA IV&V Facility Report, 2006
![Page 29: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/29.jpg)
The problem:
• GIGO - bad assumptions in, nonsense results out
• Missing a key variable for ROI calculation (the actual cost of fixing defects)
• Suffers from “counting to one” issues
• Suffers from “Flaubert math”, from the following puzzle by the French writer:
![Page 30: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/30.jpg)
What is the captain’s age?
A ship sails the ocean. It left Boston with a cargo of cotton. It grosses 200
tons [...], there are 12 passengers aboard, the wind is blowing East-North-East, the clock points to a
quarter past three in the afternoon.
![Page 31: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/31.jpg)
The claim:
Any number of statements of the form “46% of project failures originate from
poor requirements”(e.g. the infamous Chaos Report)
![Page 32: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/32.jpg)
The problem:
• These “studies” are typically opinion surveys, interviewing executives over the phone or by email
• Similar surveys find that 46% of Americans believe in creationism
![Page 33: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/33.jpg)
The trouble with opinions...
![Page 34: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/34.jpg)
The claim:
A 1990s survey by U.S. Defense showed a 75% failure rate for waterfall
projects
![Page 35: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/35.jpg)
Biggest bluff of all
$35.7 billion - 75% waste(DoD projects)
“Requirements Risks Can Drown Software Projects”
(Leishman, 2002)
![Page 36: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/36.jpg)
The problem:
• No such study ever existed (!)
![Page 37: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/37.jpg)
"At the 5th Annual Joint Aerospace Weapons Systems Support, Sensors, and
Simulation Symposium in 1999, the results of a study of 1995 Department of Defense (DoD) software spending were
presented. (Jarzombek 1999)"-- Leishman, 2002
![Page 38: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/38.jpg)
"A 1999 review of failure rates in a sample of earlier DoD projects drew grave
conclusions: Of a total $37 billion for the sample set, 75% of the projects failed or were never used, and only 2% were used
without extensive modification."-- C. Larman, 2003
Also cited by Benefield, Korhonen, Knoernschild, Gilb, Sutherland, Leitass, Leinonen, Martin (ObjectMentor), etc.
(indirect cite via Larman in many cases)
![Page 39: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/39.jpg)
"I do not have readily available access to that information..."
-- Lt. Col. (Ret.) Jarzombek(personal email, 2013)
![Page 40: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/40.jpg)
![Page 41: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/41.jpg)
![Page 42: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/42.jpg)
![Page 43: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/43.jpg)
1979 19952 % 2 %2 % 3 %19 % 20 %29 % 29 %46 % 47 %
Two-tailed chi-squared test, under reasonable (but debatable) assumptions, of the hypothesis “H0 = These results come from two samplings of
the same population, rather than one being a copy of the other” - rejected at p < 0.001
![Page 44: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/44.jpg)
More claims...
• 10x software engineers
• Cone of Uncertainty
• “Only x% of a product’s features are used”
• Various revisionist histories
![Page 45: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/45.jpg)
For more details...
![Page 46: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/46.jpg)
Bullies exploit:
• Reciprocity
• Commitment
• Social proof
• Liking
• Authority
• Scarcity
![Page 47: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/47.jpg)
You can fight back!
![Page 48: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/48.jpg)
![Page 49: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/49.jpg)
Self-defense skills
![Page 50: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/50.jpg)
Self-defense skills
• “Citation needed”
![Page 51: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/51.jpg)
Self-defense skills
• “Citation needed”
• Look for primary sources
![Page 52: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/52.jpg)
Self-defense skills
• “Citation needed”
• Look for primary sources
• Google-fu
![Page 53: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/53.jpg)
Self-defense skills
• “Citation needed”
• Look for primary sources
• Google-fu
• Academic search: Google Scholar, ACM, IEEE
![Page 54: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/54.jpg)
Self-defense skills
• “Citation needed”
• Look for primary sources
• Google-fu
• Academic search: Google Scholar, ACM, IEEE
• “Search by date” to identify spread pattern; Google n-Grams
![Page 55: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/55.jpg)
Curiosity
![Page 56: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/56.jpg)
Skepticism
![Page 57: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/57.jpg)
Tenacity
![Page 58: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/58.jpg)
Don’ts
• Don’t let bullies turn you away from “unstructured” testing
• Don’t turn into a bully, i.e. rely on borrowed authority, plagiarism, claims of dubious status
![Page 59: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/59.jpg)
Do’s
• Dare to be a scientist
• Question: claims, assumptions, authority
• Take power back from bullies:
• Occupy Software Engineering,
• Occupy Testing!
![Page 60: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/60.jpg)
![Page 61: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/61.jpg)
Remember, remember,the fifth of November,
the misleading statistics and plotI know of no reason
to blindly trust another’s plot
![Page 62: Laurent Bossavit - Skeptical Self-Defense for the Serious Tester - EuroSTAR 2013](https://reader033.vdocuments.net/reader033/viewer/2022042821/55d5fb8ebb61eb896d8b4600/html5/thumbnails/62.jpg)
Remember, remember...