a critical look at “best practices” jon bach qe/pm director [email protected]
TRANSCRIPT
A Critical Look at “Best Practices”
Jon BachQE/PM [email protected]
“Best Practices and Context-Driven: Building a Bridge”
From 2003, someone wrote…
Definition of “best”
“Of the most excellent, effective, or desirable type or quality.”
Mine: “Optimal available value”
Best Practices that aren’t
For a keynote, it’s best to have audience seating?
via @mariakedemo
Best Practice: Don’t be an asshole?
But wait…
“Someone has to be an asshole.
Polite people aren't going to change the industry
-- they made it the way it is.”
James Bach, 8/3/13
How can you argue with these?
Install the software before testing it– …but not best when testing install…– …but not best if you need a “dirty” config
Use a keyboard to type characters into a field– …but not best if a stylus is needed…– …but not best if you’re testing buffered inputs
When you find a bug, write a report– … but not best if the programmer is your brother
who is sitting right next to you
Construx
“Best practices are proven software engineering techniques, methods, lifecycles, etc. CxOne is based on software industry best practices provide insight and guidance into optimal selection, efficient deployment, and effective use of industry best practices.”
Construx (from their website)
“Best practices are proven software engineering techniques, methods, lifecycles, etc. CxOne is based on software industry best practices provide insight and guidance into optimal selection, efficient deployment, and effective use of industry best practices.”
…such as…?
“It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.”
“Gathering and agreeing on requirements is fundamental to a successful project.”
“Choosing the appropriate architecture for your application is key.”
“A best practice for constructing code includes the daily build and smoke test.”
“It is important to review other people's work.”
“It is important that testing is done proactively; meaning that test cases are planned before coding starts, and test cases are developed while the application is being designed and coded.”
© Construx (from their website)
…best, but why…?
“It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.”
“Gathering and agreeing on requirements is fundamental to a successful project.”
“Choosing the appropriate architecture for your application is key.”
“A best practice for constructing code includes the daily build and smoke test.”
“It is important to review other people's work.”
“It is important that testing is done proactively; meaning that test cases are planned before coding starts, and test cases are developed while the application is being designed and coded.”
© Construx (from their website)
…best how, and according to whom…?
It is important to choose the appropriate development lifecycle process to the project at hand because all other activities are derived from the process.
Gathering and agreeing on requirements is fundamental to a successful project.
Choosing the appropriate architecture for your application is key.
A best practice for constructing code includes the daily build and smoke test.
It is important to review other people's work.
It is important that testing is done proactively; meaning that test cases are planned before coding starts, and test cases are developed while the application is being designed and coded.
IBM – this link is broken, nice practice there
http://www.standishgroup.com/sample_research/chaos_1994_1.php
http://www.construx.com/Practicing_Earl/Context_Matters/
In fact, context matters with any "best practice". The word "best" doesn't make it immune to the reality of the context. While it may be best somewhere, it could be a "worst practice" for your project. How do you know the difference? You can't until you fully comprehend the context and that usually means trying the practice out.
A Construx employee writes critically about the term “Best Practices”, which goes against its own website (see previous slide).
Website bug … (10/15/13)
(link still broken as of 5/23/14)
Best Practice: Have Trained and Certified Test Teams
“Certification can establish the minimum and expected skills needed for test positions.”
“Introduction of ISTQB certification program is raising the standard for tester skills uniformly around the world.”
http://www.rbcs-us.com/images/documents/testing%20best%20practices.pdf
From Rex Black (RBCS Consulting) 2006
My opinion is that you do not need to be certified before you can be considered “skilled”. Certification is not a “practice”, nor does it build skill.
From STP: an article about not using “best practices” under the section titled “Best Practices”
http://www.softwaretestpro.com/tag/best-practices
Context-driven testing
http://context-driven-testing.com/
The value of any practice depends on its context.There are good practices in context, but there are no best practices.People, working together, are the most important part of any project’s context.Projects unfold over time in ways that are often not predictable.The product is a solution. If the problem isn’t solved, the product doesn’t work.Good software testing is a challenging intellectual process.Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
“It depends”
pendeo pendere pependi [to hang; to hang upon, depend on; to hang loose, hover; to be suspended, discontinued; be in suspense, uncertain, undecided].
http://catholic.archives.nd.edu
Me and Rob Sabourin
I said to Rob: “How about a keynote that’s context-driven? Maybe the content can come from my workshop about exploring best practices the day before I present the keynote?”
“I love this idea.”“This will be really cool.”“This has never been done before (that I know of).”
So… What practices will you explore? What are the pros and cons of each? What are some interesting contextual considerations?
“As a testing consultant, I don’t need the low-hanging fruit, I want the fruit that’s on top of the tree, that’s really hard to get!”“It would be cool to see the favorite 10 from the group.”“It would be cool to have an artifact -- a mind map, maybe?”
When hosting a workshop, it’s important that
you face the audience and talk to them.
Mission
Mission
Help me build the closing keynote:
“A Critical Look at Best Practices”
Tactics
Part IImagine that you are a council of elders. What so-called “best” software testing practices come to mind? Vote on a top 10 list.
Part IIPros and cons of each
Part IIIWhat should testers consider when choosing and evaluating practices?
Story
Who was here? What practices did we explore? How did we explore them? Is there a “top 10”? What are the nuances testers should consider
before using a practice? What are some real world examples and
experiences? What conclusions did we make? What artifact do we have to share? Advice to our colleagues on the other sessions.
Who
16 people (8 male, 8 female) 8 countries Testers, Test Managers, Consultants 2 years’ experience, 38 years’ experience Games, Insurance, Regulated, Government,
Banking, Anti-Virus, Big Data Dell, Nortel, Spotify, World of Tanks, Unity Neil Thompson…
From Neil’s paper…
“Best Practices and Context-Driven: Building a Bridge” - 2003
Would not call them “always-good practices” now, but “considerations for any context”
Brainstorm of “good” practices
Full list (64)
Learn how to tell a good testing story Use a test management system Carefully choose testing tools Vary your test techniques Bug advocacy: learn how to sell bugs Work smarter, not harder Make it clear to PM how much it costs to test
or deliver proper software Have Dev test it first before it goes to Test
Full list (64)
Don’t spend time doing things a machine can do better
Make sure test coverage is approved by a stakeholder
Plan the test environment and data needs early
Document enough detail to save effort Work as a team an communicate Design code to be maintainable Use exploratory testing
Full list (64)
Test early Be able to explain the value you add Work with Developers to know who tested
what Work with Dev to build in testability and
logging Weak or unclear requirements will cause
problems Prioritize Always add time to estimates
Full list (64)
Be aware of your assumptions Certify the testers Do research -- self education is important Become part of the community Create clear exit / entry / stop criteria Give yourself more time You can always find bugs in your software Be aware (observation vs. reference) Treat bugs as something positive
Full list (64)
Be aware that metrics can be dangerous Be aware of pitfalls when communicating
metrics Talk the same language (“test”, “integration”) Don’t let Dev verify bugs Use Session-Based Test Management Test software Check fixed bugs against new versions Ask questions
Full list (64)
Send testers to Per Scholas Never trust developers Talk to each other Don’t plan too much; execute as well Get feedback often Write clear descriptions of bugs Use daily standups Automate Use prototypes
Full list (64)
Communicate with end users Consult a variety of stakeholders Get good at using your product Be aware of your emotional responses Don’t be afraid of failing Gain and apply experience Look out for the unexpected Assume specs are incomplete Expect that you will never have time enough
Full list (64)
Focus and defocus Think critically Use checklists Perform smoke tests first Use a bug-tracking system Discuss issues with Developers first before
reporting Try to be effective and efficient
Vote on your 3 favorites
PICTURE
Top 10
Ask questions Talk to each other Think critically Test early Learn how to tell a good testing story Don't waste time doing things a machine can do
better Work as a team Be able to explain the value you add Don't be afraid of failing Prioritize
Now let’s see how these so-called “good” practices can go wrong…
“Ask questions” -- pitfalls
Asking too many questions could be annoying Could take too much time from stakeholders Could offend political sensibilities The answer might provoke hidden agendas Not understanding importance of who you ask the question of Forgetting that you asked the same question of the same
person The person being asked does not want to reveal they do not
know the answer Did you ask enough questions? Someone could give the wrong answer, but you don't know
“Talk to each other” pitfalls
Meetings for the sake of meetings Conflicts can come up from talking to each other Risk of too much information / confirmation bias Risk of interrupting people who do not want ot be interrupted Cultural differences could provoke conflict Speak too much -- garrulous Annoying volume to other colleagues Not easy with distributed teams, tech and time zones, language Unintentionally spread confidential info Can forget or have a lack of formal agreement what we talked
about which means wasted time Book: “Quiet -- The Power of Introverts in a
World That Can’t Stop Talking
“Think critically” pitfalls
Don't think -- court case testimony Acceptance testing Testers can go too deep for minor issues Don't take your critical thinking home Dangerous for your health -- deep psychology
problems, nothing is good enough Can be seen as over-perfectionism Can be seen as criticism Can be seen as culturally insubordinate
“Test early” pitfalls
Testers report invalid bugs because product isn't ready
If you don't have full specifications yet Project isn't yet approved by official
stakeholders There are dependencies for certain types of
testing (integration) -- preconditions Code base is too unstable, and code base
rewritten in two weeks time anyway no matter what you find
Learn to tell a good testing story -- pitfalls
Since learning takes practice, you could lose credibility as you learn
The story isn’t necessary
Don't waste time doing things a machine can do better -- pitfalls
It make take you some time to figure out that it couldn't be automated
Takes too much time to automate cases when you can just check it by hand -- machine can do it better, but it's not worth
Maybe its faster to do it manually for now BVT manual at first – easy, quick, painless to
run UNTIL the build gets more mature and iterates every hour
“Work as a team” pitfalls
Difficult to work as a team when the team dynamics might change
Meetings might be forced on people who already know how to do it individually
Einstein worked alone Sometimes you need to remove yourself
from the team in order to focus
“Work as a team” pitfalls 2
Autism / Aspergberger's will get very stressed
Dependency on each person of the team, varies on confidence, role, responsibility
Maybe that you don't get along socially with the others on the team; less effective when negative energy
If more than one "leader" on the team cause conflict
Introverts
“Be able to explain the value you add” pitfalls
Don't have to do that if it's known already and you know
There are intrinsic things that may not need to be explained
You may never know the value that you really add anyway
You don't have to justify the job because the existance of the job itself has value
“Don’t be afraid of failing” pitfalls
Some healthy fear is reasonable Don't be cocky Fear drives people to be better ("Expert
Game") and do research Testing books on the shelf
“Prioritize” pitfalls
Wasted energy because everything is equal and needs to be tested anyway
Pressure to prioritize may cause you to incorrectly prioritize
If the test approach is galumphing, prioritization is wasted effort
If you're in modeling mode, there's nothing you need to do to prioritize
There's an opportunity cost to any prioritize
Considerations for practices
Social interactions Knowing what story you need to tell
(depending on the domain of the project) Time How do you be not afraid of failure when your
software is Mission critical? Size of project (2 or 200) Budget Locale
Considerations
Lifecycle of project (Agile, waterfall, v-model) Your own technical skill Domain Legal (regulatory) Cultural Gender
In closing
Best Practice lists hide ignorance and incompetence; but make money because consultants get paid for quick wins,
BUT….
sometimes we just want to be told what to do Brainstorming practices that provide value is
easy; scrutinizing those practices can be difficult, exhausting, and even disheartening
You have a tester mindset; scrutinize any so-called “best practice” you want to get rid of.
I’m done. Let’s rumble…