classic testing mistakes: revisited

34
Classic Testing Mistakes: Revisited Matthew Heusser [email protected] Presented at the Better Software Conference San Francisco, CA - Sept. 20 th, 2005 Contributing peer reviewers: James Bach Paul Carvalho Michael Kelly Harry Robinson

Upload: makani

Post on 11-Feb-2016

51 views

Category:

Documents


0 download

DESCRIPTION

Classic Testing Mistakes: Revisited. Contributing peer reviewers: James Bach Paul Carvalho Michael Kelly Harry Robinson. Matthew Heusser [email protected] Presented at the Better Software Conference San Francisco, CA - Sept. 20 th, 2005. Organization. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Classic Testing Mistakes: Revisited

Classic Testing Mistakes:Revisited

Matthew [email protected]

Presented at the Better Software ConferenceSan Francisco, CA - Sept. 20th, 2005

Contributing peer reviewers:James Bach

Paul Carvalho

Michael Kelly

Harry Robinson

Page 2: Classic Testing Mistakes: Revisited

Organization

• Classic Mistakes: A different approach• The mistakes enumerated

– Test Management Mistakes– Test Automation Mistakes– Development Mistakes– Test Strategy Mistakes

• Root Causes• What to do tomorrow

Page 3: Classic Testing Mistakes: Revisited

On the shoulders of pioneers

Page 4: Classic Testing Mistakes: Revisited
Page 5: Classic Testing Mistakes: Revisited
Page 6: Classic Testing Mistakes: Revisited

Classic Mistake #1:De-humanize the test process

Test Management Mistakes

AKA Management by Spreadsheet,

Management by Email,

Management by MS Project …

Page 7: Classic Testing Mistakes: Revisited

Classic Mistake #2: Testers Responsible for Quality

Quality Assurance

“It’s strange that QA let that bug slip through”

Test Management Mistakes

Page 8: Classic Testing Mistakes: Revisited

Classic Mistake #3:IV&V Determines ship date

Do they really?

Test Management Mistakes

Page 9: Classic Testing Mistakes: Revisited

Classic Mistake #4:Task-based status reporting

• Examples:– Testing is “on schedule”– Testing “should be done by Tuesday”

• Consequences– Loss of credibility– Bad information for decision makers

Test Management Mistakes

Page 10: Classic Testing Mistakes: Revisited

Classic Mistake #5:Evaluating testers by bugs found

… and developers by number of bugs injected

• Consequences:– Friction– Focus on easy-to-find yet trivial bugs

(usability)– Information hiding

Test Management Mistakes

Page 11: Classic Testing Mistakes: Revisited

Classic Mistakes #6Inappropriate Models for Test Improvement

•NO

Page 12: Classic Testing Mistakes: Revisited

Classic Mistake #7:Lack of test training for developers

• Testing is a skill.

• It won’t appear like magic.

Development Testing Mistakes

Page 13: Classic Testing Mistakes: Revisited

Classic Mistake #8:Separate devs and testers

• To create friction, emphasize division

• Anything that increases the length of the feedback loop is bad.

• To improve get rid of waste and tighten the feedback loop.

Development Testing Mistakes

Page 14: Classic Testing Mistakes: Revisited

Mistake #9:When late, add Test Automation

• Someone has to learn the tool

• Someone has to record the scripts

Test Automation Mistakes

Page 15: Classic Testing Mistakes: Revisited

Mistake #10:Mine Field Test Automation

Test Automation Mistakes

Page 16: Classic Testing Mistakes: Revisited

Mistake #11:Hiring for test tool skills

• Technology skills can be taught

• Talent can’t

• The “Hit the Ground Running” Argument

Test Automation Mistakes

Page 17: Classic Testing Mistakes: Revisited

Classic Mistake #12:Insufficient diversity in test strategy

• Examples:– Only requirements based testing– Only coverage testing

• Consequence:– Missing entire classifications of defects

Test Strategy Mistakes

Page 18: Classic Testing Mistakes: Revisited

Classic Mistake #13:Over-reliance on scripted testing

All the testing we did, meticulously pulling down every menu and seeing if it worked right, didn't uncover the showstoppers that made it impossible to do what the product was intended to allow. Trying to use the product, as a customer would, found these showstoppers in a minute.

- Joel Spolsky, JoelOnSoftware.com

Test Strategy Mistakes

Page 19: Classic Testing Mistakes: Revisited

Mistake #14:Untrained exploratory Testing

Test Strategy Mistakes

• “Just think creatively”• “Try to break it”

• Exploratory Testing is a discipline

Page 20: Classic Testing Mistakes: Revisited

Classic Mistake #15:Separate test design and execution

Test Strategy Mistakes

Page 21: Classic Testing Mistakes: Revisited

Classic Mistake #16:Vacuous Documentation

• Examples:– The issue resolution document– Physical signoff/check marks– Elaborate test case templates

• Consequence:– Time spent documenting is time not spent

testing

Test Strategy Mistakes

Page 22: Classic Testing Mistakes: Revisited

Mistake #17:Trying to fix things beyond your reach

The Meta-Mistake

Page 23: Classic Testing Mistakes: Revisited

Don’t treat the symptoms

Find & Fix the root cause!

Page 24: Classic Testing Mistakes: Revisited

Root Cause #1Lack of Systems Thinking in Testing

• The law of unintended consequences

Root Causes

Page 25: Classic Testing Mistakes: Revisited

Root Cause #2:Translation Problems

Example:

- “You need to completely test this module”

Root Causes

Page 26: Classic Testing Mistakes: Revisited

Root Cause #3:Process Myopia

• Example:– The [in]famous Issue Resolution Document– “We don’t do things that way here”– Elevating process over skills

• Solutions: – The ear of the king / History Lessons– If your boss doesn’t care – ignore it

Root Causes

Page 27: Classic Testing Mistakes: Revisited

Root Cause #4:Technology Myopia

• Example– “Use XML on the next project”– “I just bought 5 copies of WinRunner …”

• Solution: – If you’re technical, they need you to do it– If you’re a manager, focus on business impact

and risk

Root Causes

Page 28: Classic Testing Mistakes: Revisited

Root Cause #5:Fred Taylor in the organization

• Examples:– Factory Mentality– High Specialization– Mixing of skill sets is verboten

• Solutions:– Peopleware, or anything by Weinberg– First Break all the rules – Jim Collins– Lead. Insulate your team.

Root Causes

Page 29: Classic Testing Mistakes: Revisited

A ‘new’ methodology

• Root Cause Analysis• Pareto Analysis• Drive out waste/tighten the feedback loop

• Then worry about better practices

(Image from Rapid Development, (c) 1996 by Steve McConnell. Used with permission from the author)

What to do tomorrow

Page 30: Classic Testing Mistakes: Revisited

Why

• New practices take permission

• It is hard (but possible) to get more done by adding work

• It is easy to get more done by subtracting work

• So start by looking to remove worst practices

Page 31: Classic Testing Mistakes: Revisited

What to do tomorrow

• Discuss• Q&A

Page 32: Classic Testing Mistakes: Revisited

Where to go for more• “Classic Mistakes in Software Testing”, Brian

Marick, STQE, 1997

• Rapid Development, Steve McConnell

• An Introduction to general systems thinking, Gerold Weinberg, 1975

• Lessons Learned in Software Testing, Kaner, Bach, Pettichord

Page 33: Classic Testing Mistakes: Revisited

Bonus Section

Page 34: Classic Testing Mistakes: Revisited

Root Cause #6:Pressure for short-term results

• Example:– “Ship to make 4th quarter numbers”

• Putting off problems instead of addressing them

• Solution:– Save your team– Professionalism means something– The Quake Example

Root Causes