testing framework
TRANSCRIPT
October 15, 2004 – 1
Welcome
IPMA and SolutionsIQ
Professional EventTesting, Testing, 1…2…3…
Improving software quality -- one bug at a time
October 15, 2004 – 2
Agenda
• Building the Test Framework– Jan McCollum, SolutionsIQ
• Break
• Practical Panel Discussion– Cheryl Hainje – AFRS Product Manager, OFM– Dotti Lane – QA Project Manager, OFM– Tim Vessey – POS Project Manager, LCB– Stein Wang – Quality Assurance Lead, SolutionsIQ
• Break
• Testing Templates & Checklists
October 15, 2004 – 3
SolutionsIQ Overview
• SolutionsIQ is a full-spectrum IT services company
• 25 years of technology services and solutions • 400+ consultants• Corporate headquarters (Bellevue, WA)• Professional Services (Bellevue, WA)• Oregon Branch Office (Lake Oswego, OR)• 8+ years of serving the State of WA
– DOC, AOC, LCB, DNR, DOL, LNI, & DSHS
October 15, 2004 – 4
SolutionsIQ Expertise
• Professional Services Division– Consulting and Analytical Solutions
• Project management• Assessments and feasibility studies• Design and architecture roadmaps
– Development and Test Solutions• Full life cycle development projects• Custom application development• EAI, portals, and business intelligence• Quality assurance and testing solutions
October 15, 2004 – 5
Building the Testing Framework
Jan McCollumManager, Quality Assurance and Testing Solutions
October 15, 2004 – 6
Setting Goals
• Knowing WHAT you want is as important as knowing how to get it
– Defining the vision– Defining the timeline– Gaining acceptance and buy in
October 15, 2004 – 7
Defining the Vision
• To define the vision look at what came before
– What went well– What went badly– What now– Where do you want to go
October 15, 2004 – 8
Testing vs. Quality Assurance
• Testing is about finding bugs
• Quality Assurance is about preventing them!
October 15, 2004 – 9
Quality Assurance
• Takes time• Is about the overall effort –
including development• Methodologies can be very formal
October 15, 2004 – 10
QA Applied to Testing
• Quality assurance principals applied to the testing effort will produce higher quality work
October 15, 2004 – 11
Establishing a Timeline
• The 6 month / 1 year / 3 year plan
– Implement processes and strategies that give the best return on investment
October 15, 2004 – 12
Quality Testing Roadmap
• After the goals and objectives are complete, make them real by publishing the quality testing roadmap
October 15, 2004 – 13
Quality Testing Roadmap
• Roadmap should include…
– Test team structure– Communications plans– Test processes– Test procedures
October 15, 2004 – 14
Quality Testing Roadmap
• Test scope• Test dependencies and impacts• Automation transition plan• Test deliverables
October 15, 2004 – 15
Gaining Acceptance and Buy In
• Development• Business management• Project management• IT management• Customer/product support
October 15, 2004 – 16
Making it Happen!
• Organizational structure• Qualified candidates• Roles and responsibilities
October 15, 2004 – 17
Test Planning
• The master test plan: a one-stop shopping guide for your project
– Contents– Contributing documents– Sign-off procedures
October 15, 2004 – 18
Test Planning
• Test matrix and test suites
– Detailed test steps– Pass/Fail results– Tester who performed tests
October 15, 2004 – 19
Test Planning
• Test case design – what is a good test case?
– Accurate – tests what it’s designed to test
– Repeatable, reusable – has a life after this release
– Economical – no unnecessary steps
October 15, 2004 – 20
Test Planning
• Test case design
– Traceable to a requirement– Appropriate for test environment,
testers– Self-standing has enough
information for anyone to run
October 15, 2004 – 21
Test Planning
• Test case design: How to make good test cases better
– Setup, environment, data– Steps, actions and expected results– Use active voice in expected results– System displays this, does that– Simple, conversational language
October 15, 2004 – 22
Test Planning
• Test case design: Why work to improve test cases?
– Productivity – less time to write and maintain cases
– Testability – less time to execute them– Scheduling – better reliability in
estimates
October 15, 2004 – 23
Defect (Bug) Management
• Deciding upon a tool
– Easy of configuration– Ability to add/change fields– Reporting capabilities
• Integrated solution
October 15, 2004 – 24
Defect (Bug) Management
• The bug lifecycle
– Who can create bugs– Who can assign bugs– Who can close bugs
October 15, 2004 – 25
Defect (Bug) Management
• The bug triage meeting– Purpose and who should go
• Reporting– Determining a trend
• Bug metrics– Number of bugs found– Bugs found in production vs. test
cycle
October 15, 2004 – 26
Moving On
• Improving the process: Requirements traceability
– Test cases for each requirement– Requirements matrix– Tracing requirements to defects
October 15, 2004 – 27
Moving On
• Improving the process: Risk-based testing
– You can’t test everything so test what is important
– The risk list and how to use it to drive test strategy
October 15, 2004 – 28
Broadening Your Scope
• Build verification testing– Also called smoke or acceptance tests– Is a subset of the major functional
areas
• Integration testing– Testing the entire system
October 15, 2004 – 29
Broadening Your Scope
• Compatibility testing– How application works with other apps
• Configuration testing– Testing on different configurations
• Setup testing– Testing the installation
• Regression testing– Verify if bug fixes are successful
October 15, 2004 – 30
Broadening Your Scope
• Black box testing
• White box testing
• Grey box testing
October 15, 2004 – 31
Improving Quality
• Testing metrics – measure your success
• Bug tracking metrics– Number found– Number found per component– Daily bug find rate
October 15, 2004 – 32
Improving Quality
• Test case effectiveness– Metric: Test case effectiveness; test
case effectiveness = bugs found in test/total found * 100
• Test coverage– Metric: Test coverage (absolute) =
tests conducted/total tests * 100
October 15, 2004 – 33
Improving Quality
• Test team performance– Metric: Test process effectiveness:
test process effectiveness = bugs fixed/bugs found * 100
– Metric: Planned days vs. actual days in test
October 15, 2004 – 34
Improving Quality
• QA and test involvement early!
• Design reviews– Why testers should attend
• Develop and use checklists
• Project closeout meetings– You should have them
