cse 7314 software testing and reliability robert oshana lectures 5 - 8
DESCRIPTION
CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8. [email protected]. Chapter 3. Master test planning. Introduction. Test planning is key to successful software testing Largely omitted due to time constraints, lack of training, cultural bias - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/1.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
Lectures 5 - 8
![Page 2: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/2.jpg)
Chapter 3
Master test planning
![Page 3: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/3.jpg)
Introduction
• Test planning is key to successful software testing
• Largely omitted due to time constraints, lack of training, cultural bias
• A test schedule is not a test plan !!• Usually measured by number of test
cases run
![Page 4: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/4.jpg)
Levels of test planning
• Master test plan; orchestrates testing at all levels– Unit– Integration– System– Acceptance– Others are alpha, beta, customer
acceptance, build, string, development
![Page 5: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/5.jpg)
Levels of test planning
![Page 6: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/6.jpg)
Importance of test planning
• Goal is to deal with the important issues of testing– Strategy– Resource utilization– Responsibilities– Risks– Priorities
• Process is more important than the document
![Page 7: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/7.jpg)
Importance of test planning
![Page 8: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/8.jpg)
Audience analysis
• Audience for a unit test plan is a lot different than the audience for a system test plan
• Different tolerances for what will be read
![Page 9: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/9.jpg)
Activity timing
• Test planning should be started as soon as possible– Same time as requirements specs and
project plan are being developed
• Could have significant impact on the project plan
• Use TBD’s !!
![Page 10: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/10.jpg)
Timing of test planning
![Page 11: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/11.jpg)
Standard templates
• Important to have a template
• IEEE Std 829-1998 for Software Test Documentation
• Customize as necessary
• Strive to improve usability over time
![Page 12: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/12.jpg)
IEEE test plan template
• Test plan identifier• Table of contents• References• Glossary• Introduction• Test items• Software risk issues• Features to be tested• Features not to be
tested
• Approach• Item pass/fail criteria• Suspension criteria
and resumption requirements
• Test deliverables• Testing tasks• Environmental needs• Responsibilities• Staffing, training,
schedule, risks
![Page 13: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/13.jpg)
1.0 Test Plan Identifier
• Keep track of most current version
• Use CM
• Must be kept up to date !
• No “post-implementation” test planning!
![Page 14: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/14.jpg)
2.0 Table of Contents
• List each topic in the plan
• References
• Glossaries
• Appendices
• Reader can use to quickly review topics of interest
![Page 15: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/15.jpg)
Introduction or Scope
• Scope of the project
• Scope of the plan
• Master test plan may cover the entire project (embedded system) or may be many MTPs for a project
![Page 16: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/16.jpg)
Scope of test and evaluation plans
![Page 17: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/17.jpg)
Test items
• Describes programmatically what should be tested
• Oriented to the level of the test plan
• Must reference requirements spec, design spec, user’s guide, operations guide, etc
![Page 18: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/18.jpg)
Risk issues section
• Used to determine what the primary focus of testing should be
• Hard to test everything in a given release
• Software risks that drive testing– Interfaces to other systems– Modules with a history of defects– Security, performance, reliability SW– Features difficult to change or test
![Page 19: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/19.jpg)
Features to be tested
• What to be tested from a customer point of view (as opposed to test items -> viewpoint of the developer)
• May help determine which features can be removed
![Page 20: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/20.jpg)
Prioritized list with cut line
![Page 21: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/21.jpg)
Features not to be tested
• Some features do not need to be tested– Not changed– Not available for use
• Used to reduce risk by raising awareness – Can help you attain additional resources
• May grow if project falls behind
![Page 22: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/22.jpg)
Approach/strategy section
• Description of how testing will be performed (approach)
• Explain any issues that have a major impact on the success of testing and the project (strategy)
• Include entry and exit criteria for each level of testing
![Page 23: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/23.jpg)
Influences on strategy decisions
![Page 24: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/24.jpg)
Methodology decisions
• “off the shelf” methodology or create your own– How many testers and when?– How many beta sites?– Testing techniques?– Testing levels?
• Environment becomes more realistic the higher you go
![Page 25: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/25.jpg)
Test level decisions
![Page 26: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/26.jpg)
Typical test levels
![Page 27: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/27.jpg)
staticanalysis
fully specifiedand compiledcode
* code inspection* correctness verification* tools (lint)
unittest ?
unit test
* code coverage* oracle compare
run timecodeanalysis
yes
no operationalprofiles
statisticaltesting withusage models
* leak detectors* instrumentation
* field data
functiontheoretic sequenceenumeration
testingoracletest grammar
and usagemodel creation
functionmappingrules
Model for testing
![Page 28: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/28.jpg)
Automating the Testing Process
Software Test StationTestSequences
ExpectedResults
Output Data/Results
Oracle
UsageModels
CraftedTest
Cases
Script SW
SoftwareUnder Test
Control
Data
Control andSequencing ofScript
TestData/
Results
Modelingtool
Modelingtool
Output Data/Responses
Output Data/Responses
Certification team
Testscriptingsoftware
Randomtest casegenerator
Testresultsverification
Teststation controlsoftware
![Page 29: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/29.jpg)
Resources
• Best laid plans can be ruined– Development running late (testers wait)– Ship date moved forward
• Testing schedule should contain contingencies
• Finding testing resources is a strategy decision
• Can also become a political issue
![Page 30: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/30.jpg)
Test coverage decisions
• Several types of coverage– Code coverage– Requirements coverage– Design and interfaces–Model coverage
![Page 31: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/31.jpg)
Walkthroughs and inspections
• Reviews of requirements, design, code, etc is an important verification technique
• Complementary activities to testing
• Walkthrough; peer review
• Inspection; formal evaluation technique
![Page 32: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/32.jpg)
Software evaluation process
![Page 33: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/33.jpg)
Walkthroughs vs inspections
Walkthroughs inspections
Participants Peer(s) led by author
Peers in designated roles
Rigor Informal to formal Formal
Training required None, informal, or structured
Structured, preferably by teams
Purpose Judge quality, find defects, training
Measure/improve quality of product and process
Effectiveness Low to medium Low to very high, depending on training and commitment
![Page 34: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/34.jpg)
Configuration management
• Describes how CM should be handled during testing– Change management– Process for reviewing, prioritizing,
fixing, and re-testing bugs
• Tradeoff between fixing too many bugs and freezing code too early
• Use a CCB
![Page 35: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/35.jpg)
Collection and validation of metrics
• Metrics collection and validation can be a significant overhead
• What metrics to collect?
• What will they be used for?
• How will they be validated?
• Need a way to measure testing status, effectiveness, quality, etc
![Page 36: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/36.jpg)
Tools and automation
• Testing tools can be a big advantage but, if not used correctly, can also be a disadvantage
• May take more time than manual• Regression may take less time• Use of testing tools should be well
planned (plan for training and integration)
![Page 37: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/37.jpg)
Changes to the test plan
• Include all the key stakeholders in development and review cycles
• Small changes may not have to go through approval cycle
• How often to update (weekly, monthly)
• How should the plan be reviewed?
![Page 38: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/38.jpg)
Meetings and communication
• Standard meetings should be described here– CCB– Presentations to users and
management
• Status reporting guidelines• Chains of command for conflict
resolution
![Page 39: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/39.jpg)
Other strategy issues
• Multiple production environments
• Beta testing
• Test environment setup and maintenance
• Use of contractual support
• Unknown quality of software
• Feature creep
![Page 40: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/40.jpg)
Items pass/fail criteria
• Each item has to have an expected result– Test cases passed and failed– Number– Type– Severity and location of bugs– Usability– Reliability and stability
• All test cases not created equal
![Page 41: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/41.jpg)
Suspension criteria and resumption requirements
• Conditions that warrant temporary suspension of testing
• Sometimes continuing on can be wasted effort
• Metrics can be used to flag these conditions
• Testing may be halted if a certain number of bugs are found
![Page 42: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/42.jpg)
Simple GANT chart
![Page 43: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/43.jpg)
Test deliverables
• List of documents, tools, other components to be developed and maintained– Test plans– Design specs– Test cases– Custom tools– Defect reports– simulators
![Page 44: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/44.jpg)
Testing tasks
• Identifies the set of tasks necessary to prepare for and perform testing
• Intertask dependencies and any special skills
![Page 45: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/45.jpg)
Environmental needs
• Hardware, software, data, interfaces, facilities, publications, security access pertaining to the testing effort
• Attempt to configure testing environment similar to real world if possible
• Where will the data come from?
![Page 46: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/46.jpg)
Environmental needs
![Page 47: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/47.jpg)
Responsibilities
• Define major responsibilities– Establishment of test environment– CM– Unit testing
• Putting a name next to a task helps to get it done !!
![Page 48: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/48.jpg)
Responsibilities matrix
![Page 49: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/49.jpg)
Staffing and training needs
• Number and type of people required depend on the scope
• Various training needs– Tools– Methodologies– Management systems (defects)– Basic business knowledge
![Page 50: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/50.jpg)
Schedule
• Testing schedule should be built around milestones in the project plan
• Milestones built around major events
• Initial generic schedule is useful (no dates)
• Plan for risks and contingencies
• Record the schedule (audit trail)
![Page 51: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/51.jpg)
Risks and contingencies
• Planning risks– Unrealistic delivery dates– Staff availability– Budget– Tool inventory– Training needs– Scope of testing– Usage assumptions– Feature creep– Poor quality software
![Page 52: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/52.jpg)
Risks and contingencies
• Contingencies– Reducing the scope of the application– Delaying implementations– Adding resources– Reducing quality processes
![Page 53: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/53.jpg)
Approvals
• Should be the person who can declare the software ready for next stage– Unit test plan (developer)– Acceptance test plan (customer)
• Should be technical or business experts (not managers)
![Page 54: CSE 7314 Software Testing and Reliability Robert Oshana Lectures 5 - 8](https://reader035.vdocuments.net/reader035/viewer/2022062518/568149ff550346895db73162/html5/thumbnails/54.jpg)
CSE 7314Software Testing and Reliability
Robert Oshana
End of Lecture