testing cs446 gw 1 testing -- cs446 gio wiederhold fall 1999 testing only proves the presence of...
Post on 19-Dec-2015
214 views
TRANSCRIPT
Testing CS446 GW 1
Testing -- CS446
Gio Wiederhold
Fall 1999
Testing only proves the presence of errors, not their absence [Edsger Dijkstra]
Testing CS446 GW 2
Objective ?
• Assure that program executes?
• Assure that it provides results?
• Assure that the results look good ?
• Assure that the results are correct wrt– expectations ?– specifications ?– customer expectations ?– interfaces to other programs ?
Testing CS446 GW 3
IMPORTANCE ?
• Fraction of development cost - Significant ?
• Objectives -- relevant ?
• Setting to do testing -- simple/complex?
Testing CS446 GW 4
IMPORTANCE ?• Fraction of development cost = 40%
– more for real-time programs -- why?– prior to delivery, % in Maintenance?
• Objectives -- conflicting relevance+ show program correctness - confidence– find errors to debug - actual approach
• Setting to do testing -- complex requirements – Fixed environment -- reproducible / observable– Inputs -- all values / extreme+branch values / timings– Outputs -- values / performance metrics
• customer expectations / analytical predictions / • past values
Testing CS446 GW 5
Extent of testing• Exhaustive
– every combination of all paths– all specified / possible input values– all times for real time
• Complete– all paths at least once, each branchpoint– all extreme values, all branch values– typical, high rates, low rates
• Affordable– assess value -- find errors vs. correct paths
P r o
b l e m
s ?
Testing CS446 GW 6
Frequency ?
• At delivery of integrated system– likely to be a disaster
• At completion of module– definition of module,– requires environment or scaffolding
• When programmers have time
• When prior regression cycle completed
• Daily build– regression longer than cycle
Testing CS446 GW 7
Types of Testing
• Unit test / component test– requires scaffolding
• Subsystem / integration test – requires service specifications
• System test / delivery test– requires copy of customer environment and data
• also ( orthogonal ) :– feature testing– performance testing– load (to maximum specification) testing– stress (to failure) testing– stability testing versus environments and input sets– reliability testing customer reports?
Testing CS446 GW 8
Regression test
After any / significant changes / updates• verify that all old stuff still works• use collection of all / most prior tests and
their results• automate comparison of results
differences found are intentional gooddifferences found are insignificant acceptabledifferences found are annoying post on error reportdifferences found are errors to be fixed error report
• act on error reports detail
Testing CS446 GW 9
Decision process
classify
cost of fix
prioritize
reallocatecurrent work
schedule into current work
ignore(but still log!)
schedule for next release
error log__ __ __ ___ __ __ ___ __ __ ___ __ __ _
cost outcome
experienceError reports
next regression test
logging
1.
5a,b,c,d.
4.
3.
6.
7.
2.
product testingregression testcustomer
product insertion
Testing CS446 GW 10
Decision process: 1,2
Error report
1. check against log: is it old or new ?
2. classify as to seriousness – critical other stuff fails– important in specification / contract– unacceptable seriously impacts usability– limited affects non-essential function– minor easily bypassed or ignored
Testing CS446 GW 11
Decision process: 3, 4
Error category
3. estimate cost – architectural change try to workaround or defer
– module interface need multiple assessments, as below
– released module cost of fix and dissemination
– completed, non-released module re-assignment cost
– module in-process cost of partial redo
4. prioritize – by combining seriousness & cost
phase
cost[Boehm]
phase
a
i
rcp 10
n*
30
100?
1
Testing CS446 GW 12
Decision process: 5, 6, 7
Repair action
5. schedule appropriately – get specific schedule– buy-in, estimate from staff
6. monitor progress– actual start-date, staff allocation– log actual costs & time up to insertion
7. perform regression tests
Testing CS446 GW 13
When is testing done?• When no bugs remain?
Asymptote:– measure rate of <new> bug reports
– correct for state: , number of customers
• When the customer has no complaints?– specifications versus expectations– complaints are good ?!
Indicates that product is useful !
How to be reimbursed?
time
time
What does it mean if no errors are found?
Testing CS446 GW 14
Value of testing over time?• Rate of error discovery
• Tests per error found
• Difficulty of errortype
• Combination: – cost/bug found=The Pesticide Paradox [Beizer:83]
time
time
time
time
Testing CS446 GW 15
White box / Black box
Internals are Visible• tester understands code• amount of code is
managable• tests are based on internal
conditions
early, unit, academic?
Internals are Hidden• tester understands function,
customer needs
• codes can be large, composed
• tests based on specifications
integration, final, business
Internals are Abstracted• test plan is formalized• code can be substantial
Pre Integration, reuse metrics
Grey box?
Testing CS446 GW 16
Specifications
• contractual– negotiated by customer / supplier– size of database in terms of employees
• derived– created during software design – size of database in terms of records
• implicit– left to insight of tester– interface / input performance at full scale
Testing CS446 GW 17
Implicit specifications
Example: Dialog design advice [Shneiderman] dont1. Strive for consistency FTP login - user2. Enable shortcuts for experts deep menu trees3. Offer informative feedback illegal operation4. Design dialogs to yield closure FTP bin mode5. Offer simple error handling editable form6. Permit easy reversal of actions log, ctrl Z7. Support internal locus of control no ptr to home page8. Reduce short-term memory load > 72 items
Assignment? List violations from your experience
Testing CS446 GW 18
Bug types• assumptions
– implit expectations • conflicts
– features vs usability vs speed -- balance
• algorithm– publications often don’t state limits– only report good aspect -- concurrency control
• overflow– arrays, files, record types -- language dependent
• leaks in memory allocation– language/executable dependent, tools available
Testing CS446 GW 19
Testability
What makes programs hard to test?• Boundary values hard to enter• Coverage hard to assess• Design for testability
– internal breakpoints for inputs at branches – reports -- voluminous -- analysis tools– switches to bypass healthy areas (WB / BB)
– temporary/permanent breakpoints/switches
• Path to report mapping
Testing CS446 GW 20
Alternatives• Verification
– objective: versus formal specification– only performed after specifications are fixed
• Validation– subjective: versus customer intent– implicit: expectations from the state-of-the-art– can be performed thoughout
• Replication of coding and comparison– used where no realistic tests can be made – costly separation of staff, – still specification dependence, questionable
• Code review ‘Static testing’
Testing CS446 GW 21
Conclusion• Planning for Testing is worthwhile
– explicit cost item, relate to quality objective– distinct, but related to debugging
• Logging is crucial– input to next test, further development
• Requires knowledge of code, function– experience with programs, customer needs– can qualified testers be obtained, retained
• Testing receives insufficient attention