1 a survey of unit testing practices prof. per runeson – lund university per beremark – spin-syd
TRANSCRIPT
1
A Survey of Unit Testing Practices
Prof. Per Runeson – Lund University
Per Beremark – SPIN-syd
2
What is Unit Testing?
“testing of individual software or hardware units or groups of related units” [IEEE610.12]
“Unit testing tests individual software components or a collection of components. Testers define the input domain for the units in question and ignore the rest of the system. Unit testing sometimes requires the construction of throwaway driver code and stubs and is often performed in a debugger” [Whittaker 00]
“a test, executed by the developer in a laboratory environment, that should demonstrate that the program meets the requirements set in the design specification” [Koomen and Pol 00]
3
What Does It Mean In Practice?
1. What do people refer to as unit testing?
2. What are the strengths regarding unit testing in industry?
3. Which are the problems regarding unit testing?
4
Key Findings on Unit Testing
• What?– Technical!– Specified or not?
• How?– Structure-based– Not formally
• Who?– Developer (team)– Not test or QA
• When?– Each
build/day/week– Takes seconds to
hours to run
• Why?– Assuring
functionality
5
Participants:SPIN-syd of Southern Sweden
• 19 companies– Telecom– Automation– Case tools– Information
systems– Banking– Health care– Consulting– Transportation
• Company size– 1-1000
developers
• 17 participants in focus group
• 15 in questionnaire
6
Survey Procedures
Unit testing is…
Unit testing-What-Strengths-Problems
Analysis-What?-How?-Where?-Who?
Q’s
Note
1. Individual reflection
2. Focus group discussion
Note
3. Document and structure
4. Analysis
5. Survey preparation
6. Survey conduct
7. Survey analysis
Q&A’sQ&A’s
Q&A’s
Note
7
3rd party verification
Unit test – smallest specified unit
Test first
Training
Focus Group Meeting
8
Analysis Framework
• What?• How?• Where?• Who?• When?• Why?
Unit test
Unit test
Unit test
Unit test
Unit test
Unit test
Unit test
Unit test
9
Questionnaire
10
Analysis
0
2
4
6
8
10
12
14
1. S
mall
est u
nit
2. T
echnica
l
3. F
unctio
ns
4. S
epar
ate
5. S
truct
ural
6. C
over
age
7. A
utom
ated
exec
ution
8. A
utom
ated
follo
w-up
9. S
pecif
ied in
text
10. S
pecifie
d in
test
code
11. E
xecu
ted
by d
evelop
ers
12. D
evelo
pers/t
ester
s
13. T
est d
epartm
ent
14. Q
uality
dep
artm
ent
15. F
or e
ach
build
16. M
any t
imes
eac
h da
y
17. A
t lea
st da
ily
18. A
t lea
st wee
kly
19. S
econds
20. M
inutes
21. H
ours
22. U
nit fu
nction
s
23. A
ccept
s unit
s
24. S
pecify
unit
25. Im
prov
e qua
lity
26. C
usto
mer
req's
Fully agree
Agree
Neutral
Disagree
Totally disagree
N/A
What? Time?When?Who decides?Who conducts?How? Why?
11
What?
• Definition– Test of smallest unit or units
• Strength– Unit identification– Test of surrounding modules
• Weaknesses– GUI test– Unit identification– Test scripts and harness maintenance– Data structures
12
How?
• Definition– Structure-based– Preferably automated
• Strength– Test framework
• Weaknesses– Documentation– Framework tailoring– Test selection– Test metrics
13
Who?
• Definition– By developer
• Strength– Independent test– Competence network
• Weaknesses– Competency– Independence– Introduction strategy
14
When?
• Definition– Quick feedback
• Strength– Continuous regression test
• Weaknesses– Stopping criteria
15
Why?
• Definition– Ensure functionality
• Strength– External requirement (safety)– Agile methods
• Weaknesses– Cost vs. value
16
Further Use of the Survey
• Clarify unit test definition in your company
• Lightweight assessment of unit testing practices– Benchmark– Improve
17
Read more… … in IEEE Software July/August 2006
18
http://serg.telecom.lth.se
19
Definitions
0
2
4
6
8
10
12
14
1. S
mall
est u
nit
2. T
echnica
l
3. F
unctio
ns
4. S
epar
ate
5. S
truct
ural
6. C
over
age
7. A
utom
ated
exec
ution
8. A
utom
ated
follo
w-up
9. S
pecif
ied in
text
10. S
pecifie
d in
test
code
11. E
xecu
ted
by d
evelop
ers
12. D
evelo
pers/t
ester
s
13. T
est d
epartm
ent
14. Q
uality
dep
artm
ent
15. F
or e
ach
build
16. M
any t
imes
eac
h da
y
17. A
t lea
st da
ily
18. A
t lea
st wee
kly
19. S
econds
20. M
inutes
21. H
ours
22. U
nit fu
nction
s
23. A
ccept
s unit
s
24. S
pecify
unit
25. Im
prov
e qua
lity
26. C
usto
mer
req's
Fully agree
Agree
Neutral
Disagree
Totally disagree
N/A
What? Time?When?Who decides?Who conducts?How? Why?
20
Strengths and weaknesses
0
2
4
6
8
10
12
1. Id
entify
units
2. G
UI com
ponen
ts
3. D
ata
depe
nden
cy
4. E
xtern
al co
de
5. R
eal-t
ime
6. T
est ca
se se
lectio
n
7. M
aintena
nce
8. S
pecif
y test
case
s
9. A
utom
ate
testi
ng
10. B
uild/ta
ilor f
ram
ewor
ks
11. B
uild sy
stem
s
12. C
M sy
stem
s
13. T
rouble
repo
rting
14. D
ocum
enta
tion
15. S
caffo
lding
16. C
over
age
17. D
evelo
pers e
xecu
te a
ll
18. E
xter
nal u
nits
19. P
riorit
y and
status
20. C
ompe
tenc
e and
skills
21. F
reque
ncy
22. C
riter
ia
23. M
otiva
tion
24. G
ain a
waren
ess
Very goodGoodNeutralBadVery badN/A
What? When?How? Why?Who?