bof4162 kovalsky

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.1

Insert Picture Here

15 Lessons Learned from15 Years of Quality Control

Jiří KovalskýNetBeans Community Manager, Oracle, Inc.

Glenn HolmerWeb Development Lead, Weyco Group, Inc.

Thomas KruseJava Developer, JUG Muenster


Program Agenda

NetBeans > About 15 Lessons Learned

● Problem

● Solution

● Proof

Q & A


Graphic Section Divider


NetBeans > About

● Integrated Development Environment

● Rich Client Platform

● Plugins

● Community


NetBeans > About

● almost 35 000 000 downloads

● ~4 000 000 lines of code

● 680 code contributors

● since 1998

Number of code contributors

● last issue ID: #236132


NetBeans Experience with Quality Control

You are a lucky bug. I'm seeing that you'll be shipped with the next five releases.


● Recurring problems with lack of time for necessary stabilization at the end of release cycles

● Unplanned slips

# 1 – Quality is feature

Problem

Fig 1. Preview of new java.awt.RealStuffLayout in NetBeans 8.0 GUI Builder


Lesson

● Add quality into feature set and keep that in mind when planning the release

● Get support from developers



NetBeans 7.0 quality didn't get enough

attention right from the start

release date twice moved

longest release ever (10 months)


July 2010:“We're smart. Let's release on 20101213!”

October 2010:“Well, if we send QA to Hawaii for a month

we could release on 20110208!”

November 2010:“Hm, 20110308 is a nice release date

too, isn't it?”NB 4.0 NB 5.5 NB 6.1 NB 6.8 NB 7.10

10

20

30

40

50

rele

ase

len

gth

in w

ee

ks


● How to set the right Release Criteria?

● How to determine whether the product is ready for shipment?

# 2 - Release Criteria

Problem

Fig 2. NetBeans engineers shortly before going on vacation


Lesson

● Listen to your customer

● Find a metric matching satisfaction of the customer

● Base Release Criteria on the metric

● Reevaluate Release Criteria periodically



NetBeans 3.0 – 7.0

● guessed overall bugs targetNetBeans 7.1 – 7.3

● calculated bugs target considering:● historical data ● expected size of changes● planned release dateNetBeans 7.4

● set number of bugs for 10k LOC per areas● agreement from development



Notice: promising trend decreasing difference 6.9 goal changed on the fly 7.0 goal too aggressive

Recommendation index:

NB 6.8 NB 6.9 NB 7.0 NB 7.1 NB 7.2 NB 7.3 NB 7.40

500

1000

1500

2000

2500

3000

3500

4000

4500

Bugs target Opened bugs

NetBeans 7.0 86 %

NetBeans 7.1 91 %

NetBeans 7.2 94 %

NetBeans 7.3 95 %


nu

mb

er

of b

ug

s a

t Co

de

Fre

eze


Lack of structured view on complex data in issue tracking system:

● bug trends● quality status of individual areas● actual distance from Release Criteria● contribution of individuals

# 3 – Metrics and dashboards

Problem

Fig 3. NetBeans managers in the NetBeans Release Control Center


Lesson

Create presentation layer on raw data from your issue tracking system and use it to

● predict progress of stabilization● estimate whether Release Criteria is achievable



● Relying only on absolute bug count is like planning a bike trip on 2D map. If you ignore hills, you will never reach destination on time.

● Meeting Release Criteria before FCS does not mean anything!

# 4 – Income vs. total bug count

Problem

Fig 4. NetBeans bugs desperately trying to escape attention of NetBeans testers


Lesson

● Income of bugs for brand new features is equally important as actual bug count

● Having 10 open bugs at the end of every week however with income of 10 new bugs daily means that product is far from stable

● Release only if you get bug income under control (i.e. 50 % bug count = bug income)



JavaScript Editor in NetBeans 7.3:

week #2: high income → Code Freeze to slip by 2 weeks

week #4: income below 40% → desired quality reached

week #7: income still under control (33%)

1 2 3 4 5 6 70

20

40

60

80

100

120

140

Total Income

week


# o

f bu

gs

in J

ava

Scr

ipt E

dito

r


● Testing on artificial projects & use cases

● Serious problems found after release

● Not covered features and peaks

● Subjective view on new features

# 5 – Engagement of end users

Problem

Fig 5. NetBeans developer fixing serious memory leak in High Resistance mode


Lesson

● Engage real users of your product during its stabilization

● Ask for concrete help and provide your support

● Give appropriate acknowledgement



NetCAT 7.3: 71 tested configurations 116 reviewed tutorials 456 bugs fixed in NetBeans 7.3

Time investment:

Members Reported defects

Productivity Fixed

NetCAT 7.3 112 1363 12,2 43 %

Community 1686 3273 1,9 32 %

147

1490

Invested hoursGained hours



Low quality bug reports

● have low fix rate● consume more time

# 6 – Quality of bug reports

Problem

Fig 6. Main telescopes near Prague for communication on NetBeans mailing lists


Lesson

● Encourage users to follow bug reporting guidelines such as:

● one problem = one bug● meaningful summary● detailed description● steps to reproduce● use attachments

● Try to automate reporting of suitable kind of bugs



#151578 #213130

34 m

on

ths

Day #1: bug reportedDay #1: bug reported

Day #2: not reproducible,more information required

Day #1: not reproducible,more information required

Day #36: silence,bug closed as invalid

Day #2: exact steps to reproduce bug provided

Day #1073: reopened, exact steps to reproduce provided

Day #2: reproduced & reassigned

Day #3: bug fixed

Day #4: bug verifiedDay #1081: reproduced & bug fixed



● Bug reports lack important information

● Difficult & annoying bug submission

● Subjectivity of reporters

● Duplicated bugs slow down stabilization

# 7 – Automated bugs reporting

Problem

Fig 7. Two NetBeans engineers parking their vehicles in NetBeans garage


Lesson

● Automate bugs reporting as much as possible

● Send all relevant information as part of the automatic report

● Automate postprocessing of the reports



1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 20130

5

10

15

20

25

30

35

Automated exception reports All exception reports

Notice:

50 % increase in exception reports since 2009

Comments:

2006 – automated exception reporting introduced

2011 – automated exception reporting further improved


% o

f exc

ep

tion

re

po

rts

in a

ll d

efe

cts


Manual testing:

● not suitable for complex matrix of supported configurations

● boring in longterm projects

Automated testing:● expensive maintenance● stability is never 100%

# 8 – Automated vs. manual testing

Problem

Fig 8. Building machines in full swing trying to compile NetBeans sources


Lesson

● Find the right balance between manual and automated testing

● NetBeans experience:

● Define basic functionality as features used almost by everyone● Develop automated tests only for such basic functionality



NetBeans 3.5 NetBeans 7.3

Sources are compilable

Basic functionality always works

Length of full certification 13 weeks 5 weeks


NetBeans 3.6

decision to invest into test automation

basic functionality in daily builds guaranteed

full certification shortened by 8 weeks


Typical problems of test specifications:

● outofdate instructions● vague or not descriptive statements● wrong granularity/formatting● unknown execution time

# 9 – Quality of test specifications

Problem

Fig 9. Proposal of NetBeans 8.0 splash screen


Lesson

● Assign each test specification its owner

● Review and update test specifications on a release basis by other volunteers

● Measure execution time regularly



NetBeans 6.9:

testing of some areas behind

execution time measured

NetBeans 7.0:

priorities changed to cover important areas primarily

invest into test specificationsNetBeans 7.1 NetBeans 7.2 NetBeans 7.3 NetBeans 7.4

0

50

100

150

200

250

300

3451

71

129

NetCAT team NetBeans QE team

Having quality test specifications pays off greatly


# o

f te

ste

d c

on

figu

ratio

ns


● How to find not tested code?

● Do we have some dead code?

# 10 – Code coverage

Problem

Fig 10. Closeup on broken chip responsible for outages of NetBeans 7.0 daily builds


Lesson

● Evaluation of code coverage results is very expensive

● Use wisely or not at all

● Cheaper way for finding not tested code is to review and update test specifications by experienced users



NetBeans 6.0

measurement and evaluation of actions code coverage took 5 months

results too late for NetBeans 6.1

Tools to find dead code:

FindBugs

PMD

NetBeans 6.0 released NetBeans 6.1 released

Actions code coverage



● Staff is a very expensive tool

● They are lazy to wake up at 2:30 am and test at least basic features in that nice cold lab!

# 11 – Testing infrastructure and tools

Problem

Fig 11. Survival knife every NetBeans engineer is equipped with when leaving office


Lesson

● Don’t reinvent the wheel

● Evaluate existing frameworks and tools, do a cost analysis and implement what suits your needs

● Prefer open source solutions



NetBeans 3.5 own tool but extremely expensive

not maintainable after departure of its author

NetBeans 5.0 attempt to stabilize the tool

expensive, failed

NetBeans 5.5 switch to a different tool

worked for several years

NetBeans 6.0 tool slowly on decline

NetBeans 7.3 switch to open source tool

NetBeans 7.4 satisfaction with the move lasts



Destabilized codebase after merge with major feature developed in parallel on a branch

# 12 – Pre-integration testing

Problem

Fig 12. Dedication of NetBeans test engineers is often admirable


Lesson

● Pretest big changes before their integration to main repository




NetBeans 3.5

first attempt to modernize project infrastructure totally failed

codebase not compilable for 8 weeks

NetBeans 4.0

new Ant based project system preintegration tested

no broken builds at allNetBeans 3.5 NetBeans 4.0

0

1000

2000

3000

4000

5000

6000

Projects bugs All bugs

Impact of 8 weeks of missing build on bugs income


● Lack of intensive communication of development towards quality assurance

● Even relatively small undocumented change can have a big impact on quality and influence testing process

# 13 – Developers proximity

Problem

Fig 13. NetBeans test engineer surrounded by two NetBeans developers


Lesson

● Close communication between development and quality assurance teams especially about:

● future plans● expected code changes● planned integrations into main branch

● Frequency and intensity can vary case by case however AlwaysInformQEAndLetThemDecide approach is a good start which can be reduced later.



Java Projects 54.000 lines of code

developed and tested by colleagues in the same office

very stable functionality

GlassFish support 56.000 lines of code

always developed by distant developer in different location

problems graduated during NetBeans 7.4 release


Java Projects GlassFish support

Lines of code 54 000 56 000

Changed lines of code 17 % 15 %

Income of bugs 68 121

Income of P1/P2 bugs 7 51

Normalized income of P1/P2 1.3 9.1

Income of bugs in NetBeans 7.4 (6 months)


● You have never enough resources, have you?

● So, should highly exposed features be tested more intensively or do all features deserve equal attention?

● Same question applies to new vs. existing functionality

# 14 – Resource balancing

Problem

Fig 14. Wife of NetBeans web master explaining new family model to her husband


Lesson

● No! Testing of new or heavily used features is a must of course. Surprised? :)

● Rule of 3 steps:

● test new or changed code

● test unchanged but highly used code

● test the rest of the code if you still have resources left or let it be



NetBeans 6.8 1 tester for 2 developers

NetBeans 6.9 1 tester for 8 developers

still tried to full test everything

NetBeans 7.0 the same approach didn't work

gave up to test everything

employed community to help cover the missing resources


NB 6.8NB 6.9

NB 7.0NB 7.1

NB 7.2NB 7.3

NB 7.4

0

1000

2000

3000

4000

5000

NB 6.8NB 6.9

NB 7.0NB 7.1

NB 7.2NB 7.3

40

50

60

70

80

satis

fact

ion

with

qu

alit

y in

%#

of b

ug

s a

t Co

de

Fre

eze


● Serious bugs found in product shortly after its release

● How to eliminate them?

# 15 – Go/NoGo includes end users

Problem

Fig 15. NetBeans engineer eagerly completing appraisal of his manager


Lesson

● Engage end users to decision process whether product is ready for release or not

● Side effect: end users will then consider the product as “theirs” and help advertise it



Community Acceptance two weeks long survey

to gather feedback on RC

to identify potential stoppers

to approve FCS

RC – Release Candidate

FCS – First Customer Ship


NB 6.5 NB 6.7 NB 6.8 NB 6.9 NB 7.0 NB 7.1 NB 7.2 NB 7.30

5

10

15

20

25

Bugs nominated as stoppers in CA survey

nu

mb

er

of p

ote

ntia

l sto

pp

ers


Questions & Answers


● How to overcome operational blindness?

● How to control quality before Feature Freeze?

# 16 – Bonus lessons

Problem

Fig 16. Incident which convinced NetBeans team to change their parking system


Lesson – Operational Blindness

Make the work more diverse

● different types of tests● automated testing● tools development / maintenance● shift responsibilities


Motivate testers suitably

● give them responsibility● reward best contributors (gifts, publicity, credit)


Lesson – Quality before Feature Freeze

What did not work:● milestones driven development● 4+1 cycles


What worked:● have a good buffer in your plan● quality (not date) driven releases

1 3 5 7 9 11 13 15 17 19 21 23 25 27 290

100

200300400500

600700800

NetBeans 6.7 NetBeans 7.3

M1 M2 M3

nu

mb

er

of r

ep

ort

ed

bu

gs

No effect of 4+1 development on overall quality


Insert Picture Here

Thank you!

Jiří KovalskýNetBeans Community Manager at Oracle, Inc.

Glenn HolmerWeb Development Lead at Weyco Group, Inc.

Thomas KruseJava Developer, JUG Muenster

bof4162 kovalsky

Technology