Advancing Requirements-Based Testing Models to Reduce Software Defects

Craig Hale, Process Improvement Manager and PresenterMara Brunner, B&M Lead

Mike Rowe, Principal Engineer

Esterline Control Systems - AVISTA

Software Requirements-Based Testing Defect Model

• Focus: requirements-based test (RBT) reviews– Quality imperative, but cost impacts– Large amount of historical data

• Model: defects per review based on number of requirements– Suspected review size a factor– Used for every review– Looked at controllable factors to improve reviews effectiveness

• Stakeholders:– Customers– Project leads and engineers– Baselines and models team

Model Goals

• Improve overall quality of safety-critical systems• Focus on improving review process

– Maximize defect detection rate• Minimize defect escapes

– Reduce defect injection rate• Reduce cost of poor quality

• Defect process performance baselines split– Application type – avionics, medical, etc.– Embedded vs. non– Complexity level

Factors

• 2011 Metrics• 738 reviews over three years • 19,201 requirements• Customers: 10, projects: 21, jobs: 36

• 2012 Metrics• 337 reviews over one year • 2,940 requirements• Customers: 5, projects: 7, jobs: 11

• Y Variables • Number of defects per review (D/R) -

discrete: ratio data type• Defects per requirement (D/Rq) -

continuous: ratio data type

Predicted Outcomes

• Expected defects in the review per number of requirements• Important to understand if exceeding expected defects• Valuable to understand if all defects were detected• Inverse relationship of defects/requirement detected and

review size

Modeling Techniques

• Non-linear regression vs. linear regression vs. power function

• Standard of error estimate varied considerably– Partitioned into nine intervals– Monte Carlo simulation

• Standard of error estimate did not change by more than 0.000001 for ten iterations

• Determined standard of error estimate for each partition

Factors and Correlation Tables

D = DefectsPT = Preparation TimeR = ReviewRq = Requirement

Data Collection: Requirements Count 2011

Data Collection: Partitioning of Reviews 2011

Output from Model 2011

4 Requirements

20 Requirements

Pilot Results 2011

Project Organization

MeanStandard Deviation Mean

Standard Deviation

Review Size -7.17% +209.9% -46.24% -67.62%

Defects Per -13.55% -16.71% -7.09% -15.13%

• Determined to automate model • Needed statistical formula for variance• More guidance on what to do when out of range

Results, Benefits and Challenges

• Points to decreasing variation in defects• Provides early indicator to fix processes and reduce defect

injection rate• Indicates benefits for small reviews and grouping• Challenged with gaining buy-in, training and keeping it simple

Hypothesis Test for Defects/Rqmt and Review Size

Reviews Defects/Rqmt Mean Review Size

June 2011 and Later

mean 0.3898 8.7226

sd 0.9387 24.4248

N 337

May 2011 and Earlier

mean 0.2484 26.4241

sd 1.3168 52.8535

N 738

Hypothesis Testt 2.0061 -7.5102

df 1073 1073

p (2-tailed) < 0.0450 0.0000% Mean Differences 56.89% -66.99%

Potential New Model Element – Years of Experience

• Purpose: Investigate the relationship between a reviewer’s years of experience and the quality of reviews that they perform

• Expected Results: Engineers with more experience would be better reviewers

• Factors: Data studied from 1-Jun-2011 through 25-May-2012 • 337 internal reviews• 11 jobs• 7 projects• 5 different customers

Data Collection: Requirements Count

Data Collection: Defects per Review

Data Collection: Review Prep Time per Review

Data Collection: Review Prep Time per Rqmt per Defect

Potential New Model Element – Years of Experience

• Findings: • Analyzed trend between the independent variable and total

years of experience• The review process showed stability with no significant

impact per years of experience

Summary

• What worked well– Utilizing historical data to predict outcomes– Encouragement of smaller data item reviews– Improving the defect detection rate of data item reviews

• Future plans: Continue to enhance the model – Requirement complexity– Expand lifecycles– Expand activities– Safety criticality