What is “quality”?

• IEEE Glossary: Degree to which a system, component, or process meets (1) specified requirements, and (2) customer or user needs or expectations

• ISO: the totality of features and characteristics of a product or service that bear on its ability to satisfy specified or implied needs

“Set of systematic activities providing evidence of the ability of the software process to produce a software product that is fit to use”◦ G. Schulmeyer and J. McManus, Software

Quality Handbook, Prentice Hall, 1998.

The metric used most often to measure software quality assurance is errors found/KLOC.

An alternate view of Quality:◦ is not absolute◦ is multidimensional, can be difficult to quantify ◦ has aspects that are not easy to measure◦ assessment is subject to constraints (e.g.,

cost)◦ is about acceptable compromises◦ criteria are not independent, can conflict

In other words, “quality” is not a well defined term.

Quality Criteria include:◦ correctness

◦ efficiency◦ flexibility◦ integrity◦ interoperability◦ maintainability◦ portability◦ reliability◦ reusability◦ testability◦ usability

You can add some of your own criteria:

Cool-ability

Crash-ability

Where-ability(Where you got the software)

Who-ability(Who you copied it from)

• Definition: Monitoring processes and products throughout the software development lifecycle to ensure the quality of the delivered product(s)

• Monitoring the processes– Provides management with objective

feedback regarding process compliance to approved plans, procedures, standards, and analyses

Monitoring the products◦ Focus on the quality of product within each

phase of the Software Design Lifecycle e.g., requirements, test plan, architecture, etc.

◦ Objective: identify and remove defects throughout the lifecycle, as early as possible

SQA processes apply when integrating purchased or customer-supplied software products into the developed product

• Question. How do you determine the “quality” of COTS components?– Current research problem

Use of standards and process models has a positive impact on the quality of the software product.

• Examples include:– ISO 9001 (www.iso.org)

– CMM (http://www.sei.cmu.edu/cmmi/)• CMU SEI, 5 levels

– SPICE (http://www.sqi.gu.edu.au/spice/what.html)• Developing a standard for software process assessment

• ISO joint committee, Europe, Australia

– IEEE 1074, IEEE 12207 http://www.techstreet.com/cgi-bin/detail?doc_no=ieee|1074_2006&product_id=1277365

How can we prove this is true?

Product assessment can include:◦ Project management reports, quality assurance

reports, training plans, test plan(s)

◦ Requirements, analysis, architecture, detailed design model, test cases

◦ Issue or problem reports

◦ Metric reports

◦ Traceability reports (http://www.ldra.com/tbreq.asp)

◦ Documentation, coding standards

Inspection:

◦ A formal evaluation technique in which an artifact (e.g., software requirements, design, or code) is examined in detail by a person or group other than the originator

◦ detect faults, violations of development standards, and other problems.

◦ review members are peers (equals) of the designer or programmer.

◦ data is collected during inspections for later analysis and to assist in future inspections.

Describes the practices and procedures to be followed for reporting, tracking, and resolving problems

◦ Who can report a problem?

◦ How is it reported?

◦ How is it tracked?

◦ Who determines if it is a problem that going to be resolved?

◦ How is it assigned for resolution?

◦ How does the person indicate it has been corrected?

◦ Who reviews it to determine if it can be closed?

• Problems can be product or process related– e.g. incorrect requirement, incomplete

class definition, code defect, ambiguous description in user documentation, process to review detailed design is not clearly defined, etc.

Metrics for each artifact (e.g., Requirements)

◦ Number of requirements

◦ Number of changes per requirement Called “churn” rate

◦ Characterization of defects Not testable, ambiguous, inconsistent, incorrect, incomplete

redundant, infeasible, … Major or minor defect Phase defect detected (which phase of SE cycle) Cost to fix

Indicator Category Management Insight Metrics

Progress Provides information on how well the project is performing with respect to its schedule.

Actual vs. planned task completions Actual vs. planned durations

Effort

Provides visibility into thecontributions of staffing onproject costs, schedule adherence, and product quality.

Actual vs. planned staffing profiles

Cost Provides tracking of actual costs against estimated costs and predicts future costs.

Actual vs. planned costs Cost and schedule variances

Review Results Provides status of action items from life-cycle review.

Status of action items

Trouble ReportsProvides insight into product and process quality and the effectiveness of the testing.

Status of trouble reports Number of trouble reports opened, closed, etc. duringreporting period

Requirements Stability Provides visibility into the magnitude and impact of requirements changes.

Number of requirements changes/clarifications Distribution of requirements over releases

Size Stability

Provides insight into the completeness and stabilityof the requirements and into the ability of the staff to complete the project within the current budget and schedule.

Size growth Distribution of size over releases

Computer Resource Utilization Provides information on how well the project is meeting its computer resource utilization goals/requirements.

Actual vs. planned profiles of computer resource utilization

Training Provides information on the training program and staff skills.

Actual vs. planned number of personnel attending classes

Media Control involves how you store your artifacts.

Identify the media for each intermediate and deliverable artifact

Documentation required to store the media, including the backup and restore process

Protect computer program physical media from:◦ unauthorized access◦ inadvertent damage◦ degradation

Post-Mortem Anaylsis (PMA)Definition: The process of looking back at a completed

project's design and its development process, in order to identify those aspects where improvements can be made in future projects

Post-mortems enable individual learning to be converted into team and organizational learning.

Other Names for PMAPMA, or its main, activities have many terms such

as: blame and flames debriefing lessons learned post implementation review post project review postpartum project audit project review retrospective team retrospective.

Post-Mortem Analysis PMA is ideally performed either soon after

the most important milestones and events or at the end of a project, both in successful and unsuccessful software development projects.

The benefit is that post-mortems can often reveal findings more frequently and differently than project completion reports alone.

Benefits Helps project team members share and

understand each other’s perspectives; Integrates individual and team learning; Identifies hidden problems; Documents good practices and problems (so

as not to repeat bad practices); Increases job satisfaction by giving people

feedback about their work;

Aim of a PMAThe aim of any post-mortem is to provide

answers to the following four key questions : What did we do well that does not have to

be further discussed (the positive aspects)? What did we learn (the negative aspects)? What should we do differently the next time

(the negative aspects which require improvements)?

What still puzzles us (share the things that are confusing or do not make sense)?

Conducting a PMAThe PMA process for average-sized and large

projects is defined in five steps: 1. Plan a project review 2. Gather project data 3. Hold a post-iteration workshop or post-

mortem review 4. Analyze the findings and synthesize the

lessons learned 5. Publication of the results

Plan a Project Review 1. A project review is planned to identify the

most suitable methods and tools used in the other steps. ◦ The post-mortem reviews, the reasons for the

review, the focus and the participants are defined

Gather Project Data 2. Both objective and subjective data are

collected from all the project participants via pre-defined metrics, surveys, debriefings, etc. to identify the useful information for the “following step” (workshop/review)

Hold a Post-Mortem Review 3. A “project history day” is the most

important step, and it is held to combine reflective analysis of project events with the actual project data after a project’s major milestone (post-iteration), or after a project has finished (post-mortem).

◦ In the case of large projects , only a few key people participate in this session

Analyze and Synthesize 4. The findings are analyzed, prioritized and

synthesized as lessons learned.

◦ This is often started during the project review day after identifying and prioritizing the positive events and problems

Publish the Results 5. The summary of the findings is published

and presented in a way that enables future projects to know what processes or tools are important to continue, and to turn problems into improvement activities.

Causes of Failure We don’t look back until the very end of the project.

We don’t set a tone of open and honest feedback.

We don’t look at the whole picture of product and process.

We don’t actually follow through on the feedback.

A Sample Post-Mortem Form http://www.klariti.com/technical-writing/Post-

Mortem-Documentation.shtml

Fagan, M., “Design and Code Inspections to Reduce Errors in Program Development”, IBM Systems Journal, 15, 3 (1976), pp. 182-211

Fagan, M., “Advances in Software Inspections”, IEEE Transactions on Software Engineering, 12, 7(July 1986), pp. 744-751

Schulmeyer G. and McManus, J., Software Quality Handbook, Prentice Hall, 1998.

IEEE Std 730™ 2002, IEEE Standard for Software Quality Assurance Plans, IEEE Computer Society, Sponsored by the Software Engineering Standards Committee

Rosenberg, L.H.; Gallo, A.M., Jr., “Software quality assurance engineering at NASA”, Proceedings of the IEEE Aerospace Conference, 2002, Volume: 5 , 2002, pp. 5-2569 -5-2575.

“Inspections”, http://www.math.uaa.alaska.edu/~afkjm/cs470/handouts/inspections.pdf