thomas l. gilchrist testing basics set 5: cm & peer reviews by thomas l. gilchrist csqe,csqa...

Thomas L. Gilchrist [email protected]

Testing BasicsSet 5: CM & Peer Reviews

By

Thomas L. GilchristCSQE,CSQA

2009


SCCMSoftware

Change & Configuration Management


SCCM (software change and configuration management) Basics

• Identification: How does and organization identify and manage the many existing versions of a program and its documentation in a manner that will enable change to be accommodated efficiently?

• Control: How does an organization control changes before and after the software (and supporting documentation) is released to the customer?

• Status Accounting: Who has responsibility for approving and ranking changes

• Verification: How can we ensure that changes have been made properly?

• Reporting: What mechanism is used to appraise others of changes that have been made (or being made).

Roger Pressman…”Software Engineering, Fifth Edition”


CM High Level View

Project CM Repository

SW EngTasks

NewSCI

TechReview

ApprovedSCI

SCMControls

SCI = Software Configuration Item


CM High Level View


SW EngTasks

NewSCI

TechReview

ApprovedSCI

SCMControls Read

OnlyCopyVer

1SCI

SCI = Software Configuration Item

Ver1

SCI


TechReview


CM High Level View


SCMControls

RequestFor

Change

Read OnlyCopyVer

1Ver1

SCI Ver1

SCI



CM High Level View


SW EngTasks

CurrentBaseline SCIApproved to

Change

SCMControls

RequestFor

Change

Read OnlyCopyVer

1Ver1

SCI Ver1

SCI



CM High Level View


SW EngTasks

SCMControls

RequestFor

Change

Read OnlyCopyVer

1Ver1

SCI Ver1

SCI


Change



CM High Level View


SW EngTasks

ModifiedSCI

TechReview

SCMControls

SCMControls

RequestFor

Change

Read OnlyCopyVer

1

ApprovedSCI

Ver1

SCI Ver1

SCI


Change


TechReview


CM High Level View


SW EngTasks

TechReview

SCMControls

SCMControls

RequestFor

Change

Read OnlyCopy

ModifiedSCI

ApprovedSCI

?

Ver2

SCI


Change


TechReview


Peer Reviews


Reviews Vs. Testing

• Dynamic Vs. Static Testing• Peer Reviews can be done

before code can be tested.• Testing evaluates product while

performing function in target environment.

• Not mutually exclusive


Peer Reviews

• Intended as a method of static verification for software by the producers’ peers to identify defects and areas where changes are needed.

• Method is not intended to replace dynamic testing, but rather to augment it.


What Peer Review Is Not

• A “silver bullet.”• A tool for personnel evaluation.• A quality assurance program.


Static Verification Techniques

• Desk Checking• Walkthroughs• Formal Technical Reviews• Inspection (Fagan)• Management Reviews• Audits• Static Program Analysis• Formal (Mathematical) Verification


Peer Review Methods

Walkthroughs

Methods Typical Goals Typical Attributes

• Minimal overhead• Developer training• Quick turnaround

•Little/no preparation•Informal process•Meetings•No measurement•Not Inspection!

Inspections•Detect and report all defects efficiently and effectively.

•Formal process•Checklists•Structured Meetings•Measurements•Verify phase

Desk Checks• Minimal overhead• Quick turnaround

•Little/no preparation•Informal process•No measurement•No Meetings•Not Inspection!

Copyright © 1998 Philip M. Johnsonwww.ics-hawaii.edu/~johnsonused with permission


Initial Level

InformalChecking

DoWork

Down-stream

Customers Re-Work

InputSource

Materials

ErrorsFound

ProcessProduct

EntryCriteriaLooselyDefined

ExitCriteriaForQualityNotDefined

Work Process

ENTRY

EXIT


Software Inspections


Measure and Improve Product Quality

DoWork

Down-stream

CustomersRe-

Work

ErrorsFound

ProcessProduct

CrossfunctionalChecklists

SoftwareInspection

InputSource

Materials

EntryCriteriaLooselyDefined

ExitCriteriaForQualityDefined

ENTRY

EXIT

Work Process


Improve Product Quality and Reduce Variation

DoWork

Down-stream

CustomersRe-

Work

ErrorsFound

ProcessProduct


InputSource

Materials

EntryCriteriaDefined

ExitCriteriaForQualityDefined

ENTRY

EXIT

SoftwareInspection

Work Process


To Improve Product AND Process

(EntryCriteria

Met)

DoWork

Down-stream

CustomersRe-

Work

InputSource

Materials

ErrorsFound

ProcessProduct


ExitCriteriaMet

SoftwareInspection

Repeatable Work Process



(EntryCriteria

Met)

DoWork

Down-stream

CustomersRe-

Work

InputSource

Materials

ErrorsFound

ProcessProduct


ExitCriteriaMet

Statistical DataAnalysis and

Causal Analysis

SoftwareInspection




(EntryCriteria

Met)

DoWork

Down-stream

CustomersRe-

Work

InputSource

Materials

ErrorsFound

ProcessProduct


ExitCriteriaMet

Statistical DataAnalysis and

Causal Analysis

PD

CA

DemingCycle

(Defect Prevention Process)

SoftwareInspection



Inspection Characteristics• Works for any kind of technical material -

plans, specifications, reports, design representations, code, etc.

• Structured, rigorous, repeatable method of crossfunctional review.

• Works in "low process maturity" environments: Product Improvement

• Works in "high process maturity" environments: Process Improvement

• Indicated when the material is important


Software Inspections

Software Inspection is a method for the improvement of technical materials with an eye to the identification and reporting of defects as early as possible.

50


Swimlane Process Tool

ProcessParticipant 1


Process AreaProcess

Participant 3Process

Participant 4

Task

Tasks that span multilple participants

Meeting

Report or Document

Y

N Decision?

Milestone

Another Task

The Team

The Process

Flow

Solid flow lines go down. Dashed flow lines go up.

R

TeamFlow Process Tool




Process AreaProcess

Participant 3Process

Participant 4

Task

Tasks that span multilple participants

Meeting

Report or Document

Y

N Decision?

Milestone

Another Task

Swimlane Process Tool


Inspection Process Flow

See Handout

SW ProcessOwner & Manager

AuthoringGroup

Scribe Moderator Author Representative(1-reviewer)

Downstream Customers(2-5 reviewers)

Inspection Team Members

Sample Checklists

High Level Documents and Standards

Finished Product, Ready for Inspection

Planning

Y

NReady to Inspect?

Inspection Materials- Crossfunctional Checklist- Target Document, Chunked & Zoned- Forms & Instructions- Logistics-High Level Documents

Overview Meeting

Inspection Materials and Instructions

Indiv idual Preparation

Errors & Questions

Error Logging Meeting

Error Logs and Meeting Summary

Y

NAnnother ChunK?

Address Errors

Addressed Product

Verify & Approve

Y

NReady to Exit Process?

Verif iedProduct

InspectionData


Entry/Exit Materials

Checklists– A list of criteria to facilitate the identification of major

errors.– Crossfunctional in nature because contents of deliverable

must address the entry-level materials as well as satisfy all the downstream customers.

High-level Materials– The source materials entered into the process in which

the deliverable was produced.

Standards– Rules and / or guidelines established to improve the

quality of a broad range of customer sets.


Inspection Materials

InspectionProcess

Target Document



InspectionProcess

Checklists

Target Document



InspectionProcess

High Level Documents

Checklists

Target Document



High Level DocumentsStandards

Checklists

Target Document InspectionProcess



High Level DocumentsStandards

Checklists

Target Document InspectionProcess

Metrics


Inspection Process

[process owner]

• Knowledge and expertise in software inspections

• Expertise in finding errors which will cause future rework in the deliverable being inspected.

Inspectors. (“Users" or “customers” of thesoftware programelement (product)being inspected.)

Author orauthoring grouprepresentative.

• Knowledge and expertise in developing high quality software.

Trained Moderator • Deliverable with all known errors removed.

• Informed author(s)

Software processowner.

All individuals whoare affected bythe quality of theinspecteddeliverable

• Inspection metrics• Error logs• Inspection checklists• Problem reports for

high level documents and standards

Software Inspection

ver3

1 2

ProjectManagement

AuthoringGroup Moderator Author

Representative Reader DownstreamCustomers Scribe

Inspection Team Members

Sample Checklists

High Level Documents and Standards

Produce Product Specific Checklists

High Level Documents

Product Specific Checklist

Produce Product

Finished Product

Planning


N

YReady to Inspect

Overview Meeting

Inspection Materials and Instructions

Individual Preparation

Errors & Questions


Error Logs and Meeting Summary

N

YAnnother ChunK?

Address Errors

Addressed Product

Verify & Approve

N

YReady to Exit Process?

VerifiedProduct

InspectionData

[A]

[A,B,C]

[B,C]

Gilchrist 3/95

Process Name:

Suppliers Inputs Process Flow Outputs Customers

Process Owner: Date/Ver:Phone: M/S:

Process ID: Page: of

See Handout


Inspection Process

Software Inspection

Has a moderator/team leader been identified and formally trained in the SW inspection technique?

Has all exit criteriadescription (process) been satisfied?

Have the necessary upstream documents and standards been identified and are they available?

Is the author confident that the deliverable is of high quality (he knows of no errors in the deliverable to be inspected).

Are there qualified people available to serve on the inspection team?

Does the authoring group supervision and management aware that they will not be able to participate in the inspection logging meetings (SW inspections are not to be used for evaluating people)?

Is there the possibility that there are major errors? Is there a written process and rules for conducting the

software inspection?

Did the inspection team spend sufficient time finding major errors?

Did the inspection team spend sufficientreporting and logging errors found (but not more that 2 hours per session)?

Were concerns and questions documented? Was an individual or resource identified as a contact

point for each question and/or issue? Were action items documented and assigned? Was the current health of the inspection process

indicated? Have Action Items been resolved? Have all errors been addressed by the author? Has the inspection team decided on the disposition

of the inspected deliverable (release or re-inspect)?

No meeting will last over 2 hours. Checklist will be used to stimulate The emphasis is on finding and reporting major errors. One and only one person from the authoring group is on the inspection team. Focus is on team effort Inspection team management/supervision does not participate in the inspection processes. A trained team leader (moderator) is used to facilitate the process and the meetings. Team members are trained and assigned specific “roles”. Rationale (i.e. Error detection is maximized by using optimum material coverage (i.e.

chunk). If there are more than 10-15 type written pages to be inspected, then the inspection will be divided into “chunks”. The number of major and minor errors, number of questions/concerns, the number of pages (or LOC) inspected

number of team members, and the time spent finding, reporting, and addressing errors will be collected for each chunk.

Documented inspection rules are followed.

To find and remove errors which would cost substantiallymigrate to downstream. To create a quality baseline measure for process improvement.

A Process checks of meetings.B Inspection process key indicators.C Product/process quality.

The authoring group considers all process exit criteria met (the deliverable being produced is ready to be used for it’s intended purpose).

22Ver 3

Gilchrist 3/95

Entry Checklists Exit Checklists

Measures

Triggers

Continued on page:

Process Objectives

Process Scope

Continued on page: Continued on page: Continued on page:

Continued on page:

Continued on page:

Process ID: Page: ofProcess Name:

Rules/Standards Continued on page:

See Handout


Software Inspection Entry Criteria Has a moderator/team leader been

identified and formally trained in the SW inspection technique?

Has all exit criteria found in the activity task description (process) been satisfied?

Have the necessary upstream documents and standards been identified and are they available?

Is the author confident that the deliverable is of high quality (he knows of no errors in the deliverable to be inspected).


Are there qualified people available to serve on the inspection team?

Does the authoring group supervision and management aware that they will not be able to participate in the inspection logging meetings (SW inspections are not to be used for evaluating people)?

Is there the possibility that there are major errors?

Is there a written process and rules for conducting the software inspection?

Software Inspection Entry Criteria


Software Inspection Exit Criteria

Did the inspection team spend sufficient time finding major errors?

Did the inspection team spend sufficient time reporting and logging errors found (but not more that 2 hours per session)?

Were concerns and questions documented? Was an individual or resource identified as a

contact point for each question and/or issue?

Were action items documented and assigned?


Was the current health of the inspection process indicated?

Have Action Items been resolved? Have all errors been addressed by the

author? Has the inspection team decided on the

disposition of the inspected deliverable (release or re-inspect)?

Software Inspection Exit Criteria


Basic Rules of Software Inspection No meeting will last over 2 hours. Checklist will be used to stimulate error

detection. The emphasis is on finding and reporting

major errors. One and only one person from the

authoring group is on the inspection team.

Focus is on team effort in assisting the author to improve product (attacking the author is not permitted).


Basic Rules of Software Inspection Inspection team management/supervision

does not participate in the inspection processes (inspections are not used to evaluate personnel).

A trained team leader (moderator) is used to facilitate the process and the meetings.

Team members are trained and assigned specific “roles”.

Rationale (i.e.,design/style) is not questioned or discussed during the logging meetings.


Basic Rules of Software Inspection Error detection is maximized by using

optimum material coverage (i.e.,10-15 typewritten pages per 2 hour chunk).

The number of major and minor errors, number of questions/concerns, the number of pages (or LOC) inspected, number of team members, and the time spent finding, reporting, and addressing errors will be collected for each chunk.

Documented inspection rules are followed.


If there are more than 10-15 type written pages to be inspected, then the inspection will be divided into “chunks”.

Basic Rules of Software Inspection


Roles and Responsibilities• Moderator• Author/Developer/Engineer/Producer• Inspector(s)

– Key inspector: if absent or unprepared, inspection meeting will be rescheduled.

– Optional inspector: if absent or unprepared, inspection meeting will be held as scheduled.

• Scribe• Reader

– Key inspector

Supervisors of authors are specifically excluded from participating in inspections but are kept informed of

results.


Overview Meeting

• Distribution of Target Document

• Distribution of Checklists, Standards, High Level Documents

• Assignment and Clarification of Roles

• Special Instructions

• Understand Expectations of Moderator/Process

• Acceptance of Materials

• Agreement to Proceed

Moderator Team


Individual Preparation

Moderator Team

• Available for Questions

• Self-Study of Documentation (Up to 2 hours)

• Identify and Describe Errors and Questions (Individually)



Moderator/Scribe* Team• Record Prep Times*• Facilitate Reporting of Errors• Log Errors/Questions*• "Playback" Error Log*• Record Logging Times*• Determine Disposition of Errors• Kickoff Next Chunk• Process Check*

• Report Prep Time

• Report and Classify Errors

• Build on Synergy

• Process Feedback