software process audits

8/7/2019 software process audits

1/22

by

Tom Gilchrist, CSQA, CSQE,

Software Process Reviews/AuditsSoftware Process Reviews/Audits

Process OverviewProcess Overview


2/22

SASQAG 10/17/2002 [email protected] 2

Before we start

SQA Context

Overview of SW Audit Process

SW Audit Examples

Information in this presentation aremy opinions and not necessary

those of my employer.


3/22


Some Terms/Ideas

Process

Deterministic vs. Non

Deterministic

Quality vs. Value


4/22


Software Quality Assurance

Check software products and processes to verifythat they comply with the applicable procedures andstandards. (Process Reviews or Audits)

Review and measure the quality of software

products and processes throughout development.(Dynamic & Static Testing)

Provide software project management (and otherappropriate parties) with the results of reviews andprocess checks.

Work with the software project during early stages toestablish plans, standards, and procedures to keeperrors from occurring in the first place.


5/22


Formal Definition

Audits provide an independent evaluation ofsoftware products or processes to ascertain

compliance to standards, specifications, and

procedures based on objective criteria that included

documents that specify:

The form or content of the product to be

produced

The process by which the products shall be

produced

How compliance to standards or guidelines

shall be measured.

IEEE STD 1028, (1988)


6/22


Audit Types

First Party AuditFirst Party Audit

Within you company or organization

Second Party AuditSecond Party Audit

Sometimes called external audits

By a Customer on his Supplier

By a Supplier on you.

Third Party AuditThird Party Audit Outside third party is contracted to do

the audit.


7/22


Audit/Process Review Principles

Conducted by individuals who areorganizationally independent of thedevelopers.

Begin early in the requirements phase andcontinue throughout the developmentprocess.

Professionally planned, conducted anddocumented.

Follow-up on corrective action.

Project Management is involved in the Auditprocess and is responsible for rework andprocess improvements.


8/22


What Software Audit Should Do

Determine:

Compliance to requirements

Conformance to plans, policies, procedures, and

standards

Drive process improvement based on:

Adequacy of plans, policies, procedures, and

standards

Effectiveness and efficiency of plans, policies,

procedures, and standards

Assess personnel familiarity to requirements and

documentation

Assure availability, use and adherence to software

standards


9/22


What Triggers an Audit?

Quality Assurance Plan Event

Date

Requests from management

Requests from developers Requests from customers

Integration with process improvementactivities

Outside requirements regulatory

Gut feel


10/22


Scope: Requirements, Time, and Target

Audit

Target

External

Standards

Organizational

Procedures and

Methods

Spread around

organization

Cover all functions and

activities

Try to hit things early

Move towards process

audits


11/22


Process Review/Audit Process

OK

PrepareAudit

Developers Project ManagerAuditor

ConductAudit

Write-upReport &Findings

Follow-upAudit

Re-Work

Findings?

NO

YES

CloseoutAudit &File END

Reviewwith

Manager

Plan(Requirements,

Scope, & Checklist)Start

CorrectiveActions


12/22


Identify Requirements

Policies/Standards Corporate, Group, IEEE

Processes/Plans SCMP, SQAP, SDP, Project Plan

Procedures Change Management, Design

Reviews, Document Standards,

Testing

Task Instructions Library updates, unit testing, peer

reviews

Success of an audit is directly proportional to preparation,

research and analysis conducted before the audit is

performed.


13/22


Requirement Types

Functional (ascertainably true or false)

Quality (range of acceptable values)


14/22


Types of Audits (Internal)

Quality System Audits

Product Audit

Process Audit

Project Audit

CM Audit


15/22


Evidence Collection

Collect Factual Information

Analyze and Evaluate the Evidence

Draw Conclusions

Generate Findings


16/22


Corrective Action ofFindings

Determine Action

Immediate Remedial Action

Process Improvement/Fix

Acceptable Risk Identify Root Cause

Corrective Actions Plan

Manage CA Plan to completion Analyze Effects of CA


17/22


Develop Audit Checklist

Focus on clear requirements (or

unclear to fix)

Select subset of requirements Focus on important steps/products

Write clear concise questions

Canned checklist vs. straw horse


18/22


Checklist Sample

Requirement Checklist Item Details Observations Results (P/F)

Company

Standard ABC-

234, page 7

Does project QA plan

will have a list of

deliverables subject to

Peer Reviews?

Check SQA document for a list

of approved peer reviews and

which documents are to be

reviewed. (if no documents are

found, then fail. If no peer

review procedures are

referenced, then fail)

Project SQA

Plan

Were the number of

audits completed

equal to the number

planned?

Check to see which audits were

planned for the last 60 days.

Check for evidence that the audit

was completed and if there were

findings, that a CA plan was

signed.

Project SQA

Plan

Were the number of

peer reviewscompleted equal to

the number planned?

For each peer review type, check

the CM records for the past 60days to see if the document type

specified in the QA plan was

checked into CM for the first

time. If so, check for records of

the peer review being completed

as per peer review process cited

in SQA plan.


19/22


Interviewing

Ask open-ended questions

Know the types of answers expected

Focus on Process and not People

Seek Corroboration and Evidence


20/22


Sample Interview Questions

How do you track your progress?

Do you have a CM Plan?

Tracing

What are you working on? Is it a configured item?

Do you have an approved CR or PR?

Is the version you are working onchecked out of CM?


21/22


Desirable Auditor Characteristics

EmotionalEmotional

Interviews

Group

dynamics

Oral reports Empathy

Dont take

things

personally

MechanicalMechanical

Sampling

Root Cause

Analysis

IntellectualIntellectual Writing

Planning

Speaking

Detail

Oriented

Concise


22/22


Desirable Auditor Characteristics(Cont.)

Knowledge of Audit process Knowledge of target (SW) processes

Knowledge of techniques

Professional attitude

Good listener

Inquisitive/analytical

Communicates at all levels

Detailed Notes and Observations

Diplomatic

software process audits

Documents