1

Economic Directorate’sQuality Audit Program

Steven S. KlementOffice of Statistical Methods and

Research for Economic Programs

Agenda

• Origins of the Quality Audit Program (QAP)• Scope• Current Program• Some Results• Discussion

2

3

Origins of the QAP

Late 1990’s

Beginning of Census Bureau Standards

2001 Office of Management and Budget (OMB) Quality Guidelines

2004 Establishment of Economic Directorate’s quality management program

2006 OMB Standards and Guidelines for Statistical Surveys

Origins of the QAP

• What form should the Quality Management Program take?

• Quality Profile vs. Quality Audit• Developing the self-assessment tool• Administration of the audit process

4

Scope

• There are about 60 programs in the Economic Directorate that are subject to QAP

• Each program will undergo an audit every five years

• QAP must average twelve audits per year to meet this goal

5

Scope (cont.)

• The Quality Audit Program (QAP) looks for compliance with the OMB Standards and Guidelines for Statistical Surveys (Sep 2006)– Does the program’s documentation show

compliance?– Is there a plan to produce quality estimates?

6

Scope (cont.)

• Auditors do not look in depth to see if execution matches the plan– If the plan calls to compute a variance, the auditor

looks to see that there is a variance in the final data product

– QAP does not look at the actual computer program which calculates the number

7

Conduct of the QAP

• Currently run by the Lead Auditor for Economic Programs– New Assistant Lead Auditor position just created

• Overseen by the Research & Methodology (R&M) Assistant Division Chiefs (ADC)

• Census Bureau’s Quality Improvement Staff assists in training

• Volunteer auditors from the Divisions within the Economic Directorate

8

9

TimelineSep Year Prior

Set audit schedule Audit Week Meet face-to-face to resolve issues

Oct Year Prior

Assign auditors Friday Audit wk

Draft final report

9 wks prior Engagement notification sent out

1 wk after Final report due

7-9 wks prior Conduct training 2 months after PAP plan due

4-7 wks prior Program area fills in checklist

8 months after 1st PAP review

4 wks prior Program area presentation

20 months after

2nd PAP review

1-3 wks prior Auditors review checklist

Every year after

Review PAP until all items complete

Scheduling

• A five-year program is put together by the Lead Auditor and reviewed by the Divisions– Includes all programs not already subject to an

outside audit– There are limited exceptions– The Division’s comment on schedule and request

deferments• The Lead Auditor schedules the actual dates

– 3-4 months out plans dates for next year’s audits– 2-3 months out auditors are selected

10

Auditor Selection

• The Divisions within the Econ Directorate submit managers & supervisors names to be auditors

• The Lead Auditor:– Coordinates schedules– Assigns auditors to each audit.

11

Conduct of the Audit

• Engagement Notification sent nine weeks out• Training conducted 7-9 weeks out• Checklist is due 4.5 weeks out• Program area presentation is four weeks out

12

13

Checklist Example3.1 Data

Editing

Rating

(Auditor)

(Auditor) Summary of findings and

recommendations {Initials}

1. Agencies must edit data to mitigate or correct detectable errors.

Condition: (Program Mgr.) – Objective evidence for existing conditions that supports or refutes compliance with the criteria of the standard.{Initials}Concern: (Auditor) – Cause: The factors responsible for the difference between the condition and criteria. Effect: The consequences (benefits / risks) due to the difference between the condition and criteria. {Initials}Recommendation: (Auditor) – Recommendations to 1) bring the program area in compliance with the standard and for 2) alternative approaches to improve program performance. {Initials}Response: (Program Mgr) – The Program Manager’s position regarding the Auditors concerns or recommendations. {Initials}

Conduct of the Audit (cont.)

• Document review is 1-3 weeks prior– It is the first part of the actual audit– Done by auditors alone

• On-site week is when auditors meet with program area– Draft findings are discussed and then finalized– Issues are resolved in groups or one-on-one

14

Final Report

• The final report is authored by the Lead Auditor– Reviewed by the auditors– Program Area is given chance to respond

• After receiving input and completing revisions, report is sent to management

15

Program Action Plan (PAP)

• Conducting an audit is of little value in of itself• The value comes in when improvements are

made• Therefore, the QAP directs that the program

area develop a PAP– The PAP is the plan for corrective action– Due two months after the audit– The auditors comment on the feasibility of the PAP

to address audit concerns

16

Follow Up

• Finally, progress with the PAP is followed by the Lead Auditor

• Once the PAP is approved Lead Auditor:– Checks for progress after six months– Continues to check yearly until PAP is completed

17

18

Some Results

• Nonresponse Bias Study:– It was found that there was a general lack of NR

bias studies in the Directorate– Study required if response rate is below 80%– Planning for study required if response rate is near

80% OR falling towards 80%• Directorate sponsored a pilot NR study• The Directorate created a new team to help

programs conduct these studies

Some Results (cont.)

• Document storage:– Most programs are well-documented– However, the documentation is scattered

• Multiple document storage systems in use• Many programs did not use any system

– For a number of programs, the audit was the first time all documentation was in one location

• The Directorate will roll out a new centralized easy to use system in summer 2010.

19

20

Some Results (cont.)

• Holds people accountable for their documentation– It must be accessible and understandable– It must outline processes actually used– Both help in knowledge transfer over time

• Compliance with standards is brought to the forefront (instead of an afterthought)

21

Key Finding

• Almost everything one program is deficient or needs improvement on…

…another program has a best practice for!• The question: ‘How much is transferable?’• Auditors take good ideas back to their own

programs……and they share their good ideas with the audited program

QAP Going Forward

• Look at ways to spread Best Practices• Identify areas that require corporate solutions• Help identify areas requiring additional or

concentrated resources

22

Questions?

• Thank you for your time

Steven S. KlementLead Auditor for Economic Programssteven.s.klement@census.gov(301)763-6598

23

URLs

• OMB Standards & Guidelines for Statistical Surveys (Sep 2006)

http://www.whitehouse.gov/omb/inforeg/statpolicy/standards_stat_surveys.pdf• Guidance on Agency Survey and Statistical

Information Collectionshttp://www.whitehouse.gov/omb/inforeg/pmc_survey_guidance_2006.pdf

24