11
AUTOPSY OF A FAILED AUTOPSY OF A FAILED EVALUATIONEVALUATION
Examination Against Examination Against The Government The Government Auditing StandardsAuditing Standards
Daniel L. StufflebeamDaniel L. Stufflebeam
22
FOCUS:FOCUS:
The Government Auditing The Government Auditing StandardsStandards
Commonly known as the Commonly known as the GAO Standards or the GAO Standards or the Yellow Book StandardsYellow Book Standards
33
THE SESSION’S PARTS THE SESSION’S PARTS AREARE
1.1. A rationale for evaluation A rationale for evaluation standardsstandards
2.2. A case showing the utility of A case showing the utility of The GAO StandardsThe GAO Standards
3.3. The contents of The contents of The GAO The GAO StandardsStandards
4.4. Recommendations for Recommendations for applying applying The GAO StandardsThe GAO Standards
44
DISCUSSION PROTOCOLDISCUSSION PROTOCOL
Record your questions on a piece Record your questions on a piece of paper during the presentation .of paper during the presentation .
At appropriate times you may pose At appropriate times you may pose your questions so that the class can your questions so that the class can hear and discuss them. hear and discuss them.
66
STANDARDS FOR STANDARDS FOR EVALUATIONS AREEVALUATIONS ARE
Widely shared principles Widely shared principles for guiding & judging for guiding & judging the conduct of the conduct of evaluationsevaluations
Developed & approved by Developed & approved by experts in the conduct & experts in the conduct & use of evaluationuse of evaluation
77
THE EVALUATION FIELD’S THE EVALUATION FIELD’S STANDARDS INCLUDESTANDARDS INCLUDE
2004 AEA2004 AEA Guiding Principles for Guiding Principles for EvaluatorsEvaluators
20112011 Program Evaluation Standards Program Evaluation Standards
2007 2007 Government Auditing Government Auditing Standards Standards (GAO Standards)(GAO Standards)
88
STANDARDS FOR STANDARDS FOR EVALUATIONS PROVIDEEVALUATIONS PROVIDE
Principled DirectionPrincipled Direction
Technical GuidanceTechnical Guidance
A Basis forA Basis for Professional Professional CredibilityCredibility
A Basis forA Basis for Evaluating Evaluations Evaluating Evaluations
A Basis forA Basis for Public Accountability Public Accountability
99
EVALUATORS IGNORE OR EVALUATORS IGNORE OR FAIL TO MEET STANDARDS FAIL TO MEET STANDARDS TOTO
Their professional perilTheir professional peril
The detriment of their The detriment of their clientsclients
1010
FAILURE TO MEET STANDARDS FAILURE TO MEET STANDARDS CAN CONTRIBUTE TOCAN CONTRIBUTE TO
Conflict of interest & biasConflict of interest & bias Erroneous conclusionsErroneous conclusions Cover-up of findingsCover-up of findings Misguided decisionsMisguided decisions Breakdown of trustBreakdown of trust Organizational repercussionsOrganizational repercussions Personal losses & tragediesPersonal losses & tragedies Lowered credibility of evaluators, Lowered credibility of evaluators,
their organizations, & their their organizations, & their professionprofession
Increased government controls Increased government controls
1111
QUESTIONS RE PART 1QUESTIONS RE PART 1
What are What are positive reasonspositive reasons for for learning & applying learning & applying evaluation standards?evaluation standards?
What are possible What are possible drawbacksdrawbacks to the use of standards?to the use of standards?
Have you seen Have you seen bad bad consequencesconsequences for evaluations for evaluations that didn’t meet standards?that didn’t meet standards?
1313
PART 2: A CASEPART 2: A CASE
A UNIVERSITY’S REVIEW OF A UNIVERSITY’S REVIEW OF ITS GRADUATE PROGRAMSITS GRADUATE PROGRAMS
(A university group should have (A university group should have followed evaluation standards followed evaluation standards but didn’t.)but didn’t.)
1414
CONTEXT WAS CONTEXT WAS PROBLEMATICPROBLEMATIC
Board had Board had voted voted confidence in confidence in the president the president (12/06)(12/06)
Faculty gave Faculty gave president & president & provost low provost low ratings (2/07)ratings (2/07)
Enrollment was Enrollment was decliningdeclining
U. faced a U. faced a fiscal crisisfiscal crisis
Review focused Review focused on resource on resource allocationallocation
Morale was lowMorale was low
1515
REVIEW’S STATED REVIEW’S STATED PURPOSES:PURPOSES:
Address a fiscal crisisAddress a fiscal crisis over the over the university’s inability to support all university’s inability to support all of its programs & maintain of its programs & maintain excellenceexcellence
Determine which programs are Determine which programs are highest strategic priorities highest strategic priorities based on qualitybased on quality
Identify Identify programs for programs for increased fundsincreased funds
1616
SCOPE OF THE REVIEWSCOPE OF THE REVIEW
To be completed within 1 yearTo be completed within 1 year
All masters & doctoral programsAll masters & doctoral programs
Launched on 7/19/06Launched on 7/19/06
114 programs were reviewed114 programs were reviewed
1717
THE REVIEW’S PLANTHE REVIEW’S PLAN
Keyed to Dickeson book (chapter 5)Keyed to Dickeson book (chapter 5) Data bookData book Program’s reportProgram’s report Dean’s reportDean’s report Review team’s reportReview team’s report Appeals of review team’s reportAppeals of review team’s report Provost’s final reportProvost’s final report Board’s decisionsBoard’s decisions
1818
THE REVIEW PLAN’S THE REVIEW PLAN’S OMISSIONSOMISSIONS
NoNo update of the university’s mission update of the university’s mission No No appeals of provost’s conclusionsappeals of provost’s conclusions No No adoption of standards for reviewsadoption of standards for reviews Minimal Minimal participation of outside participation of outside
evaluatorsevaluators No No external metaevaluationexternal metaevaluation
1919
GENERAL REVIEW GENERAL REVIEW CRITERIACRITERIA External External demanddemand Quality of student & program Quality of student & program
outcomesoutcomes Quality of program Quality of program administration & administration &
planningplanning Program Program size, scope, & productivitysize, scope, & productivity Program Program impact, justification, & impact, justification, &
essentialityessentiality Opportunity analysisOpportunity analysis Compelling program factor Compelling program factor (features (features
that make it unique & excellent)that make it unique & excellent)
2020
DEFINITION OF SUB-DEFINITION OF SUB-CRITERIACRITERIA
ManyMany
Evolved throughout the Evolved throughout the reviewreview
Caused confusion & Caused confusion & controversycontroversy
2121
CRITERIA OMITTED FROM CRITERIA OMITTED FROM DICKESON’S LISTDICKESON’S LIST
History, development, & History, development, & expectations of the programexpectations of the program
Internal demand for the programInternal demand for the program Quality of program inputs & Quality of program inputs &
processesprocesses Revenue & other resources Revenue & other resources
generatedgenerated Program costs & associated Program costs & associated
costscosts
2222
EVALUATION EVALUATION PROCEDURESPROCEDURES
Program’s Program’s self-reportself-report Document & data book reviewDocument & data book review Group & individualGroup & individual interviews interviews Variable Variable protocols for ratings protocols for ratings (1-5)(1-5) Training Training of review team leadersof review team leaders Rating Rating of each program by department, of each program by department,
dean, review team, & provostdean, review team, & provost Synthesis Synthesis by provost & staffby provost & staff
2323
REVIEW PERSONNELREVIEW PERSONNEL
Essentially Essentially internalinternal PProvost rovost was both the was both the primary decision primary decision
maker & de facto lead evaluatormaker & de facto lead evaluator Provost’s staff Provost’s staff assisted the processassisted the process A programA program representative representative wrote the wrote the
program’s report &sent it to deptartment program’s report &sent it to deptartment faculty, dean, & review teamfaculty, dean, & review team
Faculty input variedFaculty input varied across programs across programs The The deandean rated the college’s programs & rated the college’s programs &
sent reports to the department chairs & sent reports to the department chairs & review team (not in original plan)review team (not in original plan)
2424
REVIEW PERSONNEL REVIEW PERSONNEL (continued)(continued)
Seven 7-person Seven 7-person review teamsreview teams rated rated designated programs & on the same day designated programs & on the same day e-mailed all reports to the provost & e-mailed all reports to the provost & pertinent deans & department chairspertinent deans & department chairs
Review team members were Review team members were mainly from mainly from outsideoutside the program’s college the program’s college
PProvost met with deansrovost met with deans before before finalizing decisionsfinalizing decisions
Provost met with team leadersProvost met with team leaders before before releasing final reportreleasing final report
An internal An internal evaluation expertevaluation expert assisted assisted
2525
FINAL REPORTFINAL REPORT
Issued on May 11, 2007Issued on May 11, 2007 Gave Gave priorities for fundingpriorities for funding in in
each collegeeach college Announced plans to Announced plans to maintain maintain 56, 56,
increaseincrease 16, 16, merge merge 6, 6, maintain/merge maintain/merge 17 subject to 17 subject to review, review, transfer transfer 8, 8, closeclose 26, & 26, & createcreate 6 new degrees 6 new degrees
2626
FINAL REPORT LIMITATIONSFINAL REPORT LIMITATIONS
Gave Gave no evidentiary basisno evidentiary basis for for decisionsdecisions
Referenced Referenced no technical no technical appendixappendix
Referenced Referenced no accessible files no accessible files of supporting data, analyses, of supporting data, analyses, & data collection tools& data collection tools
Gave Gave no rating of each no rating of each program on each criterion & program on each criterion & overalloverall
2727
POSITIVE OUTCOMEPOSITIVE OUTCOME
Local paper Local paper applauded applauded the the report (5/12/07)report (5/12/07)
2828
NEGATIVE OUTCOMESNEGATIVE OUTCOMES
Review Review evidence & link to conclusions evidence & link to conclusions were inaccessiblewere inaccessible to many interested to many interested partiesparties
Professors, alumni, & others Professors, alumni, & others protestedprotested President announced an President announced an appeal process appeal process
(5/18/07)(5/18/07) Faculty voted to call for a Faculty voted to call for a censure censure of the of the
provost (5/18/07)provost (5/18/07) Provost Provost resigned resigned (5/20/07)(5/20/07) Appeals overturned 10 planned cuts Appeals overturned 10 planned cuts
(7/14/07)(7/14/07) Potential Potential savings from cuts were reducedsavings from cuts were reduced
2929
NEGATIVE OUTCOMES NEGATIVE OUTCOMES (continued)(continued)
CCommunity watched ommunity watched a contentious a contentious processprocess
Board fired the presidentBoard fired the president President threatened to sue President threatened to sue Board awarded the ex-president Board awarded the ex-president
$530,000 in severance pay$530,000 in severance pay Projected Projected review of undergraduate review of undergraduate
programs was canceledprograms was canceled, ceding that , ceding that area priority by default area priority by default
Reviews to resume in 2010Reviews to resume in 2010
3030
CLEARLY, THIS CLEARLY, THIS EVALUATION FAILEDEVALUATION FAILED
No standards were required No standards were required to to reach this conclusion.reach this conclusion.
However, However, adherence to adherence to approved standards might approved standards might have prevented the review’s have prevented the review’s failurefailure..
3131
QUESTIONS RE PART 2QUESTIONS RE PART 2
Are you surprised that this Are you surprised that this evaluation failed? Why or why evaluation failed? Why or why not?not?
What were this evaluationWhat were this evaluation’s strengths?
Where did the evaluation go wrong?
3232
MORE QUESTIONS RE PART MORE QUESTIONS RE PART 22
When planning a similar When planning a similar evaluation how would you avoid a evaluation how would you avoid a failure?failure?
What roles should standards play What roles should standards play in assessing a program in assessing a program evaluation?evaluation?
What are your questions?What are your questions?
3333
MY TAKE-MY TAKE-ON THE ON THE PLUS SIDEPLUS SIDE::
Review was keyed to an Review was keyed to an important need important need to restructure to restructure programs.programs.
There was significant There was significant faculty faculty involvement involvement in studying in studying programs.programs.
General General criteria criteria were were established.established.
3434
HOWEVER, THERE WERE HOWEVER, THERE WERE SERIOUS SERIOUS DEFICIENCIESDEFICIENCIES
No No independent perspectivesindependent perspectives Evaluator & decision maker Evaluator & decision maker were were
the samethe same Evidence to support conclusions Evidence to support conclusions
was not reportedwas not reported Political viability was not Political viability was not
maintainedmaintained No independent evaluation of the No independent evaluation of the
reviewreview
3737
FOR A MORE SYSTEMATIC FOR A MORE SYSTEMATIC EXAMINATION OF THE EXAMINATION OF THE CASECASE Let’s see if use of the GAO Let’s see if use of the GAO
Standards might have helped Standards might have helped ensure the study’s success.ensure the study’s success.
Let’s also use the case to Let’s also use the case to develop a working knowledge develop a working knowledge of of the GAO Standards.the GAO Standards.
3838
THE PARTICULAR USE OF THE GAO THE PARTICULAR USE OF THE GAO STANDARDS I HAVE IN MIND STANDARDS I HAVE IN MIND INVOLVESINVOLVES
Metaevaluation (the evaluation of Metaevaluation (the evaluation of evaluations)evaluations)
Formative metaevaluation is Formative metaevaluation is proactive evaluation of an evaluation proactive evaluation of an evaluation (in process) to help ensure its (in process) to help ensure its success.success.
Summative metaevaluation is Summative metaevaluation is retroactive evaluation of a retroactive evaluation of a completed evaluation to judge its completed evaluation to judge its integrity & soundness.integrity & soundness.
A Metaevaluation A Metaevaluation FrameworkFramework
Formative Metaevaluation
Summative Metaevaluation
Internal Evaluator assesses & strengthens the evaluation
Evaluator reports an attestation of the evaluation’s adherence to standards for evaluations
External Independent metaevaluator provides feedback for strengthening the ongoing evaluation
Independent metaevaluator reports on the completed evaluation’s adherence to professional standards
3939
4040
SOME BACKGROUND SOME BACKGROUND INFORMATION ON THE INFORMATION ON THE GAO STANDARDSGAO STANDARDS
4141
THE GAO STANDARDSTHE GAO STANDARDS
Are issued by the U.S. Government Are issued by the U.S. Government Accountability Office.Accountability Office.
Provide a framework for performing Provide a framework for performing high-quality audit work with high-quality audit work with competencecompetence, , integrityintegrity, , objectivityobjectivity, & , & independenceindependence..
Foster Foster transparencytransparency, , performanceperformance, , accountabilityaccountability, & , & qualityquality through the through the audit process. audit process.
Focused on U.S. government Focused on U.S. government agenciesagencies
Are used throughout the world.Are used throughout the world.
4242
BACKGROUND OF THE GAO BACKGROUND OF THE GAO STANDARDSSTANDARDS
Since 1974, the US Comptroller Since 1974, the US Comptroller General has been issuing the General has been issuing the Government Auditing Standards Government Auditing Standards (Yellow Book)(Yellow Book)
The GAO standards cover The GAO standards cover government-funded departments & government-funded departments & programs at federal, state, & local programs at federal, state, & local levelslevels
Public universities are among the Public universities are among the intended usersintended users
4343
THE GAO STANDARDSTHE GAO STANDARDSFOCUS ONFOCUS ON
FINANCIAL AUDITSFINANCIAL AUDITS
ATTESTATION ENGAGEMENTSATTESTATION ENGAGEMENTS ((conducting agreed-upon procedures on a subject matter or conducting agreed-upon procedures on a subject matter or an assertion about a subject matter & reporting on the an assertion about a subject matter & reporting on the results)results)
PERFORMANCE AUDITSPERFORMANCE AUDITS (including (including program evaluations)program evaluations)
4444
THE GAO STANDARDS THE GAO STANDARDS includeinclude
General StandardsGeneral Standards
Field Work StandardsField Work Standards
Reporting StandardsReporting Standards
4545
NOW, LET’S LOOK ATNOW, LET’S LOOK AT
THE CONTENTS OF THE GAO THE CONTENTS OF THE GAO STANDARDS &STANDARDS &
DISCUSS THEIR RELEVANCE DISCUSS THEIR RELEVANCE TO THE PROGRAM REVIEW TO THE PROGRAM REVIEW CASECASE
4646
THE GAO STANDARDS THE GAO STANDARDS INCLUDE FOUR GENERAL INCLUDE FOUR GENERAL STANDARDS:STANDARDS:
IndependenceIndependence Professional JudgmentProfessional Judgment CompetenceCompetence Quality Control & Quality Control &
AssuranceAssurance (including (including external peer review)external peer review)
4747
General Standard 1: General Standard 1: INDEPENDENCEINDEPENDENCE
The audit organization & auditor The audit organization & auditor must be must be free from personal, external, free from personal, external, & organizational impairments& organizational impairments to to independence & must independence & must avoid the avoid the appearanceappearance of such impairments. of such impairments.
Opinions, findings, conclusions, Opinions, findings, conclusions, judgments, & recommendations must judgments, & recommendations must be be impartialimpartial & viewed as impartial by & viewed as impartial by knowledgeable, objective third knowledgeable, objective third parties.parties.
4848
QUESTION REGARDING QUESTION REGARDING INDEPENDENCEINDEPENDENCE::
How well did the program How well did the program review meet the requirements review meet the requirements for independence & impartiality?for independence & impartiality?
How would you have designed How would you have designed the review differently to meet the review differently to meet the requirement for the requirement for independence? independence?
4949
General Standard 2: General Standard 2: PROFESSIONAL PROFESSIONAL JUDGMENTJUDGMENT Auditors must use professional Auditors must use professional
judgment in planning & judgment in planning & performing evaluations & in performing evaluations & in reporting the results. reporting the results.
Auditors must diligently Auditors must diligently perform, in good faith & with perform, in good faith & with integrity, the gathering & integrity, the gathering & assessment of evidence.assessment of evidence.
5050
QUESTION REGARDING QUESTION REGARDING PROFESSIONAL JUDGMENTPROFESSIONAL JUDGMENT
In the program review case, in what In the program review case, in what respects did the provost exercise respects did the provost exercise questionable judgment in gathering questionable judgment in gathering & controlling evidence & reporting & controlling evidence & reporting findings & conclusions?findings & conclusions?
How would you have conducted How would you have conducted the review differently to meet the review differently to meet the requirement for the requirement for professional judgment? professional judgment?
5151
General Standard 3: General Standard 3: COMPETENCECOMPETENCE The evaluation staff must The evaluation staff must
collectively possess adequate collectively possess adequate professional competence for professional competence for the tasks required.the tasks required.
Competence is derived from a Competence is derived from a blending of education & blending of education & experience.experience.
5252
QUESTION REGARDING QUESTION REGARDING COMPETENCECOMPETENCE
Would you agree or disagree Would you agree or disagree that competence likely was that competence likely was not the cause of the program not the cause of the program review’s failure?review’s failure?
Why or why not?Why or why not?
5353
General Standard 4: General Standard 4: QUALITY CONTROL & QUALITY CONTROL & ASSURANCEASSURANCE Each audit organization . . . must Each audit organization . . . must
establish a system of quality establish a system of quality control that provides reasonable control that provides reasonable assurance of compliance with assurance of compliance with standards & legal & regulatory standards & legal & regulatory requirements.requirements.
It must secure an external peer It must secure an external peer review at least once every 3 review at least once every 3 yearsyears
5454
QUESTIONS REGARDING QUESTIONS REGARDING QUALITY CONTROL & QUALITY CONTROL & ASSURANCEASSURANCE
Despite this standard’s Despite this standard’s reference to an reference to an organization’sorganization’s quality control system, is its quality control system, is its message applicable to individual message applicable to individual studies?studies?
What might the provost have What might the provost have done differently to assure the done differently to assure the program review’s quality & program review’s quality & acceptability?acceptability?
QUESTIONS REGARDING QUALITY QUESTIONS REGARDING QUALITY CONTROL & ASSURANCE (continued)CONTROL & ASSURANCE (continued)
Do you think the GAO Standards’ Do you think the GAO Standards’ provision for peer reviews at 3-provision for peer reviews at 3-year intervals is sufficient?year intervals is sufficient?
How might this provision be How might this provision be strengthened?strengthened?
5555
5656
What questions do What questions do you have concerning you have concerning the GAO General the GAO General Standards?Standards?
5757
NEXT, WE LOOK AT NEXT, WE LOOK AT THE GAO FIELD WORK THE GAO FIELD WORK STANDARDS FOR STANDARDS FOR PERFORMANCE AUDITSPERFORMANCE AUDITS
5858
THE FIELD WORK THE FIELD WORK STANDARDS INCLUDESTANDARDS INCLUDE
OVERARCHING OVERARCHING CONCEPTSCONCEPTS::
- ReasonablReasonable e assuranceassurance
- SignificancSignificancee
- Audit risk- Audit risk
FOUR STANDARDSFOUR STANDARDS
1.1. PlanningPlanning
2.2. SupervisionSupervision
3.3. Obtaining Obtaining Sufficient, Sufficient, Appropriate Appropriate EvidenceEvidence
4.4. Audit Audit DocumentationDocumentation
5959
FRAMEWORKFRAMEWORK FOR THE FOR THEFIELD WORK STANDARDS:FIELD WORK STANDARDS:
PlanninPlanningg
SupervisioSupervisionn
Obtaining Obtaining Sufficient, Sufficient, Appropriate Appropriate EvidenceEvidence
AuditAudit
DocumentationDocumentation
ReasonablReasonablee
assuranceassurance
SignificancSignificancee
Audit riskAudit risk
6060
OVERALL, EVALUATORS OVERALL, EVALUATORS SHOULDSHOULD
Provide Provide reasonable assurance reasonable assurance that evidence that evidence is sufficient & appropriate to support the is sufficient & appropriate to support the findings & conclusions.findings & conclusions.
ConsiderConsider the the relative significance relative significance of of relevant matters when deciding the type & relevant matters when deciding the type & extent of audit work, evaluating audit results, extent of audit work, evaluating audit results, & reporting.& reporting.
Assess & address Assess & address audit risk audit risk to ensure that to ensure that findings, conclusions, & recommendations are findings, conclusions, & recommendations are proper & complete, re. such factors as proper & complete, re. such factors as sufficient, appropriate evidence, an adequate sufficient, appropriate evidence, an adequate audit process, & avoidance of omissions, audit process, & avoidance of omissions, misrepresentation, & fraud.misrepresentation, & fraud.
6161
QUESTION REGARDING QUESTION REGARDING REASONABLE REASONABLE ASSURANCEASSURANCE Clearly, the provost did not Clearly, the provost did not
give give reasonable assurancereasonable assurance that his conclusions were that his conclusions were grounded in solid, grounded in solid, transparent evidence.transparent evidence.
What would you have done to What would you have done to provide such reasonable provide such reasonable assurance?assurance?
6262
QUESTION REGARDING QUESTION REGARDING SIGNIFICANCESIGNIFICANCE
Do you think the provost Do you think the provost accorded sufficient accorded sufficient significancesignificance to the matter of to the matter of abrogating his commitment to abrogating his commitment to make the program review’s make the program review’s evidence available for review?evidence available for review?
Why or why not?Why or why not?
6363
QUESTIONS QUESTIONS REGARDING REGARDING AUDIT AUDIT RISKRISK Clearly, the provost did not Clearly, the provost did not
exercise sufficient safeguards to exercise sufficient safeguards to avoid risksavoid risks to the success of this to the success of this program review.program review.
What were important audit risk What were important audit risk factors?factors?
What steps were needed to assure What steps were needed to assure the review’s success and promote the review’s success and promote acceptance by the faculty?acceptance by the faculty?
6464
Field Work Standard 1: Field Work Standard 1: PLANNINGPLANNING Auditors must adequately Auditors must adequately
plan & document the planning plan & document the planning of required audit work.of required audit work.
They should consider They should consider significance & audit risk in significance & audit risk in defining the audit objectives defining the audit objectives & the required scope & & the required scope & methodology.methodology.
6565
QUESTION REGARDING QUESTION REGARDING PLANNINGPLANNING Do you think the problem in Do you think the problem in
the program review was more the program review was more in deficient planning or in deficient planning or deficient execution of plans?deficient execution of plans?
Please explain.Please explain.
6666
Field Work Standard 2: Field Work Standard 2: SUPERVISIONSUPERVISION Auditors must supervise staff.Auditors must supervise staff.
Audit supervision involves Audit supervision involves providing sufficient guidance providing sufficient guidance & direction to staff, & direction to staff, monitoring & reviewing their monitoring & reviewing their performance, & providing on-performance, & providing on-the-job training.the-job training.
6767
QUESTION REGARDING QUESTION REGARDING SUPERVISIONSUPERVISION
Considering that the program review Considering that the program review was required to be completed within was required to be completed within 1 year & that 114 programs were 1 year & that 114 programs were reviewed, was the standard of reviewed, was the standard of quality likely high across all program quality likely high across all program reviews?reviews?
How would you have assured that all How would you have assured that all 114 reviews were well supervised?114 reviews were well supervised?
6868
Field Work Standard 3: Field Work Standard 3: SUFFICIENT, APPROPRIATE SUFFICIENT, APPROPRIATE EVIDENCEEVIDENCE Sufficient, appropriate evidence Sufficient, appropriate evidence
must be obtained to provide a must be obtained to provide a reasonable basis for findings & reasonable basis for findings & conclusions.conclusions.
Evidence should be assessed for its Evidence should be assessed for its relevance, validity, reliability, & relevance, validity, reliability, & sufficiency.sufficiency.
It should be adequate to persuade a It should be adequate to persuade a knowledgeable person that the knowledgeable person that the findings are reasonable.findings are reasonable.
6969
QUESTION REGARDING QUESTION REGARDING SUFFICIENT, APPROPRIATE SUFFICIENT, APPROPRIATE EVIDENCEEVIDENCE
Assume the university engaged Assume the university engaged you to conduct an independent you to conduct an independent evaluation of the program evaluation of the program review’s evidence.review’s evidence.
What information would you What information would you request?request?
How would you assess it?How would you assess it?
7070
Field Work Standard 4: Field Work Standard 4: AUDIT AUDIT DOCUMENTATIONDOCUMENTATION Auditors must prepare audit Auditors must prepare audit
documentation related to documentation related to planning, conducting, & planning, conducting, & reporting before issuing their reporting before issuing their report.report.
The documentation should The documentation should contain support for findings, contain support for findings, conclusions, & recommendations. conclusions, & recommendations.
7171
QUESTIONS REGARDING QUESTIONS REGARDING AUDIT DOCUMENTATIONAUDIT DOCUMENTATION
How important is it to How important is it to document evaluation studies?document evaluation studies?
Should such documentation Should such documentation be made available to all be made available to all interested persons?interested persons?
What are the pros and cons of What are the pros and cons of releasing such releasing such documentation?documentation?
7272
THIS CONCLUDES OUR THIS CONCLUDES OUR DISCUSSION OF THE GAO DISCUSSION OF THE GAO FIELD WORK STANDARDSFIELD WORK STANDARDS
What questions do you What questions do you have?have?
7575
REPORTING STANDARDSREPORTING STANDARDS for Performance Audits arefor Performance Audits are
1.1. ReportingReporting
2.2. Report ContentsReport Contents
3.3. Report Issuance & Report Issuance & DistributionDistribution
7676
Reporting Standard 1: Reporting Standard 1: REPORTINGREPORTING Auditors must issue Auditors must issue
reports communicating reports communicating the audit results.the audit results.
Reports should be in Reports should be in writing or some other writing or some other retrievable form.retrievable form.
7777
QUESTION REGARDING QUESTION REGARDING REPORTINGREPORTING The provost issued a written The provost issued a written
report & gave it to faculty & report & gave it to faculty & the media. the media.
Why, then, did the faculty Why, then, did the faculty object & secure the provost’s object & secure the provost’s dismissal? dismissal?
7878
Reporting Standard 2: Reporting Standard 2: REPORT CONTENTSREPORT CONTENTS Auditors’ reports should containAuditors’ reports should contain
(1) the audit’s objectives, scope, & (1) the audit’s objectives, scope, & methodsmethods
(2) findings, conclusions, & (2) findings, conclusions, & recommendations, as appropriaterecommendations, as appropriate
(3) stated compliance with standards(3) stated compliance with standards
(4) views of responsible officials(4) views of responsible officials
(5) if applicable, the nature of any omitted (5) if applicable, the nature of any omitted privileged & confidential informationprivileged & confidential information
7979
QUESTION REGARDING QUESTION REGARDING REPORT CONTENTSREPORT CONTENTS Provost’s report Provost’s report
– did not reference compliance did not reference compliance with standardswith standards
– did not disclose evidence that did not disclose evidence that supported his conclusionssupported his conclusions
How important were these How important were these omissions to the review’s omissions to the review’s success?success?
8080
Reporting Standard 3: Reporting Standard 3: DISTRIBUTING DISTRIBUTING REPORTSREPORTS Usually, reports should be given Usually, reports should be given
to governance personnel, to governance personnel, officials of the audited entity, officials of the audited entity, appropriate oversight bodies, appropriate oversight bodies, other right-to-know parties, & other right-to-know parties, & the public, as applicable.the public, as applicable.
Auditors should document any Auditors should document any limitation on report distribution.limitation on report distribution.
8181
QUESTIONS REGARDING QUESTIONS REGARDING DISTRIBUTING REPORTSDISTRIBUTING REPORTS
The provost distributed his The provost distributed his report to all interested report to all interested parties, including the media.parties, including the media.
Why, then, did the faculty Why, then, did the faculty object & cause the provost to object & cause the provost to be fired?be fired?
Had you been the provost, Had you been the provost, how would you have how would you have prevented this dire outcome?prevented this dire outcome?
8282
RATE THE PROGRAM RATE THE PROGRAM REVIEW ON EACH GAO REVIEW ON EACH GAO STANDARDSTANDARDSTANDARDSTANDARD METMET PARTIALLY PARTIALLY
METMETUNMETUNMET
IndependenceIndependence
Professional JudgmentProfessional Judgment
CompetenceCompetence
Quality Control & Quality Control & AssuranceAssurance
PlanningPlanning
SupervisionSupervision
Obtaining Sufficient, Obtaining Sufficient, Appropriate EvidenceAppropriate Evidence
Audit DocumentationAudit Documentation
ReportingReporting
Report ContentsReport Contents
Distributing ReportsDistributing Reports
MY RATING OF THE PROGRAM MY RATING OF THE PROGRAM REVIEW ON EACH GAO REVIEW ON EACH GAO STANDARDSTANDARDSTANDARDSTANDARD METMET Partially Partially
MetMetUNMETUNMET
IndependenceIndependence XX
Professional JudgmentProfessional Judgment XX
CompetenceCompetence XX
Quality Control & Quality Control & AssuranceAssurance
XX
PlanningPlanning XX
SupervisionSupervision XX
Obtaining Sufficient, Obtaining Sufficient, Appropriate EvidenceAppropriate Evidence
XX
Audit DocumentationAudit Documentation XX
ReportingReporting XX
Report ContentsReport Contents XX
Distributing ReportsDistributing Reports XX
8383
8484
IN SUMMARYIN SUMMARY
The program review failed.The program review failed.
The review’s leader did not invoke The review’s leader did not invoke approved standards for evaluations.approved standards for evaluations.
The GAO standards are useful for The GAO standards are useful for examining the review’s strengths & examining the review’s strengths & weaknessesweaknesses
Likely, this review would have succeeded Likely, this review would have succeeded had it met the GAO Standards.had it met the GAO Standards.
8585
YOUR WORDYOUR WORD
ON THE GAO STANDARDS’ ON THE GAO STANDARDS’ APPROPRIATENESSAPPROPRIATENESS FOR FOR GUIDING & ASSESSING GUIDING & ASSESSING PROGRAM REVIEWSPROGRAM REVIEWS
8686
APPLICABILITY OF THEAPPLICABILITY OF THEGAO STANDARDSGAO STANDARDS
Do the GAO Standards address Do the GAO Standards address important important issues & problemsissues & problems involved in your involved in your evaluations?evaluations?
How would the GAO Standards have to be How would the GAO Standards have to be changedchanged to make them useful in your fields? to make them useful in your fields?
What What steps would you need to take steps would you need to take before before you could confidently use these standards? you could confidently use these standards?
8888
PART 4PART 4
RECOMMENDATIONS FOR RECOMMENDATIONS FOR APPLYING THE GAO APPLYING THE GAO
STANDARDSSTANDARDS
8989
1. REMEMBER THE 1. REMEMBER THE BASICS OF BASICS OF METAEVALUATIONMETAEVALUATION The most important purpose of The most important purpose of
evaluation OR METAEVALUATION is evaluation OR METAEVALUATION is not only to PROVE but to IMPROVE.not only to PROVE but to IMPROVE.
Formative metaevaluations are Formative metaevaluations are needed to improve ongoing needed to improve ongoing evaluation work.evaluation work.
Summative metaevaluations are Summative metaevaluations are needed to prove the evaluation’s needed to prove the evaluation’s merit.merit.
9090
2. TRY OUT THE GAO 2. TRY OUT THE GAO STANDARDS FOR GUIDING & STANDARDS FOR GUIDING & JUDGING ONE OF YOUR JUDGING ONE OF YOUR NEXT EVALUATIONS.NEXT EVALUATIONS.
9191
3. TRAIN EVALUATION STAFF IN 3. TRAIN EVALUATION STAFF IN THE CONTENT & APPLICATION THE CONTENT & APPLICATION OF THE GAO STANDARDS.OF THE GAO STANDARDS.
Obtain copies of the GAO Obtain copies of the GAO Standards at no cost Standards at no cost through GAO’s Web site (through GAO’s Web site (www.gao.gov).).
9292
4. CONDUCT & USE FORMATIVE 4. CONDUCT & USE FORMATIVE METAEVALUATIONS OF YOUR METAEVALUATIONS OF YOUR EVALUATIONSEVALUATIONS
Base metaevaluations on the Base metaevaluations on the GAO Standards.GAO Standards.
Use formative feedback to Use formative feedback to ensure quality in your ensure quality in your evaluations.evaluations.
9393
5. APPLY THE DETAILS OF 5. APPLY THE DETAILS OF EACH STANDARDEACH STANDARD
Carefully review and apply Carefully review and apply the details of the GAO the details of the GAO Standards.Standards.
Make detailed notes on how Make detailed notes on how your evaluation responds to your evaluation responds to relevant points in the GAO relevant points in the GAO Standards.Standards.
9494
6. USE THE RESULTS OF YOUR 6. USE THE RESULTS OF YOUR APPLICATION OF THE GAO APPLICATION OF THE GAO STANDARDS TOSTANDARDS TO
Improve your evaluation as Improve your evaluation as you go.you go.
Document issues in the Document issues in the evaluation related to the GAO evaluation related to the GAO Standards & how they were Standards & how they were addressed.addressed.
9595
7. ADVISE YOUR SPONSORS TO 7. ADVISE YOUR SPONSORS TO FUND INDEPENDENT SUMMATIVE FUND INDEPENDENT SUMMATIVE METAEVALUATIONS OF YOUR METAEVALUATIONS OF YOUR EVALUATIONS.EVALUATIONS.
Recommend that the Recommend that the independent metaevaluator independent metaevaluator use the GAO Standards to use the GAO Standards to assess your evaluation.assess your evaluation.
Provide the independent Provide the independent metaevaluator with full metaevaluator with full documentation of your documentation of your formative metaevaluation.formative metaevaluation.
9696
8. SUPPORT RELEASE OF 8. SUPPORT RELEASE OF SUMMATIVE SUMMATIVE METAEVALUATION METAEVALUATION FINDINGS TO ALL RIGHT-FINDINGS TO ALL RIGHT-TO-KNOW AUDIENCES.TO-KNOW AUDIENCES.
9797
QUESTIONS QUESTIONS REGARDING PART 4REGARDING PART 4 Do you concur that both Do you concur that both
formative & summative formative & summative metaevaluations are metaevaluations are important?important?
Do you see value in using the Do you see value in using the GAO Standards for both kinds GAO Standards for both kinds of metaevaluation?of metaevaluation?
9999
IN CONCLUSION,IN CONCLUSION,
Standards have a crucial role in Standards have a crucial role in assuring the quality, credibility, & assuring the quality, credibility, & value of evaluations.value of evaluations.
Unprincipled evaluations can have Unprincipled evaluations can have dire consequences.dire consequences.
The GAO Standards are valuable The GAO Standards are valuable for guiding & judging evaluations.for guiding & judging evaluations.
101101
US GOVERNMENT US GOVERNMENT AUDITING STANDARDSAUDITING STANDARDS
By the Comptroller General of the By the Comptroller General of the United States (July 2007)United States (July 2007)
GAO-03-731G Government Auditing GAO-03-731G Government Auditing StandardsStandards
United States General Accounting United States General Accounting OfficeOffice
Washington, DC 20548-0001Washington, DC 20548-0001 www.gao.gov/govaud/ybk01.htmwww.gao.gov/govaud/ybk01.htm
102102
2004 AEA GUIDING PRINCIPLES2004 AEA GUIDING PRINCIPLES
Printed on inside cover of Printed on inside cover of issues of the issues of the American American Journal of EvaluationJournal of Evaluation
Also see Also see
http://www.eval.org/Publications/GuidingPrinciples.asp
103103
THE PROGRAM THE PROGRAM EVALUATION EVALUATION STANDARDSSTANDARDS
Joint Committee on Standards for Joint Committee on Standards for Educational Evaluation (2011). Educational Evaluation (2011). The Program Evaluation The Program Evaluation Standards. Standards. Thousand Oaks, CA: Thousand Oaks, CA: Sage.Sage.
ISBN 978-1-4129-8908-4 ISBN 978-1-4129-8908-4 (paperback)(paperback)
104104
DICKESON BOOKDICKESON BOOK
Dickeson, Robert C. (1999). Dickeson, Robert C. (1999). Prioritizing Prioritizing
Academic Programs and Services: Academic Programs and Services:
Reallocating Resources to Achieve Reallocating Resources to Achieve
Strategic Balance.Strategic Balance.San Francisco: Jossey-San Francisco: Jossey-
Bass.Bass.
105105
Stufflebeam, D. L., & Shinkfield, A. J. Stufflebeam, D. L., & Shinkfield, A. J. (2007). (2007). Evaluation Theory, Models, & Evaluation Theory, Models, &
Applications. Applications. San Francisco: Jossey-BassSan Francisco: Jossey-Bass
See Chapter 3: Standards for See Chapter 3: Standards for Program EvaluationsProgram Evaluations
ISBN-13: 978-0-7879-7765-8 ISBN-13: 978-0-7879-7765-8 (cloth)(cloth)
ISBN-10: 0-7879-7765-9 (cloth)ISBN-10: 0-7879-7765-9 (cloth)