GPRA Modernization Act of 2010: Potential Legislative Perspectives on Evaluation, Measurement, and AnalysisClint Brass, Analyst in Government Organization and Management, cbrass@crs.loc.gov

September 25, 2012 -- Washington Evaluators Brownbag

CRS-2

Outline of discussion

• Potential legislative perspectives on government performance

• Framework of GPRA Modernization Act• Some threshold issues for practitioners and

users• Potential frameworks for evaluation,

measurement, and analysis• Concluding observations

CRS-3

Potential legislative perspectives on government performance

CRS-4

Institutional context

• Through public law and some non-statutory means, Congress may • Give agencies their missions• Specify their work processes and organization• Provide and allocate their resources• Determine or influence their priorities

• Congress may cooperate or compete with the President to influence how agencies formulate and implement policy

• In practice, agencies may operate with more or less policy and political autonomy

CRS-5

Congress and government performance: at least two major roles

• Using policy analysis, evaluation, and performance measurement in specific contexts, to inform• Thinking• Oversight• Policy making

• Establishing and modifying performance-related policies• Processes (e.g., GPRA, evaluation, planning, reporting)• Institutions (e.g., positions and organizations)• What constitutes “evidence”• Addressing needs of multiple stakeholders, for their use:

• Congress (committees and Members)• Agency personnel• President• Public (stakeholders and individual citizens)

CRS-6

Congressional use of information and analysis: pathways and brokers

Why brokers?• “Satisficing”• Synthesis• Credibility

Use?• Thinking• Oversight• Policy making

Congress• Authorizing comtes.• Appropriations comtes.• Oversight comtes.• Budget comtes.• Members and informal 

caucuses

“Administration”• President• OMB

Agencies• Departmental heads• Bureau heads• Evaluation offices• Budget offices

(direct)

Source: adapted fromBrass (2011).

*The National Academy of Sciences is a private 501(c)(3) corporation that receives the majority of its funding from government contracts.

(indirect)

(indirect)

Federal “brokers”• GAO• CBO• CRS• Inspectors General

Nonfederal “brokers”• Academia• Think tanks• Advocacy groups• Lobbyists• The public• Nat. Acad. of Sciences*

CRS-7

Framework of GPRA Modernization Act

CRS-8

GPRA Modernization Act: comparison with GPRA 1993 (slide 1 of 2)

• Continues three agency-level plans and reports (“products”) from Government Performance and Results Act of 1993 (GPRA 1993), but with changes

• Establishes new products and processes that focus primarily on goal-setting and performance measurement in policy areas that cut across agencies

• Brings attention to using goals and measurements during policy implementation

• Increases Web-based reporting

CRS-9

GPRA Modernization Act: comparison with GPRA 1993 (slide 2 of 2)

• Requires individuals to be responsible for some goals and management tasks

• Aligns timing of many products to coincide with presidential terms and budget proposals

• Includes more central roles for the Office of Management and Budget (OMB)

• Establishes more specific requirements for congressional consultations

• Continues emphasis on goal-setting and performance measurement… along with opportunities for, but little explicit emphasis on, program evaluation

CRS-10

Timeline for implementation: requirements and deadlines

Source: CRS.

CRS-11

Illustrative relationships among contents of products and processes

Source: CRS.

CRS-12

Some threshold issues for practitioners and users

CRS-13

Different “tribes”, jargons, and emphases

Among practitioners• Performance measurement vs.

program evaluation• Impact evaluation vs. other

evaluation types (e.g., qualitative, outcome, process)

• Randomized controlled trials (RCTs) vs. other impact evaluation types

• Summative vs. formative evaluation• Policy analysis (often prospective)

vs. evaluation and measurement (often retrospective)

Among and outside of practitioners• Budgeteers, OMB, agency managers,

evaluators, performance measurers, appropriators, authorizing committees, government operations committees

• Different skill sets, schedule orientations (budgeteers vs. managers), priorities (summative versus formative), and interestsSource: CRS.

CRS-14

Tools: some key distinctions

• Program evaluation: use of one or more formal methods to assess how, and the extent to which, programs or policies achieve intended objectives or cause unintended consequences (evaluation may be ongoing activity or discrete study)

• Performance measurement: periodic counting of data related to programs or policies, which typically does not account for “external factors”

• Policy analysis: typically prospective, drawing on the above and other analytical methods like forecasting, risk assessment, theory, logic, etc.

CRS-15

Example of distinction between evaluation and measurement: impact evaluation

Source: CRS.

CRS-16

Defining “success” and “performance” (slide 1 of 2)

• Definition of “success” or “performance” is often politically contested for the same program• Many statutes do not specify goals or purposes in detail• There may be trade-offs among potentially competing

values (efficiency, effectiveness, fairness, service, etc.)• Multiple audiences bring their own perspectives and

informational needs• Agency program staff• Agency leaders• Congress• President and OMB• Service delivery partners• Non-federal stakeholders• The public

CRS-17

Defining “success” and “performance” (slide 2 of 2)

• Unit of analysis: multiple angles on performance, broken down by…• Agency• Program• Policy• Strategy• Activity (mission and mission-support)• Goal• Outcome (end outcome and intermediate outcome)• Output• Metric, measure, or indicator• Clientele• Groups of any of the above

• Multiple potential research questions and corresponding methods of evaluation, analysis, and measurement

CRS-18

Thinking about “performance”: organize by program, goal, or something else?

Source: CRS.

CRS-19

Potential frameworks for evaluation, measurement, and analysis

CRS-20

How a policy may work: logic models

Source: adapted from Hatry (2006).

CRS-21

Looking across programs and agencies

Source: CRS.

CRS-22

Potential for perverse incentives

Source: adapted from Fisher, Schoenfeldt, and Shaw (2006).

CRS-23

Concluding observations

CRS-24

“Evidence” and policy

• What is “evidence”? In practice…• Retrospective (e.g., evaluations, measurements, evaluation

syntheses)• Prospective (e.g., policy analysis tools)• Current-day (e.g., values, ethics, risk preference)

• What constitutes “use” of evidence? Arguably, when evidence informs…• Thinking• Oversight and monitoring• Policy making

• What makes “evidence” and its presentation appear credible? Assessments for credibility may look for…• Appropriate methods (often, multiple methods)• Definition(s) of success• Fair representations about performance

CRS-25

Some potential issues for Congress

• Congressional consultations and defining “success”• Agency and OMB representations about

performance• Oversight, transparency, and public participation• Crosscutting policy areas• Design and implementation of the law

• Serving Congress’s needs, agencies’ needs?• Promoting both improvement and accountability?• Do agencies have the necessary capacity—staff, skills,

technology, funding—to implement the law?

CRS-26

Questions?