system of evaluation and management control results-based budgeting the chilean experience

SYSTEM OF EVALUATION AND MANAGEMENT CONTROL

RESULTS-BASED BUDGETING

THE CHILEAN EXPERIENCE

Heidi Berner HHead of Management Control Division

Budget Office, Ministry of Finance, Chile

October, 2008

1 2

EVALUATION AND MANAGEMENT CONTROL SYSTEM

OBJECTIVE: Providing performance information and introducing practices to improve the quality of public expenditure

• Improving resource allocation• Improving the use of resources • Improving transparency

1 3

EVALUATION OF PROGRAMS AND INSTITUTIONS• 1997: Evaluation of Public Programs (EPG)• 2001: Impact Evaluation of Programs• 2002: Agency Evaluation (Comprehensive Spending Review)

1993: PERFORMANCE

INDICATORS

2001: PRESENTATION OF PROGRAMS TO BUDGET

INSTITUTIONAL WAGE INCENTIVE MECHANISMS • 1998: Management Improvement Program:

- Basic Framework Program (2001)- Advance Framework Program (2003)

• 2003: Incentive to Physicians

2001: STRATEGIC

DEFINITIONS

1997: COMPREHENSIVE

MANAGEMENT REPORT (BGI)

3

1 4

PERFORMANCE INDICATORS

COMPREHENSIVE MANAGEMENT REPORT (BGI)

PRESENTATION OF PROGRAMS TO BUDGET

2007: Technical Assistance2008: Reviewing Program Presentation

EVALUATION OF PROGRAMS AND INSTITUTIONS

2006: Increasing coverage - 155 programs or agencies will be evaluated between 2006-2010.

2009: Creation of New Line: Evaluation of New Programs

INSTITUTIONAL WAGE INCENTIVE MECHANISMS

2009: Management Improvement Program (PMG) – Framework Program of Quality2007: Institutional Efficiency Goals

2008: PUBLIC MANAGEMENT MODERNIZATION FUND

4

STRATEGIC DEFINITIONS

1 5

PERFORMANCE AND BUDGETARY INFORMATION

• Not Direct relation (not mechanical)

Use together with other categories of information

- Political Priorities

- Financial Restrictions

• Presentation to the Congress along with the Budget Law

1 6


• Aimed at telling how a government organization is performing over time

• Agencies compete with their past record• Ongoing, periodical information• Measure performance in different:

– Dimensions (effectiveness, efficiency, economy, service quality)– Delivery levels (process, output, outcome)

• Need reliable data collection methods• Support from strategic plan, management information system (PMG)• Started 1993 as a pilot, since 2001 is part of the budget preparation• Incrementalist approach, starting on voluntary basis• Budget-related (public agencies presents their indicators and its goals

during budget preparation)• Disclosure policy: Congress and general public

1 7


2001 2002 2003 2004 2005 2006 2007 2008Performance IndicatorsNumber of Agencies 72 109 111 132 133 136 139 142Number of indicators committed 275 537 1.039 1.684 1.588 1.552 1.445 1.443Average Indicators per agency 3,8 4,9 9,4 12,8 11,9 11,4 10,4 10,2% of output and outcome indicators * * * 70% 75% 81% 89% 89%% of Indicators evaluated 59% 73% 92% 94% 98% 100% 99,9% -Average % of achievement 80% 69% 76% 86% 88% 88% 93% -

Coverage and Quality of Performance Indicators

1 8


• Assess programs against their stated aims and expected results on their beneficiaries

• Requirements: relevance, independence, timeliness, transparency, public, reliable, efficient.

• Agreement with Congress and Financial Management Law and its Regulation (2003)

• Started 1997 based on logical framework methodology (EPG)• Budget-related (the results of the evaluation are taking into account in the

budget preparation)• Programs selected with Congress• Performed by independent evaluators through panels of experts or universities

and consulting firms.• Counterpart in ministries/agencies in charge of programs• Impact evaluations (2001)• Agency evaluations (2002)• Delivered in 6 to 12 months• Reported to Budget, Congress and the public• Commitments to incorporate recommendations from the evaluation• Follow up of commitments compliance

1 9

Number of Programs and Agencies Evaluated

Línea de Evaluación1997-

992000 2001 2002 2003 2004 2005 2006 2007 2008 Total

Evaluation of Public

Programs (EPG)

- N° of Programs 80 18 19 15 13 13 16 13 14 16 217

Impact Evaluations (EI)

- N° of Programs 2 4 4 14 6 7 (10) (12) 59

Comprehensive Spending Reviews (ECG)

- N° of agencies 8 2 4 5 2 4 (7) 32

Total Programs/Agencies 80 18 21 27 19 31 27 22 28 35 308

( ): On progress.


1 10

(2000 - 2008)

Evaluation of Programs (EPG - Impact)

EFFECTS % PROG.

1.- MINOR ADJUSTMENTS

2.- MODIFICATIONS IN THE DESIGN OF THE COMPONENTS AND PROCESSES

3.- MAJOR REFORMULATIONS

5.- PROGRAM CANCELLED OR COMPLETELY REFORMULATED

23%

37%

27%

7%

TOTAL PROGRAMS (174) 100%

6%4.- INSTITUTIONAL RELLOCATION


1 11

New Line of Evaluation: Evaluation of New Programs (EPN).

To assure that Chile remains in the leading ranks of countries with systematic evaluation processes by updating evaluation procedures and processes to world frontier levels.

11


i) Designing of the evaluation at the beginning of each new program

ii) Establishing control groups for the evaluation, based on randomized trials whenever is possible

iii) Establishing an international advisory committee to periodically review and assess the process of evaluation

iv) Implementing this new line gradually

1 12

PRESENTATION OF PROGRAMS TO BUDGET

• Started 2000 with preparation of 2001 Budget• Standard format to submit public programs for funding (2005 – 2009)• Ministries submit new programs and existing programs with importance budget increases

• Submission based on logical framework matrix (aims, goals, expected

results, components, indicators, target population)

• Finance makes proposal on basis of quality, consistency with government

priorities and strategies, President makes final decision

• Provides better base line for future impact evaluation

• 2008: Budget Direction gives technical assistance (new Department in the Management Control Division)

• 2009: Review of all programs presented to budget

1 13

COMPREHENSIVE MANAGEMENT REPORTS

• Started 1997, from Executive-Congress agreement

• Conceived as accountability mechanism

• Report performance against stated mandates, goals, commitments, resources

• Supports compliance with other management control mechanisms (performance indicators, program evaluation, management improvement programs) and financial management regulations

• Standard report format defined by Budget Office

• Distributed to Congress, publicly available

• Basis for gathering information for the evaluation stage of the budget cycle

1 14

• The achievement of management objectives are associated to a monetary incentive to all employees in the public institution (Law 19.553, 1998).

• The percentages of incentives are:

MANAGEMENT IMPROVEMENT PROGRAM (PMG)

% of Incentive for Institutional Performance

2007 - 2010

% of Achievement of Management Objectives

2007* 2008 2009 2010

5,7% 6,3% 7,0% 7,6% 90% - 100%

2,85% 3,15% 3,5% 3,8% 75% - 89%

0% 0% 0% 0% < 75%

* Before 2007 the % of incentives were 5% and 2,5%. The modification was done by Law 20.212, august 2007.

1 15

The PMG has four period of development clearly defined. In each of these periods the accomplishment of the management objectives committed was associated to:

Performance indicators. Period 1998/2000. Management Systems established in a Basic Framework Program

(2001). Management Systems established in a Advance Framework

Program including the requirements of ISO 9001:2000 (2005) Framework Program of Quality – certifying the processes involved

in the provision of good and services by ISO 9001:2000 standard


1 16

Basic Framework Program

Advance FrameworkProgram

Framework Program of Quality

Public Management Areas and Systems

(2001)

Public Management Areasand Systems

defined in the Basic Framework Program certified by

ISO 9001:2000 standard(2005)

Management Quality System implemented and processes of provision

of good and services certified by ISO 9001:2000 standard

(2009)

16


1 17

Basic Framework Program5 areas and 11 management systems with 4 or 6 development stages. The evaluation of the accomplishment of each stage is done by the Ministries of Finance, Internal Affairs and Presidency with the help of an Expert Network (public institutions in charge)

Advance Framework ProgramThe advanced framework program incorporates the objectives of the Basic Framework Program and the requirements of ISO 9001:2000 standard, in order to certificate each PMG system. Four stages: preparation to certification, certification and maintenance of certification for two years.


1 18

EXAMPLEObjectives of Management

Development Stages Areas Systems

I II III IV V VI VII VIII IX X

Priority Weight

Training o high 11% Performance Evaluation o medium 9% Human

Resources Hygiene, Security and Improved work environment

o medium 10%

Integral Customer Service system

o high 15% Customer Service

e- goverment o high 11% Internal audits o high 11%

Territorial management

o high 12% Management Planning and Control Planning/ Management

Control low 5%

Government Procurement o medium 10% Financial Management Financial Accounting

o medium 10%

Gender focus Gender focus o low 5%

o


1 19

• Created by the Budget Law 2008. • Budding fund finances the elaboration of proposals to

solve public management problems in some agency or program.

• The proposals can be presented by public agencies or by universities, consulting firms or private consultants.

• In January 2008 was the first bidding process. 66 proposals were submitted and 17 were selected to be developed during the first semester of 2008.

• 7 from public agencies and 10 from universities, consulting firms and private consultants.

• The budget 2009 considers resources to implement 5 of these 17 proposals.

19

PUBLIC MANAGEMENT MODERNIZATION FUND

END OF PRESENTATION

www.dipres.cl / Control de Gestión

http://www.dipres.cl/

system of evaluation and management control results-based budgeting the chilean experience

Documents