1

Not for Citation – Work in Progress

Can a Useful Aid Effectiveness Index Be Developed using the Paris Declaration

Framework?1

1 Introduction

In the present overseas development assistance (ODA) realm, the Paris Declaration on Aid

Effectiveness (PD) has brought forth both the donors and recipients on one platform to

reform the way they deliver and manage aid jointly. As governments of both donors and

recipient countries shares similar commitments to make aid more effective, this paper

attempts to evaluate how well this aid effectiveness is being measured and if it could be done

in a much better and effective way? To answer this question the paper seeks to establish an

Aid Effectiveness Index (AEI) to determine the donors’ and recipients’ overall aid

effectiveness commitment and progress towards its achievement. After four years of PD

adoption, questions arise as whether it is delivering in its objectives and if yes, than to what

extent and at what rate donors and recipients have made such progress? The most recent

OECD survey on monitoring the PD reveals that ‘some progress has been made, but not

enough’ and ‘without further reform and faster action, it will be impossible to meet the 2010

targets for improving the effectiveness of aid’ (OECD, 2008a, 3). The AEI constructed in this

paper attempts to represent the facts and figures of the progress being made as well as

demonstrate the way donors and recipients deliver or manage aid compared with their peers.

As was stated earlier, the progress is not fast enough and the AEI may be helpful in seriously

gearing up recipient countries and their external partners’ efforts to meet their international

commitments and targets for effective aid by 2010 and beyond (OECD, 2008b). Further, the

AEI can also become a useful advocacy tool for aid policy makers in different donor agencies

to assess their aid effectiveness efforts and international comparisons.

The paper is divided into six sections. The introductory section sets the background of the

research paper, while the second section is a literature review which examines the

contemporary debate on development indices. Section three establishes the methodology to

construct the AEI using the PD as a theoretical framework of aid effectiveness. Section four

explore whether Ownership, Alignment, Harmonization, MfDR and Mutual Accountability

indices can be separately developed or not. Section five consolidates the composite AEI and

finally section six offers the conclusion of the study.

What constitutes Aid Effectiveness?

In spite of the fact that aid effectiveness is considered to be an evolving paradigm, the larger

question remains - what constitutes aid effectiveness? PD is synonymous to aid effectiveness,

but is aid effectiveness defined or explained anywhere in the declaration? What exactly is aid

effectiveness according to PD? According to Stern et al, there is nowhere in the PD where aid

effectiveness is defined or explained and only in its opening statement of resolve, it is

expressed that ‘in order to accommodate increases in aid volume, aid effectiveness must

increase significantly’ (2008, 19). Three years after the declaration, different elements of PD

are put together by them and they defined aid effectiveness as an ‘arrangement for the

planning, management and deployment of aid that is efficient, reduces transaction costs and

is targeted towards development outcomes including poverty reduction’ (ibid, 20). Although

the recent donors and recipients resolve on aid effectiveness fits within the new and improved

1 Pranay Kumar Sinha, University of Birmingham (pranayatbham@gmail.com)

2

aid orthodoxy, some inherent limitations are identified in the PD. The PD process is

considered to have taken a narrow view of aid reform process and broader concerns on aid

inefficiency are still not completely addressed.

While the PD is widely agreed by donors and recipients as an aid effectiveness framework, it

is being criticised as bringing concerned stakeholders in an ‘extremely asymmetrical

conditions’ and ‘failing to provide institutional mechanisms to address the asymmetries in

power’ (Global Call to Action Against Poverty, 2008). Legitimacy of institutional ownership

of the PD is questioned as it rests within the OECD DAC member countries and the World

Bank (WB). As donors and creditors have exclusive or majority control over the OECD and

the WB governance and developing countries have little or no voice or vote, the aid reform

process is being challenged and its being proposed to move the aid reform process to United

Nations Development Cooperation Forum, which is a more representative institution than the

OECD’ (ibid; Action Aid, 2007, 3). In a recently held debate at Overseas Development

Institute (ODI), Matthew Martin2 recommended exploring the potential role of the United

Nations in making aid architecture more participatory as ‘it may be better equipped to engage

a wider range of stakeholders’ especially southern donors (ODI, 2009).

The PD is often being termed as moving ahead towards bringing aid effectiveness but with ‘a

flawed process’ (Action Aid, 2008, 5). Consultation with southern governments and civil

society was being termed ‘closed’, ‘not transparent’, and ‘token’ when few civil society

organizations raised concerns that despite being listed as participating organizations in the PD,

they have never endorsed the declaration (GCAP, 2008; Action Aid, 2007, 3). Hence civil

society has called for organizing real and substantive, not token consultation processes.

Attempts at making international aid architecture more participatory have not gone far

enough because ‘southern donors, private actors (e.g. NGOs) and global ‘vertical funds’ and

foundations have been left out of debates, despite the fact that they comprise some 25% of

global aid’ (ODI, 2009).

Further, PD is largely silent on the issue of aid allocation. It fails to recognise ‘to develop an

effective and transparent international mechanism to improve aid allocation so it goes to

those most in need, which is not misdirected geographically according to donor’s foreign

policy objectives’ (Action Aid, 2007, 12). Instead of matching the poverty reduction needs,

the aid inflows are mostly directed to favour strategically or historically important countries,

near neighbours and issues selected by donors (ibid, 13). In order to overcome this weakness

and ensure allocation of aid in a fair and transparent way, it is being advocated that ‘aid be

allocated primarily on the basis of need, with all low income countries guaranteed a

minimum annual resource transfer linked to their population and level of poverty’ (ibid, 13).

Despite aforesaid weakness, the PD signed at the high level forum in 2005 resembles the

international consensus as well as commitment to reduce aid inefficiency and provide a time-

bound framework to measure the progress on how to achieve it. For this reason, the paper

integrates the five principles of PD i.e. Ownership, Alignment, Harmonization, Managing for

Development Results (MfDR) and Mutual Accountability (MA) as a basis for constructing

the AEI.

Methodology and Limitations of the Study

The research was pre dominantly a desk-based. Academic literature as well as policy

documents was reviewed and analysed to examine the evolution and theoretical aspects of

development indices. Existing debates and critiques as well as issues critical for successful

2 Director, Debt Relief International

3

index construction were reviewed to gain more insight and identify checks and balances

while constructing the AEI. Data collected by OECD during 2006 and 2008 surveys on

monitoring the PD was used as a quantitative dataset to establish the AEI. OECD’s country

surveys/ reviews, survey guidelines, questionnaires and speeches formed an integral part of

analysis for this research. Further, an analysis of the different dimensional indices and their

adequateness was undertaken to decide whether a consolidated index can be developed or not.

The study is limited to 33 recipients and 31 donors who participated both in the first and

second round of monitoring undertaken in 2006 and 2008. Hence, the study is only

representative in nature but deploys the best available comparable data for interpretation.

2 Indices for Development: Approaches and Issues

This section focuses on approaches and issues relevant to constructing the development

indices. It attempts to identify the existing debates and critiques of different development

indices and issues which are critical for successful index construction. Finally a list of checks

and balances shall be identified which shall be useful during the construction of the AEI in

the subsequent sections.

Evolution of Development Indices

There has always been a sustained interest among academicians and policy makers to reflect

development effectiveness through cross-country indices measuring various aspects of

development. The evolution of development indices dates back to 1966 when United Nations

Research Institute for Social Development (UNRISD) first published the Level of Living

Index and subsequently the Development Index in 1972 (Kambhampati, 2004, 21). After that,

the Physical Quality of Life Index was proposed by Overseas Development Council in 1979

(Siggel, 2005, 14).

The United Nations Development Programme’s Human Development Index (HDI) first

published in 1990 and the WB’s Worldwide Governance Indicators (WGI) started in 1996 are

two such examples which are widely recognised to have established a new development

paradigm (UNDP, 2008; WB, 2009). A few other examples of development indices are the

Transparency International’s (TI) Corruption Perceptions Index (CPI) first released in 1996,

the Centre for Global Development’s (CGD) Commitment to Development Index (CDI)

started in 2003 and the Failed States Index (FSI) established by the Fund for Peace (FFP) in

2005 (TI, 2009; CGD, 2008; FFP, 2009). Latest in these additions are the Ibrahim Index of

African Governance which focuses on the quality of governance in sub-Saharan Africa and

the Save the Children’s Child Development Index for holding governments to account for

children’s wellbeing, both being first published in 2008 (Mo Ibrahim Foundation, 2008; Save

the Children, 2008).

The purpose of these indices can broadly be grouped into three categories. Firstly, to capture

the attention of policy makers, media and civil society organizations and steer it away from

the hardcore economic statistics to focus instead on developmental outcomes; secondly to

question the national policy choices by asking how two countries can experience such

different development outcomes, and lastly; to highlight wide differences within and/or

between countries (UNDP, 2009). This paper attempts to establish an AEI on the similar line

of the above mentioned initiatives and explore whether the AEI can offer any new insights,

which pre-existing PD survey reports alone cannot reveal.

Development Indices: Theoretical Aspects

4

There is an inherent inter-linkage between data, indicators, indices and information (Figure 1).

Data is the most basic component for indicators, indices as well as information. Indicators,

which are derived from data, are considered to be the most basic tool to analyse change in

certain dimension. Further indices can be created either by two or more indicators or from the

combination of several data. The most important aspect of this inter-linkages is the fact that

‘indicators, or indices, are not the end in themselves - they are the means to an end, consisting

of improved decision-making’ (Segnestam, 2002, 4).

Figure 1 From Data to Information

Source: Segnestam, 2002

Indices are supposed to ‘convey a particular message in a simpler, more easily digestible

form than simply, referring to an array of non-aggregated indicators’ (Stapleton and Garrod,

2008, 461). ‘The theory behind the development of indicators and indices can be visualized

as in Figure 2A, with a broad base of good quality primary data on which the indicators and

indices can rest on. To show a simplified picture of reality, this information pyramid is

commonly turned upside down (Figure 2B) with many indices developed using the limited

data sets that currently exists’ (Segnestam, 2002, 17).

Figure 2 The Information Pyramid

Source: Segnestam, 2002

An attempt to classify different composite indices of development can be made on the basis

of seven dimensions (i.e. content, technique and method, comparative application, focus,

clarity and flexibility, simplicity and availability), although these dimensions (Table 1) do not

fully address all the methodological and conceptual issues involved in measurement

(Booysen, 2002, 116).

Table 1 Dimension for Classifying Composite Indices of Development

Dimension Description

i. Content

i. What aspects or facets of development does the

index measure?

5

ii. Technique and

method

iii. Comparative

application

iv. Focus

v. Clarity,

simplicity and

flexibility

vi. Availability

ii. Selection of variables and components, scaling of

variables, weighting and aggregation of component

indices, validation of composite indices of

development

iii. Does the index compare the level of development

(a) across space (‘cross-section’) or time (‘time-

series’), and (b) in an absolute or relative manner?

iv. Does the index measure development in terms of

input (‘means’) or output (‘ends’)?

v. How clear, simple and flexible is the index in its

content, purpose, method, comparative application

and focus?

vi. How readily available are data on the particular

index across time and space?

Source: Frederik Booysen (2002)

Development Indices: Existing Debates and Critiques

The debates surrounding indices shows that they have progressed from being merely objects

of ‘academic curiosity’ to tools for conducting development dialogue, allocating external

assistance and influencing foreign direct investment’ (Iqbal and Shah, 2008, 1) and is clearly

evident in the example of the WGI. Both the influential development indices i.e. HDI and

WGI have been scrutinised very closely since they were first published. Desai (1991),

Doessel and Gounder (1991), Srinivasan (1994), Hicks (1997), Thomas (2006), Arndt and

Oman (2006), Kurtz and Schrank (2007), Thompson and Shah (2005), Iqbal and Shah (2008)

and Stapleton and Garrod (2008) are some of the index analysts that have raised attention on

methodology and measurement of such indices.

On the basis of existing literature, Booysen has summarised seven critiques that are levelled

at composite indices (2002, 138-141). Exclusion of one or more essential components of

development in particular indices; quantification of particular components of indices with the

aid of different variables, inability to reveal anything that a single variable alone cannot

reveal, ad hoc selection based on political motive and ideological basis, the accuracy and

comparability of the data employed, the weighting and aggregation techniques employed and

the supposed lack of practical value of composite indices forms part of those seven critiques.

In the case of the WGI, correlated errors, sample bias and lack of transparency, comparability

of indicators over time and across country (Arndt and Oman, 2006), perceptual bias, time

inconsistency of definitions and measurements, lack of conceptual framework and sample

selection bias (Kurtz and Schrank, 2007) and lack of proper understanding of ‘construct

validity’ (Thomas, 2006) are a few of the criticism raised by index analysts. Thus the

questions regarding the usefulness of the indicators for making comparisons over time and

across countries, various sorts of biases in the individual indicators and independence of the

assessments and the consequences for the aggregate indicators were raised (Kaufmann et al,

2007, 3). Issues with definition of a somewhat obscure term, concerns primarily regarding of

access to the data used in the index and the causality between indicators used in the index

were other concerns raised in WGI (ibid, 3). Though these criticisms were not considered as

‘particularly compelling’ and were considered by Kaufmann to be ‘based on

misinterpretations of indicators or the empirical evidence involving these indicators’, the

6

nature of debates and issues surrounded in construction of any development index were

nevertheless exposed (ibid, 3).

On the basis of the aforementioned points, Iqbal and Shah (2008) broadly classified these

issues into measurement and methodological ones. Problems in measurement include

ideological as well as objective biases and flawed judgments of primary sources like asking

ambiguous questions and flawed composition of respondents. Problems with cross-country

comparisons are due to unbalanced samplings and abrupt changes in weighting which leads

to high levels of measurement errors. Problems of methodology includes issues related to

equal weighting, assumption of constant global averaging, correlated errors, country specific

weighting for unbalanced sample, dominance of few and non representative sources. Lack of

transparency of data sources also forms part of technical issues.

Birdsall and Roodman (2003) shared challenges they encountered in index design and

identified the tensions between policy efforts vs. impact, timeframe, standardization of scores

and the weighting problems while designing CDI in 2003 (2-4). CDI’s aspirations to measure

policies are often compromised because of a lack of data across all the components.

It has happened that even when the scope/definition of a certain dimension was recognised

and appreciated by index analysts, concerns were raised by critics regarding the process of

translating data into an index (Sagar and Najam, 1998, 251). This suggests that the flaw that

persists in its translation can be overviewed during the process, in the invoked assumptions

and in what is being ignored thus resulting in presenting a distorted picture (ibid).

Sen argued that constructing a one-number index on human development would be ‘quite

vulgar because of the inescapable over-simplification’ (Fakuda-Parr and Kumar, 2004). But

vulgarity and over-simplification were traded off on the premise that the one-number HDI

would not be as blind to social aspects of human life as Gross National Product (GNP). That

is why indices could be over simplified and sometimes extremely misleading, but

nevertheless very important (Dervis, 2005). Thus an index can be nevertheless favoured

because it shows the inadequacy of other indices (HDI over GNP), or addresses the

distributional aspects (HDI over per capita income) or aspects of human development which

other indices neglect (Noorbakhsh, 1998).

Constructing Indices: Policy Processes and Technical Requirement

As discussed in the previous section, construction of any development index involves a

diverse range of issues and they can be broadly grouped into two categories, namely policy

requirements and technical requirements. Though these conceptual and technical

requirements related to constructing indices are often debated and criticised at policy and

academic sphere, the positive side of such policy debates and criticism have resulted in the

continuous improvements in those indices over a period of time. For instance, the HDI has

been modified to respond to such positive criticism on at least four occasions (Stanton, 2007,

16). The following table summarizes desirable properties of international indices which are

divided into requirements for the policy process and technical requirements.

7

Table 2 List of Desirable Properties of International Indices

Requirement for the Policy Process Technical Requirements

i. policy-relevant

ii. sensitive to inequality

iii. adequate (“answering the right

questions”)

iv. communicable to policymakers

and the general public

v. replicable

vi. backed by theory/scientific

concept

vii. available for a large number of

countries

viii. suitable for annual (or at least

biannual) updates

i. comparable over time

ii. comparable across countries

iii. based on valid methods of measurement

iv. non redundant (information not already

captured in components/ simpler

indicators)

v. able to differentiate among countries

vi. robust against measurement error and

moderate changes in aggregation

function

vii. stable against random fluctuations

viii. sensitive to changes over time

Source: Doris Wiesmann (2006, 60)

Issues Critical for Successful Index Construction

Four issues critical to successful index construction identified by Diamantopoulos and

Winklhofer are content specification, indicator specification, indicator collinearity, and

external validity (2001, 271). Content specification includes specification of the scope of the

variable or the domain of content the index is intended to capture. Indicator specification

expects indicators to be chosen randomly from the universe of items relating to the construct

of interest which means that the items used as indicators must cover the entire scope as

described under the content specification (ibid, 271). Indicator collinearity means the stability

of the indicator coefficients is affected by the sample size and strength of the indicator

intercorrelations and the indicators are either retained for initial inclusion or removal for the

index. External validity calls for the removal of invalid indicators from the index. On the one

hand, considerations from a theoretical perspective calls for elimination of indicators but

carries the risk of changing the construct itself whereas on the other hand, from a practical

perspective, an excessive number of indicators is undesirable because of the demands it

imposes on the data collection and the increase in the number of parameters when the

construct is embedded (ibid, 272).

In addition to the considerations above, successfully administering the development index is

also an important aspect which includes technical backstopping for annual publication in

subsequent years as well as retrospective update of result from previous years and

improvement in the methodology. Such a re-calculation may occur, for example, when new

components are included.

8

Pros and Cons of a Development Index

Table 3 Pros and Cons of a Development Index

Pros Cons

i. An Index can be used to

summarize complex or

multi-dimensional issues

supporting decision makers

ii. An Index provides the big

picture. They can be easier

to interpret than trying to

find a trend in many

separate indicators. They

facilitate the task of

ranking countries on

complex issues.

iii. An Index can help attract

public interest by

providing a summary

figure with which to

compare the performance

across countries and their

progress over time.

iv. An Index could help to

reduce the size of a list of

indicators or to include

more information within

the existing size limit.

i. An Index may send misleading, non-robust

policy messages if they are poorly

constructed or misinterpreted. Sensitivity

analysis can be used to test index for

robustness

ii. The simple ‘big picture’ results which index

show may invite politicians to draw

simplistic policy conclusions. Index should

be used in combination with the sub

indicators to draw sophisticated policy

conclusions.

iii. The construction of index involves stages

where has to be made: the selection of sub

indicators, choice of model, weighting

indicators and treatment of missing values

etc. These judgements should be transparent

and based on sound statistical principles.

iv. There could be more scope for disagreement

among member states about index than on

individual indicators. the selection of sub-

indicators and weights could be the target of

political challenge

v. The index increases the quantity of data

needed because data are required for all the

sub-indicators and for a statistically

significant analysis

Source: Saisana and Tarantola quoted in OECD (2006, 361)

Checks and Balances for Construction of the Aid Effectiveness Index

On the basis of the existing debates and critiques in the policy and academic sphere regarding

the indices, it is acknowledged that awareness of following checks and balances would

significantly reduce the chances of errors while moving ahead with construction of the AEI.

They can broadly be classified under design check, reality check and equity check.

A Design Check

i. What is the underlying aid effectiveness framework and if the PD framework

qualifies for the same? Does it capture critical aspects from a donor and recipient

perspective relevant for cross-country comparisons?

ii. Whether each of its dimensions are equally essential in determining the level of

aid effectiveness or if a few more dimensions are required to be added or few

dimensions can be dropped/substituted?

9

iii. Whether equal weighting should be given to the five variables or not? What are

the alternatives?

iv. What are the conceptual implications of the current method for folding the five

component indices into a single index?

v. Does this method mask trade-offs between various dimensions or should the index

compensate in one dimension for deficiency in another?

vi. Whether indicators correctly reflect the dimensions or not? Whether more

indicators should be added and/or some should be reduced/substituted?

vii. What are the causal relationships among different dimensions of aid

effectiveness?

viii. Whether the composition of the index is flawed if it is significantly and positively

correlated with each of its component variables individually?

A Reality Check

i. Whether the AEI is good enough to reflect the existing aid effectiveness realities?

ii. Is it possible that aid effectiveness for sample donors or recipients are not as high as

the AEI suggests and the gaps between and among donors or recipients are larger than

the AEI portrays or vice versa? If yes, then what are the alternatives that would

portray a more realistic picture than the AEI does?

iii. Whether the AEI would be rather judged or considered to be as yet another redundant

composite development indicator by critics?

An Equity Check

i. Does the formulation of AEI portray the global inequities existing in the donor-

recipient relation?

ii. Will the AEI continue to ignore the inequities which exist in aid allocation and

distribution or could it reflect such inequities?

3 Constructing the Aid Effectiveness Index

The present section focuses on defining the five dimensions used for constructing the AIE

and different source of data used. It is followed by establishing an AIE framework for

calculating the different dimensional indices and possible aggregation methodology to

construct the AEI that shall be used in the subsequent sections.

The Aid Effectiveness Index

The AEI is based on measurement of five dimensions of the PD -

i. Ownership – capturing the extent of ownership exercised by developing countries as

measured on the basis of their operational development strategies.

ii. Alignment – capturing the extent that donors are basing their overall support on

partner countries’ national development strategies, institutions and procedures.

Donors usage of country system is based on reliability partner country system which

is translated in the donors practice of predictability of aid, aid untying, avoidance of

10

parallel implementation structures, aid flows on national priorities and strengthening

capacity by coordinated support.

iii. Harmonisation – capturing the extent of harmonization achieved by donors as

measured by common arrangement and shared analysis.

iv. Managing for Development Results – capturing the extent of efforts put forth by

both donors and recipients while managing resources and improved decision-making

v. Mutual Accountability – capturing the extent of efforts put forth by both donors and

recipients to hold each other mutually accountable for development results.

Data Sources

Data is primarily derived from the OECD’s baseline survey for the 12 indicators in 2006 and

the monitoring survey conducted in 2008. Data for the AEI is based on five sources which are

an integral part of the PD monitoring framework. These sources are

i. Desk review based on the WB Comprehensive Development Framework (CDF)

analysis which includes analysis of twenty indicators and draws on a range of sources

of information. It includes country-specific long-term vision statements, consultative

group and roundtable meeting documents, Poverty Reduction Strategies (PRS) with

associated progress reports and the assistance strategies of the main development

agencies active in each country, etc.

ii. Desk review based on the WB Country Policy and Institutional Assessment (CPIA)

which includes sixteen indicators, one of which - CPIA indicator 13 - measures the

quality of partner countries budget and financial management systems.

iii. Assessment of national procurement systems developed by OECD-DAC Joint

Venture on Procurement (JV-PROC)

iv. OECD-DAC monitoring of untied aid where data were drawn from reporting by DAC

donors to the annual DAC Questionnaire on untied aid.

v. Donors and recipients questionnaire at the country level

The specific data sources used in each indicator for the PD baseline and monitoring survey

are summarised in Table 4.

Table 4 Data Sources for the Paris Declaration Indicators

Data Sources Indicators

2006 Baseline survey 2008 Monitoring Survey

1 Desk Review based on the WB

CDF analysis Desk Review based on the WB CDF

analysis

2(a) Desk Review based on the WB

CPIA Desk Review based on the WB CPIA

2(b) Not conducted Assessment of National Procurement

Systems developed by OECD-DAC

JV-PROC

3 Donor’s and Government

Questionnaire Donor’s and Government

Questionnaire

11

4 Donor’s Questionnaire Donor’s Questionnaire

5(a) Donor’s Questionnaire Donor’s Questionnaire

5(b) Donor’s Questionnaire Donor’s Questionnaire

6 Donor’s Questionnaire Donor’s Questionnaire

7 Donor’s and Government

Questionnaire Donor’s and Government

Questionnaire

8 OECD-DAC Monitoring of untied

aid OECD-DAC Monitoring of untied aid

9 Donor’s Questionnaire Donor’s Questionnaire

10(a) Donor’s Questionnaire Donor’s Questionnaire

10(b) Donor’s Questionnaire Donor’s Questionnaire

11 Qualitative Assessment based on

the WB CDF analysis Qualitative Assessment based on the

WB CDF analysis

12 Donor’s Questionnaire Government Questionnaire

Source: OECD (2006b) and OECD (2008a)

12

Aid

Effectiv

eness In

dex

Calculating the Aid Effectiveness Index

Figure 3 Aid Effectiveness Index Framework

Dimension/Component Indicator Indicator Dimension

Index Index

Ownership 1 Partners have operational development strategies ODS Index Ownership

Index

2a Reliable PFM systems Reliable Country 2b Reliable Procurement systems System Index

3 Aid flows are aligned on national priorities AFANP Index

4 Strengthen capacity by co-ordinated support SCCS Index

Alignment 5a Use of country public financial management systems Use of Country Alignment 5b Use of country procurement systems System Index Index

6 Avoiding parallel implementation structures Avoidance of PIU Index

7 Aid is more predictable Predictability of Aid Index

8 Aid is untied Untying Aid Index

9 Use of common arrangements or procedures Common Arrangement

Index

Harmonization Harmonization

10a Missions to the Field Shared Analysis Index Index

10b Country Analytic Work

MfDR 11 Result Oriented Frameworks ROF Index MfDR Index

Mutual Accountability 12 Mutual Accountability MA Index MA Index

Aid

Effectiv

eness In

dex

A

id E

ffectiven

ess Ind

ex

Aid

Effectiv

eness In

dex

Source: Researcher’s Framework

13

The above diagram (Figure 3) summarizes how the AEI will be constructed. Since aid

effectiveness is considered to be an inter-play between donors-recipients interaction, the

proposed AEI is proposed to be constructed for donors as well as recipients separately.

Aggregation Methodology

On the basis of the existing five dimensions and twelve indicators, following options of

aggregation methodology can be followed to construct the AEI –

Option 1: Principle of Equal Weighting of all the Five Dimensions

The AEI is a simple average of the Ownership Index, Alignment Index, Harmonization Index,

MfDR index and Mutual Accountability index, and is obtained by dividing the sum of these

five indices by 5. So,

Where,

i. The Ownership index = operational development strategies index

ii. The Alignment index = 1/7 (Reliable Country System Index + Aid flows are Aligned

on National Priorities (AFANP) index + Strengthen Capacity by Co-ordinated

Support (SCCS) index + Use of Country System Index + Avoidance of Project

implementation Unit (PIU) Index + Predictability of Aid Index + Untying Aid Index)

a. The Reliable Country System Index = ½ (Reliable Public Financial

Management (PFM) System Index + Reliable Procurement System Index) and

b. the Use of Country System Index = ½ (Use of Country PFM System Index +

Use of Country Procurement Systems Index)

iii. The Harmonization Index = ½ (Common Arrangement Index + Shared Analysis

Index)

a. The Shared Analysis Index = ½ (Mission to the Field Index + Country

Analytic Work Index)

iv. The MfDR index = Result Oriented Framework (ROF) index

v. The Mutual Accountability index = Mutual Accountability Index

This option is based on the principle of equal weighting of all the five dimensions and gives

equal importance to all the dimensions of the AIE Framework i.e. Ownership Index,

Alignment Index, Harmonization Index, MfDR Index and Mutual Accountability Index

(Figure 3).

Option 2: Principle of Equal Weighting of all the Twelve Indicators

The AEI is a simple average of the twelve indicators of Ownership Index, Alignment Index,

Harmonization Index, MfDR Index and Mutual Accountability Index, and is so derived by

dividing the sum of these twelve indices by 12. So the AEI = 1/12 (ODS Index + Reliable

Country System Index + AFANP index + SCCS index + Use of Country System Index +

14

Avoidance of PIU Index + Predictability of Aid Index + Untying Aid Index + Common

Arrangement Index + Shared Analysis Index + ROF Index + Mutual Accountability Index).

This option is based on the principle of equal weighting of all the twelve indicators

irrespective of their position in individual dimensions and gives equal importance to all the

indicators of the AIE Framework (Figure 3).

Goalposts for Calculating the Aid Effectiveness Index

Like the HDI, ‘the performance in each dimension is expressed as a value between 0 and 1 by

applying the following general formula’ (UNDP, 2008, 356)

Before the AIE is calculated, an index needs to be created for each of the dimensions stated

above. To calculate these indices, minimum and maximum values (goalposts) are chosen for

each indicator (Table 5).

Table 5 Goalposts3 for Calculating the Aid Effectiveness Index

Paris Declaration Indicators Maximum Value Minimum Value

1 Partners have operational development

strategies 5 1

2a Reliable PFM systems 6 1

2b Reliable Procurement systems 4 1

3 Aid flows are aligned on national priorities Highest value obtained

by donors or recipients

Lowest value obtained

by donors or recipients

4 Strengthen capacity by co-ordinated support As Above As Above

5a Use of country public financial management

systems As Above As Above

5b Use of country procurement systems As Above As Above

3 The goalpost for indicators 1,2a, 2b and 11 is based on the highest and lowest value obtained by recipients in

the PD survey. For example Afghanistan’s National Development Strategy (indicator 1) incorporates some

elements of good practice in 2007 that’s why it falls under D category and hence scored 2 (Annexure 2). And,

Zambian National Development Strategy is largely developed towards achieving good practice in 2007 that’s

why it falls under B category and hence scored 4 (Annexure 4).

The goalpost for indicators 3, 4, 5, 7, 8, 9,10a and 10b is based on the highest and lowest value obtained by

donors and recipients in the PD survey. For example 86% (highest value among the donors) of the ADB aid

flows are aligned to national priorities in 2007 that’s why it is the maximum value for indicator 3 and 0%

(lowest value among the donors) of the Gavi Alliance’s aid flows are aligned to national priorities in the same

year that’s why it is the minimum value for indicator 3 (Annexure 12).

The goalpost for indicators 6 is based on the lowest and highest value obtained by donors and recipients in the

PD survey. For example UNDP has 296 parallel implementation structures operational in 2007 and Gavi

Alliance, Ireland, New Zealand and Portugal have 0 parallel implementation structures operational in the same

year (Annexure 12). Since the target is to avoid the parallel implementation structure that’s why highest the

number of PIU, lowest is the performance and vice versa.

The goalpost for indicator 12 is based on ‘yes’ or ‘no’ value in the PD survey.

15

6 Avoiding parallel implementation structures Lowest value obtained

by donors or recipients

Highest value obtained

by donors or recipients

7 Aid is more predictable Highest value obtained

by donors or recipients

Lowest value obtained

by donors or recipients

8 Aid is untied As Above As Above

9 Use of common arrangements or procedures As Above As Above

10a Missions to the Field As Above As Above

10b Country Analytic Work As Above As Above

11 Result Oriented Frameworks 5 1

12 Mutual Accountability 1 (Yes) 0 (No)

Source: Researchers Calculation based on OECD, 2008a

Sampling and Cross-Country Comparisons

Despite data being available for all the twelve indicators for 55 recipient countries, the AEI is

constructed on the basis of 33 recipient countries. To ensure the same set of cross country

comparisons (Wiesmann, 2006; Booysen 2002; Arndt and Oman, 2006; Kaufmann et al,

2007; Iqbal and Shah, 2008), these 33 recipient countries have been selected on the basis of

their participation in both the 2006 baseline and the 2008 monitoring survey. Further the data

are available for the 55 donors that took part in the 2008 monitoring survey but following

criteria are used to include 31 donors in the AEI -

i. ‘All donors that have reported over USD 100 million aid earmarked for the

government sector in at least three countries in the surveyed countries; and

ii. All donors who do not meet the first criteria but would like to publish their results in

the 2008 survey overview’ (OECD, 2008b, 91).

It is important to mention that the ‘progress for donors is measured for the same set of

countries which have recorded the donor’s aid in both the 2006 baseline survey and the 2008

survey’ thus to ‘allow for a comparison of progress in the same set of countries between 2005

and 2007 for each donor’ (OECD, 2008b, 91).

As the quantum of each donor’s ODA varies at different surveyed countries, the AIE

primarily uses the indicator value4 derived from the PD monitoring data. ‘The indicator

4 Illustrative Example

The following example illustrates how the indicator value and the average country ratio are calculated for three

countries.

%

=58%

16

value is a weighted average; based on each donor’s portfolio in the surveyed countries i.e.

each country is weighted by the volume of activity (OECD, 2008b, 102). It is important to

mention that it is not using average country ratio, ‘which is an un-weighted average’ and

‘which provides a comparative measure of the baseline, irrespective of the volume of activity

in each country; i.e. it gives equal weight to each country’ (ibid).

On the basis of the above principles, the next section will focus on constructing indices on

five different dimensions of the PD i.e. Ownership Index, Alignment Index, Harmonization

Index, MfDR Index and Mutual Accountability Index.

4.1 Can an Ownership Index be Developed? The Ownership Index is based on the first principle of the PD framework and is captured only

by one indicator. The indicator is defined as ‘partners having operational development

strategies - number of countries with national development strategies (including PRSs) that

have clear strategic priorities linked to a medium-term expenditure framework and reflected

in annual budgets’ (OECD, 2005a, 9). Since ODS Index is the only index to capture

ownership (see section 3), henceforth it is being termed as an ‘Ownership Index’.

Table 6 How far the Donors and Recipients are meeting the targets under Ownership

Principle?

Indicator 2005

Baseline

2007

Achievement

Distance to Cover

before 2010

2010

targets

1 Operational

Development Strategies 17% 24% 51% 75%

Source: OECD 2008 (b)

Countries with sound operational strategies have increased from 17% to 24%, but the aim is

to reach 75% by 2010 (Table 6). The annual rate of progress needs to be roughly five times

greater for the 2007-2010 period compared to the 2005-2007 period. As discussed earlier in

section 3, the Ownership Index from the recipient’s side is constructed and can be seen in

table 11.

4.2 Can an Alignment Index be Developed?

This subsection attempts to construct the second dimension of five dimensional AIE i.e.

Alignment Index after establishing and consolidating seven individual indices (see section 3).

While constructing the Reliable Country System Index, the limitations encountered shall be

explained as why the Reliable Procurement System Index is dropped from the Reliable

Country System Index.

Construction of an Alignment Index

Alignment Index is based on the second principle of the Paris Declaration framework and is

expected to be captured by seven indicators. The reliability of a country’s public financial

management and procurement systems and its usage by donors, alignment of aid flows on

national priorities, strengthening partner’s capacity by coordinated support, avoidance of

parallel implementation structures, predictability and aid untying are the seven indicators.

The respective progress on different indicators under the alignment principle (see section 3)

can be seen in Table 7.

In this example the indicator value is much lower than the average country ratio because of the high weight (87

out of 100 units) for Country C. (Source OECD, 2008b, 101)

17

Table 7 How far the Donors and Recipients are meeting the targets under the

Alignment Principle?

Indicator 2005

Baseline

2007

Achievement

Distance to

Cover

before 2010

2010 Targets

2a Reliable PFM systems Nil 36% 14% 50% of countries

improve score

2b Reliable country

procurement systems

Not

available Not available

Not

available

50% of countries

improve score

3 Aid flows are recorded

in country budgets 42% 48% 37% 85

4 Technical assistance is

aligned and co-ordinated 48% 60% Covered 50

5a Donors use country

PFM systems 40% 45% 35% 80

5b Donors use country

procurement systems 39% 43% 37% 80

6 Donors avoid parallel

PIUs 1817 1601 990 611

7 Aid is more

predictable 41% 46% 25% 71

8 Aid is untied 75% 88%5

Progress

over time Progress over time

Source: OECD 2008 (b)

Initially, the Alignment Index is expected to be obtained from the following formula

(discussed in section 3). Alignment Index = 1/7 (Reliable Country System Index + AFANP

index + SCCS index + Use of Country System Index + Avoidance of PIU Index +

Predictability of Aid Index + Untying Aid Index), where

a. The Reliable Country System Index = ½ (Reliable PFM system Index +

Reliable Procurement System Index) and

b. the Use of Country System Index = ½ (Use of Country PFM system + Use of

Country Procurement Systems)

Reliable Country System Index

Reliable Country System Index is measured by the ‘number of partner countries that have

procurement and PFM systems that either (a) adhere to broadly accepted good practices or (b)

have a reform programme in place to achieve these’ (OECD, 2005a). So, the Reliable

Country System Index is to be obtained from the reliable PFM System Index and the Reliable

5 OECD DAC has used then available latest data which relate to 2006 donor commitments and recorded the

progress over one year, rather than the usual two applied to the other indicators in 2008 survey report.

18

Procurement System Index with following formula – Reliable Country System Index = ½

(Reliable PFM System Index + Reliable Procurement System Index).

Reliable Public Financial Management System Index

Reliable PFM system Index is measured by the ‘number of partner countries that have PFM

systems that either (a) adhere to broadly accepted good practices or (b) have a reform

programme in place to achieve these’ (OECD, 2005a). Since 2005, 36% of countries have

improved their score (OECD, 2008b, 23) and the aim is to reach 50% by 2010 (Table 7). The

Reliable PFM System Index is based on the OECD dataset.

The reliable PFM System Index from the recipient’s side is constructed and can be seen in

Table 11.

Reliable Procurement System Index

The Reliable Procurement System Index is assessed by the degree to which existing

procurement systems that adhere to broadly accepted good practices or have a reform

programme in place to achieve these (OECD, 2005a).

The Reliable Procurement System Index from the recipient’s side is enclosed in Annexure 9.

The data for 2006 survey is not available as the methodology to assess this indicator was not

available at that time. Therefore it cannot compare the level of reliability of procurement

system across countries or over time6. It also fails to bring any practical value to the Reliable

Country System Index because the aspect of procurement already falls under the overall

ambit of PFM System Index, which is already captured.

On the basis of the constraints mentioned above, it has been decided not to construct the

Reliable Country System Index with following formula i.e. Reliable Country System Index

= ½ (Reliable PFM System Index + Reliable Procurement System Index). Capturing a

reliable procurement system would be a duplication of effort which is why it has been

decided to drop it from the Reliable Country System Index, instead capture the Reliable PFM

system Index as a default Reliable Country System Index.

Aid Flows are Aligned on National Priorities (AFANP) Index

The objective of the Indicator 3 of the Alignment Index is to promote budget realism and is

defined as aid flows aligned on national priorities (OECD, 2008b). It is measured by the

‘total volume of aid – not only budget support – recorded in countries annual budgets as a

percentage of donor’s disbursements’. The AFANP Index from the recipient’s side and

donor’s side can be seen in Table 12 and 13 respectively. Half of all the aid flows to the

government sector are still not recorded in country budgets as countries with total volume of

aid recorded in countries annual budgets as a percentage of donors’ disbursements have

increased only from 42% (2005) to 47% (2007) that fall significantly short to reach 85% by

2010 (Table 7). The annual rate of progress needs to be roughly seven to eight times greater

for the 2007-2010 period compared to the 2005-2007 period (ibid).

Strengthen Capacity by Co-ordinated Support (SCCS) Index

The Indicator 4 of the Alignment Index is being defined as strengthening capacity by co-

ordinated support and is being measured by the ‘percent of donor capacity-development

support provided through coordinated programmes consistent with partners’ national

development strategies’ (OECD, 2005a). The SCCS Index from the recipient’s and donor’s

side can be seen in Table 12 and 13 respectively. 2010 target i.e. 50% of technical co-

6 Only 12 out of 33 countries have taken part in the 2008 survey. Thus, any conclusion on the improvement of

scores cannot be derived at the moment and will only be possible after 2010 survey

19

operation to be co-ordinated for capacity development is already met (Table 7) as there is an

improvement of 12% evidenced between 2005 (48%) and 2007 (60%).

Use of Country System Index

Use of the Country System Index is measured by the ‘percent of donors and of aid flows that

use PFM systems and procurement systems in partner countries, which either (a) adhere to

broadly accepted good practices or (b) have a reform programme in place to achieve

these’(OECD, 2005a). So, the Use of Country System Index is to be obtained from the Use

of PFM System Index and the Use of Procurement System Index with following formula –

Use of Country System Index = ½ (Use of PFM System Index + Use of Procurement System

Index).

Use of Public Financial Management System Index

Use of the PFM System Index is measured by the ‘percent of donors and of aid flows that use

country PFM systems in partner countries that either (a) adhere to broadly accepted good

practices or (b) have a reform programme in place to achieve these’ (ibid.). Use of PFM

system index from the recipient’s side and donor’s side can be seen in Table 12 and 13

respectively. The use of the country PFM system by donors has increased only from 40%

(2005) to 45% (2007) that falls significantly short of reaching 80% by 2010 and the annual

rate of progress needs to be seven times greater for the 2007-2010 period compared to the

2005-2007 period (Table 7).

Use of Procurement System Index

Use of Procurement System Index is measured by the ‘percent of donors and of aid flows that

use country procurement systems in partner countries that either (a) adhere to broadly

accepted good practices or (b) have a reform programme in place to achieve these’ (OECD,

2005a). Use of the country procurement system index from the recipient’s side and donor’s

side is enclosed in Annexures 27 and 28 respectively. The use of a country procurement

system by donors has increased only from 39% (2005) to 43% (2007) that falls significantly

short of reaching 80% by 2010 and the annual rate of progress needs to be nine times greater

for the 2007-2010 period compared to the 2005-2007 period (Table 7).

Avoidance of Parallel Implementation Structures Index

Avoidance of a Parallel Implementation Structures Index is measured by the number of

parallel PIUs per country and the target is to reduce the two-thirds of the stock of such

parallel PIUs (OECD, 2005a). Avoidance of Parallel Implementation Structures Index from

the recipient’s side and donor’s side is can be seen in Table 12 and 13 respectively. The use

of parallel implementation structure by donors has reduced from 1817 in 2005 only to 1601

in 2007. The target is to reach 611 by 2010 and the annual rate of progress needs to be nine

times greater for the 2007-2010 period compared to the 2005-2007 period (Table 7). In some

cases, ‘the reported changes in the stock of parallel PIUs are the result of major

reclassifications by particular donors of their PIUs’ (OECD, 2008b). On the basis of the more

specific survey definition provided in 2008, some existing units were taken out of the 2005

listings of parallel PIUs at the same time some were included as respondents paid closer

attention to the overall thrust of the survey guidance on PIUs (ibid.). Unlike other indices, the

Avoidance of Parallel Implementation Structures Index is constructed by this formula7-

7 Because of the target of indicator which calls for reduction of two-thirds of the stock of such parallel PIUs. All

other indicators call for improvement in the indicators.

20

Predictability of Aid Index

Predictability of Aid Index is measured by the percent of aid disbursements released

according to agreed schedules in annual or multiyear frameworks and the target is to halve

the proportion of aid not disbursed within the fiscal year for which it was scheduled (OECD,

2005a). Predictability of Aid Index from the recipient’s side and donor’s side can be seen in

Table 12 and 13 respectively. The percent of aid disbursements released according to agreed

schedules in annual or multiyear frameworks has increased only from 41% (2005) to 46%

(2007) that falls significantly short of reaching 71% by 2010 and the annual rate of progress

needs to be five times greater for the 2007-2010 period compared to the 2005-2007 period

(Table 7).

Untying of Aid Index

The Untying of Aid Index is measured by the percent of bilateral aid that is untied and there

is no specific target as continued progress is to be observed over a period of time (OECD,

2005a). The Untying of Aid Index from the recipient’s side and donor’s side is enclosed in

Annexures 33 and 34 respectively. The percent of untied aid has increased from 75% in 2005

to 88% in 2006. There is no target to reach and this indicator intends to observe ‘continued

progress over time’ (Table 7).

Consolidation of Alignment Index

The Alignment Index is expected to be constructed from the formula explained earlier in

section 3. While constructing the Reliable Country System Index, it is decided that the

Reliable Procurement System Index should be dropped from the Alignment Index 8. On the

similar line, the Use of Country Procurement Systems Index is also dropped from the Use of

Country System Index because it would not bring any practical value to the Alignment Index.

Since the reliability of PFM/ procurement system is the recipients’ efforts towards aid

effectiveness therefore the data for these indicators is not available for the donors’. Whereas,

other indicators of the Alignment Index are the donors’ efforts towards aid effectiveness.

Therefore the Reliability of PFM/ Procurement System Index was not prepared for either of

the stakeholders and has been decided to be dropped from the Alignment Index.

Henceforth, it is decided that the Alignment Index will be constructed with following formula

- Alignment Index = 1/6 (AFANP index + SCCS index + Use of Country System Index +

Avoidance of PIU Index + Predictability of Aid Index + Untying Aid Index).

The Alignment Index from the recipient’s and donor’s side is constructed and can be seen in

Table 12 and 13 respectively.

4.3 Can a Harmonization Index be Developed?

This subsection attempts to construct the third dimensional index of AIE i.e. Harmonization

Index (see section 3) after defining its indicators and establishing individual indices and

finally consolidating the same.

Construction of a Harmonization Index

8 As discussed earlier that even if the Procurement System Index is excluded from the Reliable Country System

Index, it would not affect the overall Alignment Index. It is because the aspect of procurement already falls

under the overall ambit of PFM system, which is already being captured by the Reliable PFM System Index.

21

The Harmonization Index is based on the third principle of the PD framework and is captured

by two indicators i.e. Common Arrangement Index (indicator 9) and Shared Analysis Index

(indicator 10a and 10 b). The Harmonization Index is constructed by following formula –

Harmonization Index = ½ (Common Arrangement Index + Shared Analysis Index) where

the Shared Analysis Index = ½ (Mission to the Field Index + Country Analytic Work Index).

All these indicators are measured by donors and recipients’ questionnaires at the country

level. Like Alignment Index, Harmonization Index is also constructed for both the

stakeholders of the PD i.e. donors and recipients.

Table 8 How far are the Donors and Recipients are meeting the targets under

Harmonization Principle?

Indicator 2005

Baseline

2007

Achievement

Distance to

Cover before

2010

2010

Targets

9 Donors use co-

ordinated mechanisms

for aid delivery

43% 47% 19% 66%

10 (a) Donors co-

ordinate their missions 18% 21% 19% 40%

10 (b) Donors co-

ordinate their country

studies

42% 44% 22% 66%

Source: OECD, 2008b

Common Arrangement (CA) Index

The objective of the Indicator 9 of the Harmonization Index is to assess donor’s use of

common arrangements or procedures for aid delivery and is measured by the percentage of

aid provided for programme-based approaches (OECD, 2005a). The CA Index from the

recipient’s side and donor’s side can be seen in Table 12 and 13 respectively. The percent of

donor’s use of common arrangements or procedures for aid delivery has increased only from

43% (2005) to 47% (2007) that falls significantly short of reaching 66% by 2010 and the

annual rate of progress needs to be roughly five times greater for the 2007-2010 period

compared to the 2005-2007 period (Table 8).

Shared Analysis Index

The objective of the indicator 10 of the Harmonization Index is to encourage shared analysis

and is divided into two sub indicators i.e. percent of (a) field missions and/or (b) country

analytic work, including diagnostic reviews that are joint (OECD, 2005a). So, the Shared

Analysis Index = ½ (Mission to the Field Index + Country Analytic Work Index).

Mission to the Field Index

The Mission to the Field Index is being measured by the percent of donor’s field missions

that are jointly done with the partner countries (ibid.). Mission to the Field Index from the

recipient’s side and donor’s side can be seen in Table 12 and 13 respectively. The percent of

joint field missions has increased only from 18% (2005) to 21% (2007) that falls significantly

short to reach 40% by 2010 and the annual rate of progress needs to be roughly six times

greater for the 2007-2010 period compared to the 2005-2007 period (Table 8).

Country Analytic Work Index

22

Country Analytic Work Index is being measured by the percent of country analytic work,

including diagnostic reviews that are jointly done by donors with the partner countries

(OECD, 2005a). Country Analytic Work Index from the recipient’s side and donor’s side can

be seen in Table 12 and 13 respectively. The percent of joint country analytic work has

increased only from 42% (2005) to 44% (2007) which falls significantly short of reaching

66% by 2010 and the annual rate of progress needs to be roughly eleven times greater for the

2007-2010 period compared to the 2005-2007 period (Table 8).

Consolidation of a Harmonization Index

As discussed earlier, the Harmonization Index is constructed by the following formula –

Harmonization Index = ½ (Common Arrangement Index + Shared Analysis Index)

Where, The Shared Analysis Index = ½ (Mission to the Field Index + Country Analytic Work

Index)

The Harmonization Index from the recipient’s and donor’s side is constructed and can be

seen in Table 12 and 13 respectively.

4.4 Can a ‘Managing for Development Result’ Index be Developed?

This subsection attempts to construct the fourth dimensional index of AEI i.e. MfDR Index

(see section 3) after defining the indicator as well as criteria and methodology used to collect

the data.

Construction of a Managing for Development Result Index

MfDR Index is based on the fourth principle of PD framework and is captured only by one

indicator i.e. the Result Oriented Frameworks (ROF). The indicator is measured by ‘number

of countries with transparent and monitorable performance assessment frameworks to assess

progress against (a) the national development strategies and (b) sector programme’ (OECD,

2005a). This indicator is being measured on the basis of qualitative assessment based on the

WB CDF. Since the ROF Index is the only indicator to capture MfDR, henceforth it is being

termed as MfDR Index (see section 3). Like Ownership Index, MfDR Index is also

constructed only for recipients.

Table 9 How far the Donors and Recipients are meeting the targets under MfDR

Principle?

Indicator 2005

Baseline

2007

Achievement

Distance to

Cover before

2010

2010

targets

Sound frameworks to

monitor results 7% 9% 26% 35%

Source: OECD 2008b

The percent of donors’ use of common arrangements or procedures for aid delivery has

increased only from 7% (2005) to 9% (2007) that falls significantly short of reaching 35% by

2010 and the annual rate of progress needs to be roughly thirteen times greater for the 2007-

2010 period compared to the 2005-2007 period (Table 9).

The MfDR Index from the recipient’s side is constructed and enclosed in Table 11.

23

4.5 Can a Mutual Accountability Index be Developed?

This subsection attempts to construct the fifth and last dimensional index of AIE i.e. Mutual

Accountability Index (see section 3) after defining the indicator as well as criteria and

methodology used to collect the data.

Construction of a Mutual Accountability Index

MA Index is based on the fifth principle of PD and is captured only by one indicator i.e. MA.

The indicator is measured by ‘number of partner countries that undertake mutual assessments

of progress in implementing agreed commitments on aid effectiveness including those in this

declaration’ (OECD, 2005a). Like Ownership and MfDR Index, the MA Index is also

constructed only for the recipients.

Table 10 How far the Donors and Recipients are meeting the targets under Mutual

Accountability Principle?

Indicator 2005

Baseline

2007

Achievement

Distance to

Cover before

2010

2010

Targets

Mechanisms for mutual

accountability 22% 26% 74% 100%

Source: OECD 2008b

The percent of partner countries that undertake mutual assessments of progress has increased

only from 22% (2005) to 26% (2007) that falls significantly short of reaching 100% by 2010

and the annual rate of progress needs to be roughly eighteen times greater for the 2007-2010

period compared to the 2005-2007 period (Table 10).

The Mutual Accountability Index from the recipient’s side is constructed and enclosed in .

5 Consolidation of Aid Effectiveness Index

This section attempts to consolidate the five dimensions of the AEI on the basis of different

options of aggregation methodology (discussed earlier in section 3). If the two options

discussed earlier would not be adequate then the new option would be offered to establish the

AEI.

Do Aggregation Options established earlier consolidate the Aid Effectiveness Index?

As discussed earlier, the AEI is expected to follow either of two options. The first option is

based on the principle of equal weighting of all the five dimensions and gives equal

importance to all the dimensions of the PD i.e. Ownership Index, Alignment Index,

Harmonization Index, MfDR Index and MA Index. Whereas, the second option is based on

the principle of equal weighting of all the twelve indicators irrespective of their position in

individual dimensions and gives equal importance to all the indicators of the PD.

Consolidation of indices was attempted at two levels (for the Alignment index and for the

Harmonization Index). As the aspect of procurement is already being captured under the PFM

system, the Reliable Procurement System Index and the Use of Country Procurement System

Index was dropped from the Reliable Country System Index and the Use of Country System

Index respectively. Furthermore, the Reliable Country System Index was dropped from the

Alignment Index because it represents recipients’ side of aid effectiveness efforts compared

to other indicators of Alignment Index, which are donor’ side of aid effectiveness efforts.

24

On the one hand, Ownership, MfDR as well as MA indices respectively compare the level of

ownership, MfDR and MA across recipient countries in an absolute rather relative manner

where as on the other hand, Alignment and Harmonization Indices compare recipient and

donor countries in a relative rather an absolute manner (Booysen, 2002)9. The absolute nature

of comparison within Ownership, MfDR and MA indices restricts10

the cross country

comparison to a relative manner and thus makes it unviable to club them with other indices of

the PD i.e. Alignment and Harmonization Index.

The MA Index suffers the measurement problem which is reflected in the flawed composition

of respondents as well as ambiguous nature of survey question11

asked to measure MA of

recipients and donors. In 2006 survey it’s collected through the donor’s questionnaire but in

2008 survey it is collected through government’s questionnaire. It reflects either the

limitation of survey methodology or ideological bias as it is inconsistent in determining as

which stakeholder is responsible for adhering to and reporting ‘MA’ principle of the PD.

Further, it merely asks ‘yes’ or ‘no’ to establish whether MA is progressing among recipients

thus limiting the scope of any robust comparison among different recipients. Nevertheless,

it’s important to mention that donors are not scored to capture their side of progress on MA

principle, thus donors MA index is not constructed.

Hence, question arises as whether all the five dimensional indices (Ownership Index,

Alignment Index, Harmonization Index, MfDR Index and MA Index) is adequate as a basis

on which to construct donors and recipients AEI separately? While attempting to establish

indices on five different dimensions and twelve indicators individually, the aforesaid

limitations observed suggests that none of the two options are adequate as a basis on which to

construct the AEI. Furthermore, these limitations raises important questions as to what

aspects of aid effectiveness are to be measured for the two stakeholders i.e. donors and

recipients. Whether only few (Ownership Index, MfDR Index and MA Index) would be

adequate to build up recipient’s AEI? And does Alignment and Harmonization Index jointly

make the donor’s AEI?

What is the Next Aggregation Option to Consolidate the Aid Effectiveness Index?

Since indicators on ownership, reliability of country PFM and MfDR exclusively capture the

recipient’s efforts towards aid effectiveness agenda of the PD. That is why it is decided to

construct the AEI for recipients (Table 11) with this formula i.e. Recipients Ownership,

Alignment & MfDR Index = 1/3 (Ownership Index + Reliable Country PFM System Index12

+ MfDR Index). And since rest of the indicators of alignment and harmonization exclusively

9 The Ownership and MfDR Indices are measured on the basis of five categories (see Annexure 1 and 50) and

the MA Index (see Annexure 53) is measured on the basis of two categories.

10 Evident by the ranks held by recipients i.e. either 1 or 2 or 3 in Ownership, MfDR and MA indices instead of

1 to 29 in Harmonization and Alignment Indices

11 Has a mutual assessment of progress in implementing agreed commitments been conducted in your country?

12 The reliability of PFM system is the prerogative of recipients rather donors and Booth and Fritz (2008) made

similar observations. They noted that, ‘Donors tend to justify their bypassing of country systems of public

financial management (PFM) and procurement by pointing to weaknesses in those systems that make them

unreliable. This is quite plausible. For this reason, the agreed Paris Declaration targets for utilisation of

country PFM and procurement systems are conditional upon the systems’ attaining a certain level of reliability’

(Booth D and Fritz V, 2008, 21)

25

capture the donor’s efforts towards aid effectiveness agenda of the PD, it is decided to

construct the AEI for donors (Table 12) with this formula i.e. Donors Alignment &

Harmonization Index = 1/8 (AFANP index + SCCS index + Use of Country System Index +

Avoidance of PIU Index + Predictability of Aid Index + Untying Aid Index + Common

Arrangement Index + Shared Analysis Index).

Because of the measurement problem in the reporting of ‘MA’ principle of the PD, the MA

Index does not qualify to be included either in the recipients or donors Aid Effectiveness

Index.

Ethiopia, Ghana and Vietnam are the top rankers and Mauritania, the Democratic Republic of

Congo and Mozambique are the worst performers in Recipient’s Ownership, Alignment &

MfDR Index (Table 13). Ireland, the United Kingdom and the Netherlands (among bilaterals)

and the World Bank, the African Development Bank and IFAD (among multilaterals) are the

top rankers in Donor’s Alignment & Harmonization Index. Portugal, Korea and the United

States (among bilaterals) and the United Nations (among multilaterals) are the worst

performers in Donor’s Alignment & Harmonization Index (Table 12). Ghana, Bangladesh

and Tanzania are the top rankers and the Democratic Republic of Congo, Mongolia and

Yemen are the worst performers in Recipient’s Harmonization & Alignment Index (Table 13).

It’s important to ask why are the data captured under Harmonization & Alignment Index are

reflected both in recipients and donors indices and what does it reflect (Table 12 and 13)?

The next section attempts to answer this question and explore the relationship between

different donors and recipients’ indices.

26

Table 11 Recipient’s Ownership, Alignment & MfDR Index

33 Recipients Ownership13 Reliable PFM

System14 MfDR15 Index Ranking

Rwanda 1 1 NA 1 1

Ethiopia 1 1 0.5 0.83 2

Ghana 1 1 0.5 0.83 2

Viet Nam 1 1 0.5 0.83 2

Burkina Faso 1 0.66 0.5 0.72 5

Mali 0.5 0.66 1 0.72 5

Niger 0.5 0.66 1 0.72 5

Zambia 1 0.66 0.5 0.72 5

Afghanistan 0 1 1 0.66 9

Cape Verde 0.5 1 0.5 0.66 9

Honduras 0.5 1 0.5 0.66 9

Moldova 0.5 1 0.5 0.66 9

Mongolia 0.5 1 0.5 0.66 9

Nicaragua 0.5 1 0.5 0.66 9

Tanzania 1 1 0 0.66 9

Uganda 1 1 0 0.66 9

Albania 0.5 0.33 1 0.61 17

Burundi 0.5 0.33 1 0.61 17

Bolivia 0.5 0.66 0.5 0.55 19

Kenya 0.5 0.66 0.5 0.55 19

Senegal 0.5 0.66 0.5 0.55 19

Bangladesh 0.5 NA 0.5 0.5 22

13

Calculating the Ownership Index -

For Mali, with a national development strategy that reflects action taken towards achieving good practice is

scored C or 3 (Annexure 2), the Ownership Index is 0.5.

14

Calculating the Reliability of PFM System Index

For Zambia, which has scored 3.5 (Annexure 5) for the quality of its budget and financial management systems,

the Reliability of PFM System Index is 0.66.

15

Calculating the MfDR Index

For Afghanistan, where a mutual assessment of progress in implementing agreed commitments has been

conducted, the MfDR Index is 1.

27

33 Recipients Ownership13 Reliable PFM

System14 MfDR15 Index Ranking

Benin 0.5 0.33 0.5 0.44 23

Cambodia 0.5 0.33 0.5 0.44 23

Kyrgyz Republic 0.5 0.33 0.5 0.44 23

Malawi 0.5 0.33 0.5 0.44 23

Yemen 0.5 0.33 0.5 0.44 23

Mozambique 0.5 0.66 0 0.38 28

Congo, Dem Rep 0 0 1 0.33 29

Mauritania 0.5 0 0.5 0.33 29

Dominican Republic NA NA NA NA 31

Egypt NA NA NA NA 31

Peru NA NA NA NA 31

Source: Researcher’s Calculation based on OECD Data

28

Table 12 Donor’s Alignment & Harmonization Index

Donors AFANP16 SCCS

Use of

Country PFM

System

Avoidance of

PIU17 Predictability

of Aid Untying

Aid Common

Arrangement Shared

Analysis18 Index Rank

Ireland 0.45 1 0.97 1 0.75 1 1 0.64 0.851 1

United Kingdom 0.72 0.66 0.94 0.939 0.7 1 0.844 0.69 0.811 2

16

Calculating the AFANP Index

As 65% (see Annexure 12) of the United Kingdom’s total volume of aid is recorded in recipient countries’ annual budgets, the UK’s AFANP Index is 0.72.

Similarly the SCCS, Use of Country PFM System Index, Predictability of Aid, Untying of Aid Index and Common Arrangement Index are prepared from the Alignment Data

for 31 Donors (Annexure 12)

17 Calculating the Avoidance of PIU Index

In 2007, the United Kingdom has only 18 dedicated PIU structures (see Annexure 12) for day-to-day management and implementation of its aid-financed projects and

programmes in recipient countries. Thus the UK’s Avoidance of PIU Index is 0.939.

(see section 5.2.5 for the index formula )

18 Calculating the Shared Analysis Index

As discussed in section 6.2.2 the Shared Analysis Index = ½ (Mission to the Field Index + Country Analytic Work Index). So, for the United Kingdom, with the Mission to

the Field Index of 0.61 in 2007(Annexure 45) and the country Analytic Work Index of 0.77 in 2007 (Annexure 47), the Shared Analysis Index is 0.69.

Thus,

29

Donors AFANP16 SCCS

Use of Country

PFM System

Avoidance of PIU17

Predictability of Aid

Untying Aid

Common Arrangement

Shared Analysis18

Index Rank

Netherlands 0.69 0.59 0.77 0.956 0.65 0.999 0.845 0.55 0.756 3

Denmark 0.73 0.76 0.64 0.85 0.57 0.962 0.75 0.72 0.747 4

Norway 0.73 0.56 0.712 0.97 0.6 1 0.58 0.645 0.724 5

World Bank 0.8 0.87 0.65 0.73 0.8 NA 0.658 0.47 0.711 6

Finland 0.63 0.68 0.717 0.98 0.38 0.92 0.73 0.628 0.708 7

Asian Dev. Bank 1 0.78 0.458 0.864 1 NA 0.404 0.165 0.667 8

Germany 0.61 0.72 0.479 0.908 0.61 0.995 0.404 0.608 0.666 9

Sweden 0.53 0.5 0.69 0.92 0.61 1 0.55 0.519 0.664 10

Japan 0.45 0.77 0.75 0.99 0.52 0.95 0.614 0.175 0.652 11

Canada 0.47 0.6 0.91 0.864 0.57 0.733 0.66 0.203 0.626 12

IFAD 0.56 0.66 0.66 0.902 0.42 NA 0.376 0.805 0.626 12

European

Commission 0.67 0.48 0.477 0.64 0.72 NA 0.54 0.67 0.599 14

Global Fund 0.44 NA 0.49 0.99 0.42 NA 0.89 0.221 0.575 15

Switzerland 0.43 0.49 0.52 0.8 0.5 0.89 0.44 0.515 0.573 16

New Zealand 0.19 0.6 0.62 1 0.24 1 0.57 0.33 0.568 17

France 0.63 0.46 0.475 0.77 0.45 0.81 0.374 0.466 0.554 18

African Dev. Bank 0.61 0.27 1 0.61 0.61 NA 0.375 0.295 0.538 19

IDB 0.51 0.59 0.41 0.76 0.64 NA 0.45 0.385 0.535 20

Austria 0.39 0.26 0.452 0.908 0.35 0.962 0.34 0.3 0.495 21

Australia 0.19 0.33 0.13 0.952 0.4 1 0.22 0.643 0.483 22

GAVI Alliance NA NA 0.34 1 0 NA 0.438 0.5 0.455 23

Belgium 0.55 0.26 0.19 0.58 0.4 0.85 0.19 0.435 0.431 24

30

Donors AFANP16 SCCS

Use of Country

PFM System

Avoidance of PIU17

Predictability of Aid

Untying Aid

Common Arrangement

Shared Analysis18

Index Rank

Luxembourg 0.32 0.06 0 0.966 0.27 1 0.29 0.540 0.430 25

Italy 0.32 0.73 0.203 0.864 0.22 NA 0.3 0.315 0.421 26

United Nations 0.37 0.62 0.201 0 0.28 NA 0.401 0.605 0.353 27

Spain 0.19 0.44 0.61 0.76 0.31 0 0.14 0.344 0.349 28

United States 0.19 0.52 0.01 0.29 0.3 0.732 0.432 0.235 0.338 29

Korea 0.31 0.8 0.09 0.962 0.11 NA 0 0.075 0.335 30

Portugal 0 0 0.004 1 0.51 0.5 0.02 0 0.254 31

Source: Researcher’s Calculation based on OECD Data

31

32

Table 13 Recipient Harmonization & Alignment Index

33 Recipients 3 4 5a 6 7 8 9 (10a+10b)

/2 Index Rank

Ghana 0.92 0.77 0.66 1 0.78 0.821 1 0.74 0.83 1

Bangladesh 0.89 0.7 1 0.93 1 0.8493 0.7 0.37 0.80 2

Tanzania 0.77 0.55 0.93 0.908 0.5147 0.982 0.87 0.501 0.75 3

Uganda 0.98 0.51 0.74 0.7 0.683 0.67 0.9498 0.473 0.71 4

Ethiopia 0.46 0.66 0.6 0.69 0.67 0.605 0.949 0.7 0.666 5

Zambia 0.72 0.125 0.77 0.86 0.81 1 0.647 0.352 0.660 6

Afghanistan 0.62 0.4522 0.62 0.92 0.634 0.88 0.53 0.495 0.643 7

Kenya 0.5 0.61 0.69 0.96 0.33 0.656 0.386 1 0.641 8

Viet Nam 0.72 0.67 0.81 0.67 0.625 0.35 0.82 0.435 0.637 9

Honduras 1 0.948 0.71 0.84 0.57 0.45 0.16 0.388 0.633 10

Burkina Faso 0.89 0.49 0.563 0.33 0.89 0.822 0.81 0.25 0.630 11

Rwanda 0.317 0.944 0.54 0.8 0.58 0.89 0.51 0.38 0.62 12

Nicaragua 0.82 0.3 0.623 0.74 0.682 0.66 0.63 0.445 0.61 13

Bolivia 0.76 0.941 0.5 0.97 0.13 0.48 0.543 0.53 0.606 14

Mali 0.6303 0.8 0.44 0.662 0.6 0.85 0.547 0.285 0.601 15

Egypt 0.4077 0.98 0.156 0.87 0.73 0.44 0.68 0.5 0.59 16

Malawi 0.495 0.42 0.65 0.73 0.48 0.793 0.57 0.54 0.58 17

Niger 0.87 0.38 0.33 0.76 0.72 0.652 0.6833 0.232 0.578 18

Mozambique 0.76 0 0.567 0.92 0.675 0.8 0.641 0.249 0.576 19

Cape Verde 0.86 0.2 0.29 0.98 0.957 0.1 0.39 0.825 0.575 20 Dominican

Republic 0.314 1 0.64 0.84 0.07 0 0.91 0.665 0.55 21

Senegal 0.83 0.452 0.24 0.7 0.5142 0.8495 0.52 0.215 0.54 22

Burundi 0.35 0.23 0.42 0.9 0.31 0.795 0.466 0.54 0.50 23

Moldova 0.39 0.03 0.53 0.669 0.71 0.95 0.384 0.33 0.499 24

Benin 0 0.44 0.61 0.67 0.15 0.98 0.6831 0.44 0.496 25

Kyrgyz Republic 0.488 0.78 0.171 0.44 0.55 0.99 0.18 0.365 0.495 26

Albania 0.6343 0.39 0.159 0.93 0.11 0.602 0.12 0.405 0.418 27

Peru 0.484 0.644 0.58 0.51 0.52 0.21 0.09 0.25 0.411 28

Cambodia 0.8 0.126 0.178 0.19 0.95 0.47 0.35 0.07 0.39 29

Mauritania 0.4076 0.43 0.1 0.91 0.4 0.26 0.46 0.13 0.38 30

Yemen 0.07 0.31 0.06 0.91 0.16 0.69 0.239 0.39 0.35 31

Mongolia 0.11 0.649 0.21 0.71 0.17 0.42 0 0.132 0.30 32

Congo, Dem. Rep. 0.41 0.18 0 0 0 0.86 0.231 0.23 0.23 33

Source: Researcher’s Calculation based on OECD Data

33

Aid Effectiveness Prism: Exploring Relationship between Donors and Recipients

Index

As discussed earlier, it is important to explore the relationship between different

donors and recipients’ indices developed in previous section (Table 11, 12 and 13).

Why are the data captured under different indicators are reflected both in recipients

and donors indices and what does it reflect? This important question on complex

relationship between donors and recipients indices will attempt to be explored through

an Aid Effectiveness Prism (AEP). The AEP reflects this relationship between donors

and recipients on two dimensions i.e. input and output dimension. These twelve sub

indices of aid effectiveness can be measured and are best reflected as they move from

donors’ policy space to recipients’ policy space and vice versa. Few of the indicators

are considered to be a recipients’ side of bargain and most are donors’ side of bargain.

On the basis of analysis of different indices, their indicators and the way they are

defined and measured in the PD, the following relationships emerges between donors

and recipients indices and can be observed in the AEP –

When the aid effectiveness outcomes (AEO) are exclusively generated from within

the donors’ policy space and can be clearly seen from both donors as well as

recipients’ side of the AEP

Donor Harmonization & Alignment Index (Table 12) fits this category. Its sub indices

like AFANP Index, SCCS Index, Use of Country PFM System Index, Avoidance of

Parallel Implementation Structures Index, Predictability of Aid Index, Common

Arrangement Index and Shared Analysis Index represents the donor’s side of bargain.

The consolidate index is considered to be a part of input dimension in the AEP

because donors have to take concrete steps to make improvement in its respective

indicators and recipients cannot take any policy step to make any improvement in

those indicators (Figure 4).

For example, untying of aid is a donor’s side of bargain and is measured by the

percent of bilateral aid that is untied. It is represented in the input dimension in the

AEP because donors have to take concrete steps to reduce the level of tied aid. It is an

outcome generated from within the donors’ policy space and the same can be seen

from recipients’ side of aid effectiveness prism in Aid Untying Index. It is considered

to be a part of an output dimension because recipients’ while accessing tied aid,

cannot take any policy step to reduce it.

34

Figure 4 Donors Harmonization & Alignment Index vis-à-vis Recipients

Harmonization & Alignment Index

Source: Researcher, 2009

When the AEO are generated from within the recipients policy space and cannot be

clearly seen from the donors side of the AEP

Recipient’s Ownership, Alignment & MfDR Index (Table 11) fits this category. In its

consolidated form, it cannot be clearly seen as how aid effectiveness effort of

recipients have translated on the donors’ side (Figure 5). Since there is no data

captured at donors’ side, it remains challenging to figure out as how much progress

have been done at donors end in respecting country’s ownership, reliability of system

or MfDR result. It is possible that data is captured by the PD process but is not

disaggregated and made available by the PD survey process.

To see further what happens to this kind of aid effectiveness relationship, it is useful

to unfold the aggregate index (Recipients’ Ownership, Alignment and MfDR Index)

and scrutinise closely.

35

Figure 5 Recipients Ownership, Alignment & MfDR Index vis-a vis Donors

Ownership, Alignment and MfDR Index

Source: Researcher, 2009

When the AEO are generated from within the recipients policy space and should be

clearly seen from both the donors as well as recipients’ side of the AEP but cannot

be seen because of flawed PD indicators/ measurement

Case 1: Recipients Ownership Index vis-à-vis Donors Alignment Index

The PD envisages that donors must be committed to ensuring that their aid flows are

aligned on national priorities i.e. ‘to base their overall support — country strategies,

policy dialogues and development co-operation programmes — on partners’ national

development strategies’ (OECD, 2005a). Thus it is expected that donors would align

their aid flows on the basis of scores made by recipient country in ownership indicator

(Figure 6). But in the present form it has not been possible because of two reasons.

Firstly, the PD does not capture data for donors for this indicator and secondly the

methodology to measure this indicator is flawed (see footnote 22).

36

Figure 6 Recipients Ownership Index vis-à-vis Donors Alignment (Indicator 3)

Index

Source: Researcher, 2009

Case 2: Recipients Reliability of PFM System Index vis-à-vis Donors Use of

Country System Index

The PD envisages that as the recipients’ Reliability of PFM System Index is improving,

donors must start respecting it and the Usage of Recipients PFM System Index should

increase.

Improvement in the recipient’s reliability of PFM system is represented in the input

dimension of the AEP because recipients have to take concrete steps to improve the

level of reliability of their PFM systems (Figure 7). It is an outcome generated from

within the recipients’ policy space and the same can be seen and verified from donors’

side of AEP through Donors’ Use of Country PFM System Index. Donors’ Use of

Country PFM System Index forms part of an output dimension because donors ideally

would not start using the recipients’ PFM system unless it is reliable to their agreed

standards. But in the present form it is not possible to observe this relationship because

the data, despite being captured in the donor’s questionnaires is not disaggregated

donor wise at the recipient level.

37

Figure 7 Recipients Reliability of PFM System Index vis-à-vis Donors Use of

Country PFM System Index

Source: Researcher, 2009

When the AEO are generated from both the recipients and donors policy space but

cannot be clearly seen from either the recipients’ or donor’s side of the AEP

The Mutual Accountability Index fits this category as principally it should be captured

both for the recipients and donors. But in its present form in the PD, it is only

captured for the recipients. Though this AEO is generated from both the recipients

and donors policy space but because of the measurement problem and ideological or

objective bias observed in its reporting, it does not qualify to be included either in the

recipients or donors AEI. Thus, it cannot be clearly seen from either recipients’ or

donor’s side of the AEP (Figure 8).

38

Figure 8 Recipients MA Index vis-à-vis Donors MA Index

Source: Researcher, 2009

Conclusion

This section concludes that none of the aggregation options established earlier is

adequate as a basis on which to consolidate the AEI. Therefore an alternative

aggregation methodology is used to construct the AEI. The indicators are segregated

and indices are created on the basis of whether they represent recipients’ side of

bargain or donors’ side of bargain. The MA component is left out from the donors as

well as recipients side of index because the way it’s measured in the PD survey,

restricts the scope of any robust comparison among recipients as well as donors.

The AEP demonstrates that the PD survey hides the progress on indicators which are

donor’s side of bargain19

i.e. Donors Harmonization & Alignment Index. Interestingly,

it shows progress on these indicators through the Recipients Harmonization &

Alignment Index, where recipients cannot take any policy step to make any

improvement in those indicators. Furthermore, it does not present as how the aid

effectiveness effort of recipients have translated the policies at donors’ side. It is

challenging to figure out as how much progress have been done at donors end in

respecting country’s ownership, reliability of system or MfDR result because either

the data is not captured at donors side to do so or the methodology to measure few of

such indicators are flawed. There is a possibility that they could have been captured in

the questionnaires but are not disaggregated donor wise at the recipient level and thus

are not made available in the survey report.

19

donors input to achieve aid effectiveness

39

6 Summary & Conclusion

This section offers the conclusion by revisiting the issues raised in existing debates

regarding the development indices and examining where the composite AEI stands in

regard to these (chapter 2). This provides a means as well as an end to concluding

how the established AEI rates with regard to its strengths and weaknesses of

development indices, and thereby answer the overall research question i.e. can a

useful AEI be designed using PD criteria’s and survey data?

Where does the AEI stand in Existing Debates on quality of Development

Indices?

On the basis of Booysen’s classification of composites indices of development, one of

the strengths of the three composite indices20

constructed in chapter 9 is its ‘content’

as it explicitly conveys that it measures only government to government (including

multilaterals) aspects of aid flows (2002, 116). Nevertheless it also indicates its

limitation as it does not cover Southern donors, private actors (e.g. NGOs), global

vertical funds and foundations despite the fact that they comprise some 25% of global

aid (ODI, 2009). It is also not easy to determine if each of the PD dimensions are

equally essential in determining the level of aid effectiveness or if a few more

dimensions are required or can be dropped/substituted. The AEI does aspire to

measure other aspects of aid effectiveness i.e. aid as a % of GNI or debt relief

measures, which is traded off because of lack of their inclusion in the PD Framework.

The AEI could be criticised like the WGI because of various sorts of biases in the

individual indicators. It could be ideological21

as well as objective biases22

/ 23

(Booysen, 2002; Iqbal and Shah, 2008) or perceptual bias24

(Kurtz and Schrank,

2007). On the basis of presence of such biases, the five dimensions of the AIE can be

termed as an ad hoc selection of indicators based on donors’ political motive

(Booysen, 2002) and concerns can further be raised about the questions if certain

20

Recipients Ownership, Harmonization & MfDR Index, Recipient Harmonization & Alignment Index

and Donor’s Alignment & Harmonization Index 21

The objective of the indicator 5 is to improve the actual use of the recipient country’s financial

management and procurement systems by donors, which should increase as such systems are

increasingly becoming reliable as envisaged in indicator 2. It is observed that even if some recipient

country achieves high scores or reveals significant improvement in reliability of its PFM system, ‘a

substantial part of the donors aid to the public system may still be disbursed outside the recipient

country system’ (Bissio, 2008, 12)

22 The target for indicator 8 i.e. aid untying only promises continued progress over time which reflects

the objective biasness in such indicators as it prohibit donors to make any substantial progress. 23

Indicator 3 which is measured by % of aid flows to the government sector reported on recipient’s

national budgets should be replaced and measured by % of donor’s aid flows that are not routed

through recipient’s national budget. If the objective of the PD is to align the donor aid flows to

recipient’s national budget i.e. national priorities, then the current indicator runs the risk to defeat this

purpose. Because even if recipient government would achieve reporting 100% of aid flows to the govt

sector on their national budgets, there would be aid flows which will not be aligned on recipients

national priorities as donors would continue disbursing ODA for the non-government sector. 24

The survey question asked to assess indicator 6 i.e. avoidance of parallel implementation structures

suffer the perceptual bias and can be improved by asking the right questions. Whether the reductions in

numbers of PIUs are because of few PIUs might have reached the end of their normal life cycle before

the 2007 survey. Or is it simply because of real efforts made by not creating any new parallel PIUs

while new projects are being signed after 2005 survey (2008 Survey, 46)?

40

indicators are ambiguous or suffer flawed composition of respondents25

(Iqbal and

Shah, 2008).

But the AIE would definitely score high on any criticism of sample selection bias

(Kurtz and Schrank, 2007; Arndt and Oman, 2006) as the donors26

and recipients27

are

not randomly selected rather based on criteria suitable i.e. comparability over time

and across countries, for the development of index. This has reduced any problems

arising with cross-country comparisons (over time as well as across country) due to

unbalanced sampling, which was the case with the WGI (Arndt and Oman, 2006;

Iqbal and Shah, 2008). It has also avoided making any assumption of constant global

averaging and country specific weighting for any unbalanced sample, for which the

WGI was criticised by Iqbal and Shah (2008).

It is difficult to ascertain how the AEI would fare against with the criticisms of

accuracy of the data employed or the independence of the assessments or flawed

judgments of primary sources (Booysen, 2002; Kaufmann et al, 2007; Iqbal and Shah,

2008). Booth and Fritz argued that ‘the survey guidance has given too much freedom

to respondents to define their own terms’ and ‘the ways in which respondents used

this freedom provides fairly clear evidence on some things’ (2008, 21). And the

‘donor respondents seem often to have been more concerned about portraying their

agency’s practices as “politically correct” than about setting an accurate baseline

against which to judge future efforts’ (ibid). Nevertheless, it could be premature to

judge the AEI on this parameter and only an informed debate can ascertain the

validity of such criticisms.

Lack of any conceptual framework is one of the weaknesses witnessed in different

development indices and for instance, the WGI is also criticised by Kurtz and Schrank

(2007) for the same. But compared to the WGI, the AEI would score high as it

follows an analytical framework i.e. the PD, which is an international consensus as

well as commitment to reduce aid inefficiency.

Like other composite development indices, the AEI also cannot withstand the critique

of exclusion of one or more essential components (Booysen, 2002). Both the donors

as well as recipients side of index suffers the weakness of exclusion of the MA

component that is an equally essential component of the PD. But it is important to

revisit that while designing the CDI, Birdsall and Roodman (2003) also encountered

the same challenge in their design where aspirations to measure policies were often

compromised because of a lack of data across all the components.

25

Ownership dimension i.e. partners have operational development strategy can be questioned on the

ground as whose ownership? As the way it’s being defined and tracked by the World Bank in PD

surveys, it seems to ‘justify processes for the maintenance of hard conditionality and other forms of

donor oversight of policy formulation’ (Whitfield, 2008, 85). 26

Out of 55 donors that took part in the 2008 monitoring survey only 31 donors are included. The two

criteria applied were 1) all donors that have reported over USD 100 million aid earmarked for the

government sector in at least three countries in the surveyed countries; and 2) all donors who do not

meet the first criteria but would like to publish their results in the 2008 survey overview’ (OECD, 2008,

91). 27

Out of 55 recipient countries that took part in the 2008 monitoring survey only 33 recipient countries

are included. The criterion applied was to ensure the cross country comparison which is possible only

by measuring the set of 33 baseline countries that have participated in both the 2006 baseline survey

and the 2008 monitoring survey (OECD, 2008, 91).

41

A few other limitations of the AEI are its inability to reflect any correlated errors in

individual indicator assessments, construct validity and failure to determine the

causality and collinearity between twelve indicators used in the index (Arndt and

Oman, 2006; Thomas, 2006; Diamantopoulos and Winklhofer, 2001). However, the

AEI scores high as there is not any abrupt changes in weighting28

which could have

led to high levels of measurement errors (Booysen, 2002; Iqbal and Shah, 2008).

The AEI could be scrutinised closely on the issue of vulgarity and over-simplification

of the one-number AEI or its aggregate indicators (Parr and Kumar, 2004; Arndt and

Oman, 2006; Kurtz and Shrank 2006; Thomas 2006). But it might be appreciated

because it avoids any negative implication of folding five separate dimension indices

into a single AEI. As on the one hand, the indicators of Alignment and Harmonization

were clubbed and folded on the basis of its relevance to donor and on the other hand,

the indicators of Ownership, Reliability of Country System and MfDR were for the

recipient side. If all the five separate dimension indices were folded into a single AEI,

it would have either hidden the inadequacy of such dimensional indices29

or

preference/ negligence of few other dimensional indices30

. Nevertheless, grouping and

folding in its present form suffers the risk of over-simplification that could be

extremely misleading at times (Dervis, 2005).

The AEI definitely ranks high regarding offering more insights and revealing more

than pre-existing PD survey reports alone reveals (Booysen, 2002). Survey reports

which predominantly convey the recipient’s progress on five principles hide the

donor’s progress on the same principles. For instance, the Donors’ Alignment &

Harmonization Index not only reveals how the donors are making progress against

each other but also identifies as which donors have improved their scoring or

experienced slippage.

Sagar and Najam (1998) have raised concerns regarding the process of translating

data into an index. However, because of its transparency and access to the data used,

the AEI would sustain this concern. Other development indices like WGI is criticised

because of its lack of transparency of data sources (Arndt and Oman, 2006; Iqbal and

Shah, 2008). The AEI also would not be criticised on the grounds of time

inconsistency of definitions and measurements (Kurtz and Schrank, 2007) because it

has dropped the MA and reliable procurement system dimension of AEI that suffered

these weaknesses.

Further, the formulation of the AEI also reveals the global inequities existing in the

donor-recipient relation. The complex relationship between and within donors and

recipients index was best demonstrated by the AEP. Interestingly the AEP

demonstrates that the PD survey hides the progress on indicators which are donor’s

side of bargain i.e. Donors Harmonization & Alignment Index, which is donors input

28

It was not easy to establish as whether equal weighting should be given to the five dimensions or not.

Since giving preference to any dimension by giving additional weight suffers the risk of being

questioned on the basis of motive and ideology, it was decided to give equal weighting to the indicators

instead of dimensions. Furthermore, as all the five dimensions were not required to construct the

indices from recipient and donors perspective, equal weighting was given to indicators instead of

dimensions used for respective indices. 29

If the MA Index would have been added, it could have hidden the inadequacy for which it was

dropped from the donors and recipients composite indices

30 MfDR and Ownership indices were preferred for recipients efforts towards aid effectiveness and

Mutual Accountability is dropped from both donor and recipient index

42

to achieving aid effectiveness and it shows progress on these indicators through the

Recipients Harmonization & Alignment Index, where recipients cannot take any

policy steps to make any improvement in those indicators.

The AEI fulfils the technical as well as the policy process requirements which

Wiesmann (2006) has listed as desirable properties of international indices (see

section 2). The AEI scores high on its technical requirement as it is comparable over

time and across countries based on valid methods of measurement, non-redundant,

able to differentiate among countries and is sensitive to changes over time. A possible

weakness of the AEI is that it might not be robust against measurement error31

or

stable against random fluctuations.

Further the AEI also scores high on the policy process requirements as it is highly

relevant to the existing aid reform efforts. It is also highly sensitive to the existing

inequality as it demonstrates the donor and recipient side of the bargain separately and

different biases associated in different indicators and methodology employed. It is

replicable as well as flexible as it can integrate the forthcoming third round of the PD

monitoring in 2011. Though it is available for large number of stakeholders32

, it is not

suitable for annual updates because of lack of data availability due to the PD

monitoring cycle.

The AEI is also adequate in answering the right question, evaluating how well this aid

effectiveness is being measured and if it could be done in a much better and effective

way. The AEI determines the donors’ and recipients’ overall aid effectiveness

commitment and progress towards its achievement. The AEI can also be

communicated to policymakers and the general public and it may be helpful in

seriously gearing up recipient countries and their external partners’ efforts to meet

their international commitments and targets for effective aid by 2010 and beyond.

Conclusion

On the basis of the AEIs comparative standing in the existing debates of development

indices and its strengths and weaknesses of qualifying for an international

development index, it can be concluded that a useful AEI can be designed using PD

criteria and survey data. It’s important to note that the limitations of the Paris

Declaration concept of Aid Effectiveness are inevitably reflected in the AEI. But as

the PD is expected to be monitored again in 2011, the AEI would certainly bring

much practical value as it can reflect how the donors and recipients are progressing

against their peers respectively in different indices (Booysen, 2002).

The study has further demonstrated that the AEI can be consolidated on the basis of

best possible aggregation methodology (discussed in previous section). On the one

hand, the indicators that are found to be recipients’ side of bargain and exclusively

capture the recipient’s efforts towards aid effectiveness agenda of the PD are clubbed

under Recipients Ownership, Alignment & MfDR Index. And on the other hand, the

indicators that are found to be donors’ side of bargain and exclusively capture the

donor’s effort are clubbed under Donor’s Alignment & Harmonization Index. The

MA component is left out from the donors as well as recipients side of index because

the way it’s measured in the PD survey, restricts the scope of any robust comparison

among recipients as well as donors.

31

moderate changes in aggregation function

32 33 recipients and 31 donors including multilateral agencies

43

Despite not being an integral part of the AEI, the AEP is quite useful to reflect the

complex relationship within donors and recipients indices on two dimensions i.e.

input and output dimension. It found that the PD survey hides the progress on

indicators that are donors’ side of bargain and shows progress on these indicators

through the recipients’ side. Furthermore, the AEP also reveals that the PD survey

does not present as how the aid effectiveness effort of recipients have translated the

policies at donors’ side.

The aforesaid finding and analysis brings the research to the central understanding

that the AEI has the potential of summarizing complex and multi-dimensional aid

effectiveness issues supporting decision makers but with a caveat that it may also

send misleading, non-robust policy messages if it is poorly constructed or

misinterpreted (Saisana and Tarantola, 2002). In the absence of a sensitivity analysis

to test the robustness of index, the dissertation refrains from suggesting any policy

messages.

Bibliography

Action Aid (2007) Making Aid Accountable and Effective Available at

http://www.actionaid.org/docs/making%20aid%20accountable%20and%20effective.pdf Accessed

on 17th

June, 2009.

Arndt C and Oman C (2006), Uses and Abuses of Governance Indicators,

Development Center Studies, OECD, Paris.

Birdsall N and Roodman D (2003), The Commitment to Development Index: A

Scorecard of Rich-Country Policies, Center for Global Development, Washington,

D.C. Available at www.cgdev.org/doc/cdi/technicaldescrip05.pdf Accessed on 17th

June, 2009.

Booth D and Fritz V (2008) Good Governance, Aid Modalities and Poverty

Reduction From Better Theory to Better Practice Research Project Report for the

Advisory Board for Irish Aid, Dublin Available at

http://www.gsdrc.org/go/display&type=Document&id=3013&source=rss Accessed

on 14th

September, 2009.

Booysen F (2002) An Overview and Evaluation of Composite Indices of

Development, Journal of Social Indicators Research, 59(2), pp115-151

Centre for Global Development (2008) Commitment to Development Index

Available at http://www.cgdev.org/section/initiatives/_active/cdi/ Accessed on 17th

June, 2009.

Dervis K (2005) The World Bank’s Independent Evaluation Group Conference:

The Effectiveness of Assistance for Human and Social Development Available at

http://www.worldbank.org/oed/conference/human_social/docs/oed_conference_transc

ript_1.pdf Accessed on 1st July, 2009.

44

Desai M (1991) Human Development, Concepts and Measurement Journal of

European Economic Review Vol. 35, pp 350–357.

Diamantopoulos A and Winklhofer H (2001), Index Construction with Formative

Indicators: An Alternative to Scale Development, Journal of Marketing Research,

Vol. 37, pp. 269-277.

Doessel D P and Gounder R (1994), Theory and Measurement of Living Levels:

Some Empirical Results for the Human Development Index, Journal of

International Development Vol 6, pp. 415–435.

Fakuda-Parr S and Kumar A K S (2004) Readings in Human Development:

Concepts, Measures and Policies for a Development Paradigm, Oxford University

Press

Global Call to Action against Poverty (2008) A Critical Approach to the Aid

Effectiveness Agenda Available at

http://www.whiteband.org/resources/issues/aid/the-aid-effectiveness-debate/a-critical-

approach-to-the-aid-effectiveness-agenda Accessed on 17th

June, 2009.

Hicks D A (1997), The Inequality-Adjusted Human Development Index: A

Constructive Proposal, World Development Vol 25, pp. 1283–1298.

Iqbal K and Shah A (2008) Truth in Advertisement: How do Worldwide

Governance Indicators stack up? Unpublished paper, World Bank Institute,

Washington, DC

Kambhampati U S (2004) Development and the Developing World, Polity Press,

Cambridge.

Kaufmann D, Kraay A and Mastruzzi M (2007), The Worldwide Governance

Indicators Project: Answering the Critiques, World Bank Policy Research Department

Working Paper No.4149, Washington.

Kurtz J M and Shrank A (2007), Growth and Governance: Models, Measures and

Mechanism The Journal of Politics, 69 (2), pp. 538-554.

Maxwell S (2005) The Washington Consensus is Dead! Long Live the Meta-

narrative! ODI Working Paper No. 243. London, Available at

http://www.odi.org.uk/resources/details.asp?id=3855&title=odi-working-papers

Accessed on 1st July, 2009.

Menocal A R and Rogerson A (2006) Which Way the Future of Aid? Southern

Civil Society Perspectives on Current Debates on Reform to the International Aid System ODI Working Paper No. 259, London Available at

http://www.odi.org.uk/resources/details.asp?id=3855&title=odi-working-papers

Accessed on 1st August, 2009.

Mo Ibrahim Foundation (2008) Ibrahim Index of African Governance Available at

http://www.cgdev.org/section/initiatives/_active/cdi/

http://www.moibrahimfoundation.org/the-index.asp Accessed on 1st July, 2009.

45

Noorbakhsh F (1998), The Human Development Index: Some Technical Issues

and Alternative Indices, Journal of International Development, Vol. 10 (5), pp. 589

– 605.

OECD (2005a) Paris Declaration on Aid Effectiveness Available at

www.oecd.org/dataoecd/11/41/34428351.pdf Accessed on 17th

June, 2009.

OECD (2005b) Baselines and suggested targets for the 12 Indicators of Progress -

Paris Declaration on Aid Effectiveness Available at

http://www.oecd.org/dataoecd/45/46/35230756.pdf Accessed on 17th

June, 2009.

OECD (2006a) Statistics, Knowledge and Policy: Key Indicators to Inform

Decision Making Organization for Economic Cooperation and Development

Publishing, Paris.

OECD (2006b) The 2006 Survey on Monitoring the Paris Declaration Website

Available at

http://www.oecd.org/document/20/0,3343,en_2649_3236398_38521876_1_1_1_1,00.

html Accessed on 1st July, 2009

OECD (2008a) 2008 Survey on Monitoring the Paris Declaration Available at

http://www.oecd.org/document/0/0,3343,en_2649_3236398_41203264_1_1_1_1,00.h

tml Accessed on 17th

June, 2009.

OECD (2008b) 2008 Survey on Monitoring the Paris declaration: Making Aid

More Effective By 2010 Available at

http://www.oecd.org/dataoecd/58/41/41202121.pdf Accessed on 17th

June, 2009.

Overseas Development Institute (2009) How Serious are Donors about Aid

Effectiveness? The challenges to 2010 and beyond Available at http://www.odi.org.uk/events/report.asp?id=495&title=serious-donors-about-aid-effectiveness-

challenges-2010-beyond Accessed on 17th

June, 2009.

Roodman D (2006) Building and Running an Effective Policy Index: Lessons at

the Commitment to Development Index Center for Global Development,

Washington DC Available at http://www.cgdev.org/content/general/detail/6661

Accessed on 1st July, 2009.

Save the Children (2008) Child Development Index Available at

http://www.cgdev.org/section/initiatives/_active/cdi/

http://www.savethechildren.org.uk/en/docs/child-development-index.pdf Accessed on

1st July, 2009.

Segnestam L (2002) Indicators of Environment and Sustainable Development

Theories and Practical Experience Environmental Economic Series Paper no. 89.

The World Bank Environment Department, Washington.

Siggel E (2005) Development Economics: A Policy Analysis Approach, Ashgate

Publishing Limited, Hants.

46

Srinivasan T N (1994), Human Development: A New Paradigm or Reinvention of

the Wheel? American Economic Review, Vol 84, pp. 238–243.

Stanton E A (2007) The Human Development Index: A History, Working Paper No.

127 Political Economy Research Institute, Amherst Available at

http://www.peri.umass.edu/Publication.236+M52d58c45817.0.html Accessed on 1st

July, 2009.

Stapleton L M and Garrod G D (2008), The Commitment to Development Index:

An Information Theory Approach, Journal of Ecological Economics, Vol. 66 (2-3)

pp 461-467

Stern E, Altinger L, Feinstein O, Marañón M, Ruegenberg D, Schulz N and Nielsen N

S (2008) The Paris Declaration, Aid Effectiveness and Development Effectiveness

Evaluation of the Paris Declaration Available at

http://www.oecd.org/document/60/0,3343,en_21571361_34047972_38242748_1_1_1

_1,00.html Accessed on 17th

June, 2009.

The Fund for Peace (2009) Failed States Index 2009 Available at

http://www.cgdev.org/section/initiatives/_active/cdi/

http://www.fundforpeace.org/web/index.php?option=com_content&task=view&id=99

&Itemid=323 Accessed on 1st july, 2009.

Thomas M A (2006), What Do the Worldwide Governance Indicators Measure?

Unpublished paper, John Hopkins University, Washington DC

Thompson T and Shah A (2005), Transparency International’s Corruption

Perception Index: Whose Perceptions are they Anyway? Available at

http://web.worldbank.org/WBSITE/EXTERNAL/WBI/EXTWBIGOVANTCOR/0,,co

ntentMDK:20725263~menuPK:1977002~pagePK:64168445~piPK:64168309~theSite

PK:1740530,00.html Accessed on 21st July, 2009.

Transparency International (2009) Transparency International’s Corruption

Perceptions Index Available at

http://www.transparency.org/policy_research/surveys_indices/cpi Accessed on 21st

June, 2009.

UNDP (2008) Technical Note 1 Calculating the Human Development Indices

Available at http://hdr.undp.org/en/statistics/indices/hdi/ Accessed on 21st June, 2009.

UNDP (2009) How is the HDI used? Available at

http://hdr.undp.org/en/statistics/faq/question,69,en.html Accessed on 21st June, 2009.

Wiesmann D (2006) A Global Hunger Index: Measurement Concept, Ranking of

Countries, and Trends, FCND Discussion Paper 212, International Food Policy

Research Institute, Washington Available at

http://www.ifpri.org/sites/default/files/publications/fcndp212.pdf Accessed on 21st

June, 2009.

47

World Bank (2009) Governance Matters 2009: The Worldwide Governance

Indicators Project Available at http://info.worldbank.org/governance/wgi/index.asp

Accessed on 21st June, 2009.