evaluating programme performance

8/8/2019 Evaluating Programme Performance

1/30

Evaluating Programme Performance:

The Lack-Gap-Mismatch Analysis

First draft - please do not quote.

AbstractThis document introduces the construct - "Lack-Gap-Mismatch (LGM) Analysis" that can

be used to identify and categorize problems faced in a network. A basic template of the

issue under study is done by studying the inputs, throughputs and outputs. This template isused to compare the perceived needs (or desired scenario) with the existing situation which

brings out the lacks (that which is not there - promulgations and raising of consciousness

need to be done),gaps (that which is not sufficient - enhancements and stepping up of

efforts need to be done) and mismatches (that which is incompatible - changes alterationsneed to be done).

The Lack-Gap-Mismatch analysis was first conceptualized through a number of brainstorming sessions in community development that took place in squatter settlementsof Pune, India during 1989. After a comprehensive socio-economic survey of the

settlement residents, the results were tabulated and discussed by them in a series of

community meetings. In seeking to categorize the problems, they identified andstudied the causes of problems in their settlements. They felt that identifying this was

important in finding an appropriate solution at the appropriate level. The approaches

and programmes/projects that they later recommended to the city authorities was

greatly facilitated by this approach.

A significant output of these meetings and other subsequent interactions was theproblem classification methodology adopted. The differing variety of problems that

the residents faced led to the idea of classifying them as 'lacks', 'gaps', or 'mismatches.'This construct was then put to practical test in diverse situations until it took its

present shape.

Introduction

The Lack-Gap-Mismatch (L-G-M) Analysis is a method devised to evaluate the

existing status and shortcomings of any situation or issue, for example, the informal

credit market, a poverty alleviation programme or a low-income housing settlement.

As the name suggests, there are three problem areas being studied here: 'lacks' - that

which is not there; 'gaps' - that which is not sufficient; and 'mismatches' - that which is

not compatible. Thus, problems identified in the field fall into one of these three

categories and work on making recommendations for action is greatly facilitated. Anypolicy, programme or project being evaluated may have its strong as well as weak


2/30

points. The L-G-M Analysis helps in identifying and differentiating between these

points. Recommendations made as a result will direct resources where they are most

needed while retaining its good or beneficial points.

To understand the L-G-M construct, it is necessary to firstly diagnose the issue being

studied. This is done by identification of the particular flow of inputs, throughputs andoutputs through the system under study. A comparison of scenarios is done, where the

perceived needs or 'desired scenario' is compared to the existing situation. This

comparison will highlight the problems that are occurring and can be accordingly

categorized as a lack, gap or mismatch. Corrective action needed to be taken to

remove the lacks, gaps and mismatches will involve identification of the actors, their

preconditions, operations to be carried out and intended effects.

Inputs, Throughputs and Outputs

One of the first steps to be taken is to diagnose the system by studying its inputs,throughputs and outputs. This will generate the basic template over which future

analysis can be based. The inputs and outputs could be internal or external to thesystem - and it should be distinguished as such. This helps in classifying the

contributing components of the system. It also helps in directing corrective action that

needs to be taken.

1. The Flow of Inputs - that which is introduced/inducted into the system or sub-system as necessary ingredients for its initiation. What are the inputs that are

necessary to be included into the system for it to 'begin' or be initiated? There

are two types of inputs: one from within (or internal to) the system, and theother from outside (or external to) the system. (For example, collecting earth

and brick-making machines to produce bricks or an NGO initiating community

meetings).

Characteristics of the inputs that have to be examined can include, type andformat (quality), quantity of required inputs, supplier or 'provider', temporal

point of feeding into the process, cost factors and labour requirements.

2. The Flow ofThroughputs - that which is transformed by a process to a formwhich makes it usable and functional within the system. What is the process oftransformation of the input components to a usable form? In the throughput

process, the components are converted or modified from a non-usable state to a

usable state (e.g. processing of earth to produce bricks for housing, or a group

of squatters deciding to form a self-help association).


3/30

Characteristics of the throughputs that have to be examined can include,

original state of the components, process of transformation or change, finalstate of the components created, actors initiating and maintaining the process,

quality of the process - linear or cyclical, type of intermediate products created,

if any, time involved in the production and finance requirements.

3. The Flow of Outputs - that which is generated or produced by the system in anintermediate or final form. What are outputs of the system? Qualitative and

quantitative attributes of the product that has been output have to be studied.

The output may be a final-form product used external to the system, or an

intermediate-form product used internally at a particular stage in the process

(e.g. bricks for house building, the house itself, or credit supplied by the self-

help association).

Characteristics of the outputs that have to be examined can include, qualitative

attributes of the product, quantitative measurements of the product,characteristics of the market that deals in the product, user characteristics, by-

products produced, if any, and re-sale/re-use of the products.

Comparison of Desired Scenario and Existing Provisions:

The L-G-M construct is better suited for linear rather than longitudinal or time-series

evaluation. A comparison of a 'desired scenario' with the existing situation is

developed from the 'perceived needs' of the target group being studied.

There are two distinct entities being compared: the 'perceived needs or desiredscenario' and the 'existing provisions or situation'. While data pertaining to the

existing situation can be collected directly from the field, data for 'perceivedneeds/desired scenario' is more complex to gather. The following discussion might

offer some guidelines:

Perceived needs or Desired Scenario

There are three criteria here: needs, wants, desires. The three represent a hierarchy of

priorities, with desires having least priority and need being an immediate and

important priority. The scenario can be developed in several ways:

y Internally, by direct questions to the target population in expressing theirdesires, wants and needs. Personal observations, informal discussions and

community discussions can also be used to gather views.y Externally, by observing and recording stated goals and objectives of the

responsible implementing agency or organization. These could be immediate,


4/30

intermediate or eventual goals and objectives. Opinions of academics, resource

persons, politicians and administrators not directly related to the project or

programme can also be used to develop the scenario.

Existing Situation

Collecting data indicative of the existing provision/situation can be done directly from

the field. Emphasis should be placed on several different methods of data gathering:

standardized questionnaire survey with target respondents, observations by theresearcher, scheduled interviews and opinion surveys with various individuals

involved directly or indirectly. While this may cause overlaps to a certain extent,

differing views and cross-checking will strengthen the findings.

A comparison of the desired scenario/perceived needs with the existing situation will

bring into focus the various problems being faced.

The Lack-Gap-Mismatch Analysis

The character of the problems identified in the previous steps is classified as a lack,

gap or mismatch, using criteria mentioned below.

1. The Lack Sub-analysis'Lack' (that which is not there) refers to a situation where the existing

provisions made do not meet perceived needs of the target. In other words, thedesired scenario does not exist or is not complete due to the 'lack' of certain

necessities.Corrective action to overcome a lack will take the form of, for example,

promulgations, declarations. or raising consciousness. Therefore, new

recommendations or policies would have to be devised so as to remedy

problems encountered in this category. This may entail organizational

(structural or policy guidelines) and/or operational (action programmmes or

procedures) modifications.

The criteria that can be used to identify lacks may include: defect, deficiency,demerit, deficit, dearth, deprivation, fault, flaw, imperfection, inadequacy,

incompetence, indigence, need, poverty, scarcity, or want.

2. The Gap Sub-analysis'Gap' (that which is insufficient) refers to a situation where the provisions made

are those perceived by the target population, but are essentially insufficient or

inadequate.

Recommendations made to remedy problems encountered in this categorywould cover, for example, a more intensive implementation of the same policy

or pogram or higher financial allocations. Thus, corrective action for gaps


5/30

would take the form of enhancements or stepping up efforts to facilitate the

accomplishment of existing pograms/policies.The criteria used in identifying associated with gaps can include: break,

cessation, delay, disparity, inadequate, insufficiency, interval, lag, lapse, lull,

shortcoming, shortage, suspension, or undersupply.

3. The Mismatch Sub-Analysis'Mismatch' (that which is not compatible) refers to a situation where the

provisions made are incongruous to, and do not match the perceiveddesires/wants/needs of the target population.

Problems encountered in this category have to be tackled by a change or

modification in existing policies and programmes, where redundant sections or

portions are removed and/or replaced.The criteria associated with identification of mismatches can include: anomaly,

abnormality, atypical, clash, conflict, contradiction, contrary, deviancy,

difference, disparate, dissimilarity, divergency, improper, ill-adopted, ill-suited,inept, ineligibility, inappropriateness, incongruous, inconsistency, unequipped,

unequal, unfit, unqualified, unrelated or unsuitable.

Post-evaluation Application: After the L-G-M Analysis

As indicated earlier, a lack calls for promulgations or the raising of consciousness, agap for enhancements or stepping up efforts, and a mismatch for modifications oralterations. In order to ensure that these are carried out in a step-by-step and smooth

manner, four steps of action to be taken can be distinguished: actors, preconditions,

operations and effects.

1. Actors, or Who is to do it? The key to the identification of participating actorsis to make best use of the available institutional and professional resources,

keeping the overlaps and intrusions to a minimum. Direct role players and

indirect/supportive actors have to be identified and distinguished.2. Preconditions, or What preconditions need to be satisfied before any action can

be taken? These preconditions would apply to both the operations that have to

be carried out and actors who are to carry them out. This is essential to ensurethat maximum participation and commitment are contributed by the actors so as

to achieve the intended effect.

3. Operations, or How can it be done? A clear set of actions and sub-actions needto be detailed out in overcoming lacks, closing gaps and undoing mismatches.Temporal and geographical variations in the operations, if any, also need to be

specified.

4. Effects, or What is the intended effect of the action/operation? This will have tomatch the overall goals and objectives of the project/ programme. Detailing the


6/30

intended effects for every action will help in the consequent monitoring and

evaluation, and to initiate another cycle of L-G-M analysis.

Conclusions

By identifying each problem through this process, the character of the action to betaken will reflect the type of classification made: whether it is a lack, gap or

mismatch. The recommended actions to be taken can take the form of a policy,

pogram or project.

The key to usefulness of the L-G-M construct lies in its suitability for diagnosis and

perception of a problem, and guiding corrective action. This characterization is

essential due to the widely differing sets of problems or issues that are identified, the

causes and its effects. By doing this, scarce physical and human resources can be

conserved and utilized in an appropriate manner, and direct it where it is most needed.

Strategic Planning Checklist

Beryl Levinger

The following checklist can be used to assess the readiness of an organization toengage in strategic planning. Its primary utility is to plan for training and technical

assistance that can help overcome the obstacles noted. Strategic planning is intimately

related to organizational partnering in that partnerships can alter in very fundamental

ways the relationship any organization has with the external environment in which it

operates.

The organization is ready to engagein strategic planning ifeach of the following

conditions prevails (recordyes or no as appropriate):

1. There is a willingness to work toward developing the best fit between the

organization and its external environment by examining the following questions:

y where are we going? (mission)y how do we get there? (strategies)y what is our blueprint for action? (budget)y how do we know if we're on track? (control)

Yes___ No___


7/30

2. There is a high likelihood that consensus can be reached on the following issues:

y Who are we?y What are the basic social and political needs we exist to fill?y What do we do to recognize or anticipate and respond to these needs?y Who are our key stakeholders and how should we respond to them?y What are our philosophies and core values?y What makes us distinctive or unique?

Yes___ No___

3. There is an absence of impending doom and crisis.

Yes___ No___

4. There is deeply held commitment on the part of top leadership to engage in

strategic planning.

Yes___ No___

5. There is a shared understanding about the nature of strategic planning among

organizational stakeholders.

Yes___ No___

6.. There is a competent group of people willing and able to serve on the strategic

planning team.

Yes___ No___

7. There is ability within the organization to respond to problems with solutions that

are politically, ethically, technically, and culturally acceptable.

Yes___ No___

8. There is agreement on the planning process/model to be used.

Yes___ No___


8/30

9. There is consensus regarding the organizational mandate given to the planning

team.

Yes___ No___

10. There are adequate resources (including facilitators from either within or outsidethe organization) so that the planning team can do each of the following tasks:

y clarify organizational mission and valuesy identify clients/stakeholdersy assess the external environmenty assess the internal environmenty identify the strategic issues it facesy formulate strategies to manage these issues (options generation)y establish an effective organizational vision for the futurey convert the vision into activity plans, budgets, and key result areas that can be

monitored

y monitor performance "actuals" versus "expectations"y make adjustments to the plan

Yes___ No___

11. There is access to data that reflect the political, economic, social and technological

trends that effect the organization's clients, customers, competitors and collaborators.

Yes___ No___

12. There is access to data that reflect the organization's current resources and

performance level.

Yes___ No___

A Simple Capacity Assessment Tool

(SCAT)

Beryl Levinger and Evan Bloom

The application of this organizational capacity assessment tool requires broadparticipation by the organization that wishes to employ it.The Simple Capacity

Assessment Tool (SCAT) has been developed to 1) provide support organizations

with procedures for assessing the organizational capacity of potential partners; and, 2)


9/30

provide a process through which relevant, context specific indicators can be

developed in a collaborative manner. A support organization working with the SCATwill identify appropriate indicators, develop an appropriate scoring nomenclature, and

begin to develop a strategy for implementing a program of organizational capacity

assessments.

The SCAT includes seven organizational categories (e.g. Governance, Management

Practices) which are further broken down into sub-components (e.g.. Board,

Organizational Structure). For each sub-component, criteria of organizational capacity

or performance are developed in collaboration with staff from the organization that is

being assessed. Support organizations can help facilitate this process.

One illustrative example of performance criteria has been provided for each sub-

component. Additional space has been provided for criteria that will be developed in

the process of working with the SCAT. When developing criteria, the following rules

should be observed:

y Present only one concept or attribute at a time. Criteria that have multipleattributes present real problems during validation. For example, "strategies are

realistic in the context of the NGO's activities andcan be translated into clear

program objectives" should be presented as two separate criteria.

y Follow polarity. All criterion should follow the same parallel structure so thata yes response is registered as uniformly desirable or uniformly undesirable.

The same rule holds true when you are using scaled responses such as

1=nascent, 2=emerging, 3=expanding, 4=mature.

y Develop neutral criteria. Avoid adjectives and adverbs such as "excellent" or"always," which can make it impossible to rate organizational capacities as

either very strong or very weak. Criteria should be written in a neutral way so

that the scoring method you select has full value.

y Develop organization-centered criteria. The assessment tool will be validatedthrough data sources made available by the organization being assessed. For

example, validation of "the organization is seen as a full and credible partner by

the government" would require consulting government sources. This is beyondthe scope of this type of organizational capacity assessment. Separate sets of

tools, however, can and should be developed to pursue these external

perspectives.

A rating scale of 1 through 4 (1=nascent; 2=emerging; 3=expanding; 4=mature) is

used in the SCAT. Scoring should be assigned to sub-components. To calculate


10/30

average scores by major category, sum all scores (elements) under each category and

divide by the number of elements. Write the results in the category box provided next

to the category heading.

Support organizations should consider the following points before finalizing their own

capacity assessment tool:

1. Purpose ofEvaluation: Identify the purpose(s) particular to this assessment.

2. Team Composition: Select an assessment team to include people who know the

organization and who will be in a position to carry out some of the recommendations

that result from the assessment. Also select people who are external to the

organization and can bring objectivity and an independent perspective to the situation.

One member of the team should have some expertise in management and

organizational development. It may be useful to include as many people on the team

as the management deems necessary, either because of their knowledge of theorganization or because of their need to be educated about its strengths and

weaknesses. A representative mix of management functions or divisions of theorganization should be included, as well as representatives of the membership or

constituency served by the organization.

3. Identify Information Sources: Identify internal and external information sources

and schedule individual interviews, group meetings and data collection sessions togather information. Determine who on the assessment team will interview the

information sources and who will be responsible for collecting data. Meeting with

small groups of constituents, members and staff representatives in focus groups(discussion groups) -- where a small group of respondents is guided by a facilitator

into responding to questions at increasing levels of focus and depth -- is an efficient

way of gathering valuable information.

4. Interviews: Conduct interviews, meetings and data collection sessions according toa schedule that has been submitted to respondents in advance. Each interview session

should commence with an explanation about the purposes of the assessment and the

uses that will be made of the information. Specifically emphasize when theinformation source(s) will receive assessment results and their involvement in the

utilization of the information.

Information should be gathered and recommendations presented in a way that

emphasizes that the organization being assessed is not being judged against an

absolute set of standards as the only form an organization can take. Rather, it isimportant to remind respondents that the functions and factors being offered for

comparative purposes are suggestions about possible ways of doing things that are


11/30

derived from extensive experience in managing organizations in a multiplicity of

environments.

5. Developing Recommendations: The assessment team should be prepared todevelop recommendations on how the organization can best address the issues

identified in the assessment and how to build upon the strengths highlighted.

6. Transparency: Recommendations should be discussed with organization

management and front-line workers, not just those members of the organization who

have participated in the assessment.

Capacity Assessment Tool

Name of Organization: ________________________

Date of 1st Assessment: Conducted by:

Date of 2nd Assessment: Conducted by:

Date of 3rd Assessment: Conducted by:

Date of 4th Assessment: Conducted by:

Scoring: 1 = Nascent; 2 = Emerging; 3 = Expanding; 4 = Mature

1st 2nd 3rd 4th

Assess Assess Assess Assess

A. Governance

1. Board

a. Board provides appropriate level of institutional oversight.

b.

c.

2. Mission, Goals and Philosophy

a. Organization's mission is well defined.

b.

c.

3.Executive Leadership


12/30

a. Executive leadership has a clear vision of organization'smission.

b.

c.

4. Legal Status

a. Documentation related to legal status is in order.

b.

c.

B. Management Practices

1. Organizational Structure

a. Lines of authority facilitate agile decision-making.

b.

c.

2. Information Systems

a. Timely information is available to support decision-making.

b.

c.

3. Administrative Procedures

a. Administrative tasks are systematized.

b.

c.

4. Planning

a. Organization develops operational plans that guide action.

b.

5. Program Development


13/30

a. Baseline data collected by organization guides program.

b.

c.

6. Program Reporting

a. Program reports accurately reflect strengths andweaknesses.

b.

c.

C. Human Resources

1. Personnel Management

a. Program is in place to facilitate staff development.

b.

c.

2. Diversity Issues

a. Organization's work force is diverse.

b.

c.

3. Supervisory Practices

a. Supervisory practices facilitate staff growth anddevelopment.

b.

c.

4. Salary and Benefits

a. Salary and benefits are sufficient to retain skilled staff.

b.

c.

D. Financial Resources


14/30

1. Accounting

a. Accounting practices yield accurate financial data.

b.

c.

2. Budgeting

a. Budget process is integrated with program planning.

b.

c.

3. Financial/Inventory Controls

a. Independent audits are an integral part of the financialcontrol system.

b.

c.

4. Financial Reporting

a. Financial reporting is timely.

b.

c.

E. Service Delivery

1. Sectoral Expertise

a. Organization has the experience necessary to accomplishits mission.

b.

c.

2. Constituency Ownership

a. Stakeholders influence service delivery.

b.

c.


15/30

3. M&E Systems

a. Project implementation is monitored against benchmarks.

b.

c.

F.External Relations

1. Constituency Relations

a. Organization regards its constituency as a full partner.

b.

c.

3. Government Relations

a. Organization has mechanisms in place to influence

relevant government policies

b.

c.

4. Donor Relations

a. Organization has practices and procedures for recognizing

donors.

b.

c.

5. Public Relations

a. Organization uses multiple channels for attracting support.

a.

b.

6. MediaRelations

a. Organization maintains diverse contacts with media

outlets.

b.


16/30

c.

G. Partnering

1. Intra-sectoral partnering

a. Organizations engage in intra-sectoral partnerships tofurther mission.

b.

c.

2. Inter-sectoral partnering

a. Organizations engage in inter-sectoral partnerships to

further mission.

b.

c.

Decision-Making Matrix for Training

Beryl Levinger

The Decision-Making Matrix for Training (DMT) focuses on tasks associated with the

management of training and is intended to assist support organization that wish tosponsor or finance capacity building training activities. It contains a listing (in columnone) of the major decisions that have to be made in connection with the design and

implementation of training, suggests some viable options with respect to each of these

decisions (column two) and offers guiding principles for selecting among options

(column three). A fourth column is included so that a support organization planning

team can identify which option(s) it has selected.

This tool is designed to help support organizations develop and manage scopes of

work for capacity building contractors, and to enable support organizations to plan

their own capacity building efforts. To use the tool effectively, it is not necessary tofollow the decision-making sequence as presented.

Decision-Making Matrix for Training

Issues Illustrative Options Guiding Principles Decisions


17/30

Who shouldbe trained

initially?

Begin by training the trainers;

by training senior leadership;by training the supervisors of

frontline staff; or, by

strengthening frontline staff.

1. A cascade strategy is generally a cost-effective

approach to training. Begin with training of

trainers. Then cascade down to senior leadership,

supervisors and frontline workers.

2. Many organizational problems cannot be solved

at a single organizational level. Therefore, include

training activities in the overall design that mix

levels, functions and organizational perspectives.

How should

trainees be

selected andgrouped?

Restrict training to members

of a single organization or

include staff from multiple

organizations; restrict trainingto members of a singleorganizational unit or includestaff from multiple units;

restrict training to one

organizational level orinclude people with a mix of

organizational

responsibilities; select

individuals as trainees or

require that eligibility for

training be restricted to teams

that can work together to puttraining into practice

1. In general, trainee groupings should correspondto the roles and functions within organizations that

have a bearing on the problem that the training is

attempting to address. Usually, this involves

trainees from multiple units.

2. Where there are important points of similarity, a

mix of organizations and/or levels within an

organization offers productive cross-fertilization ofideas, promotes innovation, and enhances the

training climate.

3. Highly competitive or hierarchical environmentsmay not tolerate a mix of levels and/or

organizations. In such instances, a preparatory

stage may be required to achieve the optimal mix

of participants.

Who should

design andimplement

the training?

Internal facilitators; external

international facilitators;external local facilitators; or,

a mix of these types.

1. Training is best designed and implemented by

those closest to the organization who possess therequisite capacity building and training skills. In

ascending order of preference, this means thatpriority is given to international facilitators;

external local facilitators; internal facilitators.

2. Where international or external facilitators are

used, their role should primarily focus on trainingof trainers (TOT) and short-term backstopping

local trainers who are graduates of TOT programs.

Where should

the training

take place?

On-site; off-site at a similarorganization; off-site at aconference center; off-site at

a training or technical

institution; at a residential ornon-residential setting.

1. Training designed to effect a major culture

change is best conducted off-site in a residential

setting.

2. Skills-based, short-duration training is oftenmost appropriately conducted on-site.

What

materials are

needed tosupport the

training?

Packaged, off-the-shelf

courseware; open-ended,

locally prepared exercises;case studies; a mix of types.

1. Whatever their provenance, good training

materials allow participants to solve authentic

open-ended problems in group settings andaddress a range of organizational constraints

including inadequate knowledge and


18/30

organizational culture.

What training

methodologyshould be

employed?

Open-ended discussion and

exercises designed to

inculcate new attitudes and

values; hands-on, structured,skills-based training designed

to impart specific knowledge

essential to capacity building;

a mix of types.

1. In general, fundamental changes in

organizational capacity require shifts in

organizational culture. This is best accomplishedby providing trainees with group-centered

opportunities to explore new values and give voice

to any resistance they may feel toward them.

What should

the training

objectives

be?

Mastery of specific functional

skills; changes inorganizational culture; team-

building; enhanced capacity

for organizational learning;

introduction of TQM or some

other customer-focused

change; new capabilities to

respond to changes in theexternal environment.

1. Selection of objectives must be realistic. In

general, training can pave the way for fundamental

organizational change, but is not, by itself,

sufficient to accomplish such change. Overly

ambitious objectives may turn people off when the

expected results fail to materialize.

2. Training should be viewed as but one element ofa capacity building strategy.

What should

the length,

duration, and

timing of

training be?

Short (under a week) and

intensive; moderate (5-10

days) and intensive; long (11

days and up) and intensive;short and extensive (e.g., 4

days over 4 months);

moderate, and extensive;

long, and extensive.

1. Intensive training is usually less difficult to plan

and deliver, but may result in intolerable levels of

disruption to the organization's functioning.

2. When training is extensive, it is more difficult to

maintain momentum for change but easier to

discuss problems associated with the application

of training content to an organizational setting.

How shouldthe training

be evaluated?

Participant feedback; clientfeedback; observation of

participants; observation of

services to clients;

assessments of the

organization's functional

capacities; comparative

assessments (before and after

training) of the organization's

effectiveness in achieving itsmission; or, a mix of severalof these options.

1. The ultimate purpose of training is to achieve a

fundamental change in an organization's

achievement of mission. Impact evaluation must

address this issue.

2. Formative evaluation is critical to improving an

organization's training program and can be

gathered from many difference sources including

participants, clients, and observers.

What follow-

up to trainingshould there

be?

Additional training; on-sitetechnical assistance to

support implementation of

new ideas; "maintenancemeetings" for trainees;newsletters; networking offormer trainees; provision of

opportunities for some

trainees to serve as trainers in

1. The planning and implementation of follow-upis an essential component of any training design.


19/30

future training sessions.

How should

training befinanced?

Fully funded by the sponsor

(e.g., the support

organization); through a cost-

sharing arrangement betweenthe organization and the

sponsor; fully funded by the

organization; modest trainee

financial contribution to

cover some aspect of the

training program (e.g.,

materials, lodging, in-kinddonation of time).

1. Cost-sharing helps organizations to value the

training they receive.

2. Trainees should be expected to contribute to the

cost of training only when participation in trainingis at their option, and when direct benefit will

accrue to them as a result of their having

participated in the training (e.g., salary increments,

promotions).

Decision-Making Matrix for Technical Assistance

Beryl Levinger

The Decision-Making Matrix for Technical Assistance (DMTA) focuses on tasks

associated with the management of technical assistance and is intended to assist

support organizations that wish to sponsor technical assistance in support oforganizational capacity building. It contains a listing (in column one) of the major

decisions that have to be made in connection with the design and implementation of a

technical assistance program, suggests some viable options with respect to each of

these decisions (column two) and offers guiding principles for selecting amongoptions (column three). A fourth column is included so that a support organization

planning team can identify which option(s) it has selected. This tool is designed to

help support organizations develop and manage scopes of work for technicalassistance contractors. To use the tool effectively, it is not necessary to follow the

decision-making sequence as presented. Blank boxes are provided for recording

support organization decisions on TA options.

Decision-Making Matrix for Technical Assistance

Issues Illustrative Options Guiding Principles Decisions

Who should

receivetechnical

assistance?

Trainers; senior leadership;

supervisors of frontline staff;

frontline staff; anyone with atargeted type of need or

organizational responsibility;

anyone falling below a targeted level

1. Technical assistance should be viewed

from multiple perspectives: as a reward for

promising performance, as a tool forsustaining the change momentum, and as an

approach for addressing unsatisfactory

performance.


20/30

of performance; anyone surpassing a

targeted level of performance;

evaluators; or a mix of the above.

How should

technicalassistance be

delivered?

Provide technical assistance toindividuals on a customized basis;

provide technical assistance to small

groups of individuals confronting

similar problems on a customizedbasis; provide technical assistance to

individuals using standardized tools

and materials; provide technical

assistance to small groups ofindividuals using standardized tools

and materials.

1. Technical assistance is more costly and of

more variable quality if customized than ifbuilt around the use of standardized tools

and materials.

2. Where problems are likely to be unique

or unanticipated, customized technical

assistance is essential.

3.If standardized tools and materials allowfor good organizational diagnosis and open-ended discussion, they can be extremelyvaluable in enhancing organizational

capacity.

Who shoulddesign and

provide

technicalassistance?

Internal consultants; external

international consultants; external

local consultants; or, a mix of these

types.

1. Technical assistance is best designed and

delivered by those closest to theorganization who possess the requisitecapacity building and diagnostic skills. In

ascending order of preference, this meansthat priority is given to international

facilitators; external local facilitators;

internal facilitators.

2. Where international or external

facilitators are used, their role shouldprimarily focus on training in-house or

locally available consultants and providingthem with short-term backstopping.

3. In an environment characterized by high

levels of threat, fearfulness and internalcontrol, external consultants are preferable

to internal consultants.

Where shouldthe deliveryof technical

assistance

take place?

On-site; off-site at a similar

organization; off-site at a venue the

ensures confidentiality and open

communication.

1. In general, technical assistance is best

delivered on-site.

2. Off-site technical assistance may be mostappropriate in environments characterized

by high levels of threat, fearfulness andinternal control.

What should

the

relationship

be between

technical

To reinforce the skills imparted

through training; to reinforce

cultural changes imparted through

training; to troubleshoot

implementational difficulties; to

1. Technical assistance should be viewed as

a complement to training in any program of

organizational development.

2. While there are many options for


21/30

assistance and

training?extend concepts introduced through

training; to gather information and

build skills prior to introduction of

formal training; to screen

participants for subsequent training;

or, a mix of the above.

achieving this complementarity, it is

essential that the relationship between

training and technical assistance be made

explicit and transparent.

What

methodology

should be

employed for

the provisionof technical

assistance?

Open-ended discussion designed to

reinforce new attitudes and values;

informal joint review of

accomplishments and problems to

date with emphasis on problem-

solving; application of structuredtools that can be used to assess

progress, identify problems, and

revise practices; a mix of types.

1. Technical assistance should include

support for both the application of new

skills or practices as well as for the cultural

changes that underlie capacity building

efforts. Whatever methodology is used mustaddress both skills and cultural constraints

to change.

What should

the objectiveof technical

assistance be?

Mastery of specific functional skills;

changes in organizational culture;

team-building; enhanced capacity

for organizational learning;introduction of TQM or some other

customer-focused change; new

capabilities to respond to changes inthe external environment.

1. Selection of objectives must be realistic.

In general, technical assistance can, in

concert with training, pave the way forfundamental organizational change, but is

not, by itself, sufficient to accomplish such

change. Overly ambitious objectives may

turn people off when the expected results

fail to materialize.

2. Technical assistance objectives should be

focused on specific, relatively narrow yet

observable changes in organizationalcapacity.

What should

the timing of

technical

assistance be?

According to a pre-arranged

scheduled (e.g., quarterly); when

milestones or benchmarks are

accomplished; on demand as

problems are encountered; a mix oftypes.

1. The benefits derived from technicalassistance are directly related to its

timeliness and predictability.

2. Pre-arranged schedules augmented by"on

demand" support are useful at the beginning

of the technical assistance process. As

organizations gain experience, scheduling

by milestones or benchmark is more cost-effective.

How should

technical

assistance be

evaluated?

Recipient feedback; organizational

client feedback; observation ofrecipients; observation of services to

clients; assessments of the

organization's functional capacities;

comparative assessments (before

and after technical assistance) of theorganization's effectiveness in

achieving its mission; or, a mix of

several of these options.

1. The ultimate purpose of technical

assistance is to achieve a fundamental

change in an organization's achievement ofmission. Impact evaluation must address

this issue.

2. Formative evaluation is critical to

improving a technical assistance program

and can be gathered from many difference

sources including participants, clients, andobservers.


22/30

How should

technical

assistance be

funded?

Fully funded by the sponsor (e.g.,

the support organization); through a

cost-sharing arrangement betweenthe organization and the sponsor;

fully funded by the organization.

1. Cost-sharing helps organizations to value

the technical assistance they receive.

2. It is generally necessary to create ademand for technical assistance. One way to

accomplish this is by gradually reducing the

degree to which it is subsidized.

What should

be thetechnical

assistance

exit strategy?

Provision of a fixed number of daysof consulting that gets "drawn

down"; achievement of agreed upon

benchmarks or performancestandards; "graduation" from

technical assistance recipient to

technical assistance provider; a mix

of the above.

1. At the outset of technical assistance

provision, recipients should know how

much and what kinds of support can beavailable to them and under what

circumstances such support will be

forthcoming.

2. One outcome of technical assistanceshould be that an organization has a plan in

place for meeting its future technical

assistance needs.

What follow-up to

technicalassistance

should there

be?

None; infrequent follow-upassessment and "maintenance"visits;newsletters; networking former

technical assistance recipients;provision of opportunities for former

technical assistance recipients to

serve as future technical assistance

providers.

1. Organizational development is a never-ending process. The most important follow-

up to technical assistance is support for anorganization as it endeavors to meet future

technical assistance needs on its own. Such

support can take many different forms.

Selecting Organizations for Strengthening Matrix (SOS)

Beryl Levinger

The Selecting Organizations for Strengthening (SOS) matrix is designed to provide

support organizations with procedures for assessing and selecting nongovernmental

organizations (NGOs) for capacity building support. Organizations are measuredagainst four metrics: fit, program attractiveness, coverage exclusivity, and competitive

advantage.

The SOS uses a scale of 20=low, 40=moderate, and 60=high to score and compareprospective organizations as exemplified in the illustrative graph presented at the end

of this package.

The four parameters for decision making are:


23/30

A. Fit, or the degree to which an organization reflects the support organization's

development priorities. Determinants of fit include:

y congruence between the organization's purpose and mission and the supportorganization's strategic objectives;

y demand for the organization's existing skills by the support organization'sprimary customers; and;

y degree of ease with which the support organization can share resources andcoordinate activities with the candidate organization.

B. Program attractiveness, or the degree to which an organization is attractive to the

support organization from a cost-effectiveness standpoint. In other words, we areinterested in knowing whether capacity building support for this particular

organization represents a sound investment of the support organization's current and

future resources. Determinants of program attractiveness include the following:

y the organization's appeal to constituencies that are capable of providing currentand future support;

y stability of the organization's funding base;y the degree to which the organization's work yields measurable, reportable

program results; andy the degree to which the support organization would be able to discontinue its

support with relative ease, if necessary (i.e., low exit barriers for the support

organization)

C. Coverageexclusivity, or the extent to which other organizations provide similarservices as the organization being evaluated. If there are no comparable organizations,

then the program is classified as "highly exclusive." If there are just a few similarly

positioned organizations, then it is considered "moderately exclusive." If there are

many other organizations offering similar services, then its coverage exclusivity is

deemed "low."

D. Competitive position, or the degree to which the organization has a stronger

capability and potential to deliver specific services of interest to the supportorganization than other institutions. Competitive position is a reflection of the

organization's effectiveness, quality, credibility, and market dominance. Determinantsof a strong competitive position include:

y absolute quality of the organization's delivery system;y quality of service and/or service delivery in relation to that of competitors;y level of client support;y market share of the target clientele served by the organization;


24/30

y ability to conduct needed research into the program and/or properly monitorprogram performance;

y ability to communicate to stakeholders; andy cost- effectiveness of service delivery.

In general, support organizations should not consider providing capacity buildingassistance to organizations which score low on eitherfit orprogram attractiveness.

Limited program attractiveness augurs poorly for sustainable change. Limited fit

suggests that the organization under consideration does not have a legitimate claim on

the support organization's resources at the present time. Organizations 3 and 4 in the

graph below fit this profile and would be dropped from further consideration for

capacity building support from the support organization.

In contrast, organizations that score well on fit and program attractiveness but only

moderately on the other two categories represent prime targets for assistance as they

have valuable threshold competencies and the internal capacity to benefit fromadditional institutional strengthening investments. Organization 1 in the graph below

fits this profile.

Organizations which score high on all four categories may also be considered for

additional institutional-strengthening support although in general they are not a highpriority group since they probably have the capacity to sustain their own internal

development. Instead, such groups should be targeted as potential agents for sharing

their capacities with other institutions under appropriate partnership arrangements.

Organization 2 in the graph below fits this profile.

Finally, organizations that score high on fit and program attractiveness, but low in the

other categories constitute a lower level institutional development target group for

support organizations because of the large initial investment that must be made in

such organizations. Organization 5 in the graph below fits this profile. Networking

and other informal approaches may be used with such institutions until they become

stronger targets for support organization investments.

Legend: 20 = low; 40 = moderate; 60 = high

INTERPRETATION:

Low scores in fit and/or program attractiveness: eliminate from further consideration

(Organizations 3 and 4).


25/30

High scores in two remaining categories: facilitate opportunities for organization to

share its expertise by promoting appropriate partnership opportunities (Organization

2).

Moderate scores in both remaining categories: give first tier priority to organization

for future capacity building support (Organization 1).

Mix of moderate and low scores in both remaining categories: give second-tier

priority to organization for future capacity building support.

Low scores in both remaining categories: give third-tier priority to organization for

future capacity building support (Organization 5).

Glossary of Methods andTools

Each of the methods described above is a combination of tools, held together by aguiding principle. Dozens of exercises exist to cultivate collaborative development

planning and action. These are the tools with which social scientists and other

development practitioners encourage and enable stakeholder participation. Some toolsare designed to inspire creative solutions, others are used for investigative or analytic

purposes. One tool might be useful for sharing or collecting information, whereas

another is an activity for transferring that information into plans or actions. These

brief descriptions are intended to provide the reader with a glossary of terminologythat practitioners of participatory development use to describe the tools of their trade.

y Access to resources.A series of participatory exercises that allows development practitioners tocollect information and raises awareness among beneficiaries about the ways in

which access to resources varies according to gender and other important social

variables. This userfriendly tool draws on the everyday experience of

participants and is useful to men, women, trainers, project staff, and field-workers.

y Analysis of tasks.A gender analysis tool that raises community awareness about the distributionof domestic, market, and community activities according to gender and

familiarizes planners with the degree of role flexibility that is associated with

different tasks. Such information and awareness is necessary to prepare andexecute development interventions that will benefit both men and women.


26/30

y Focus group meetings.Relatively lowcost, semistructured, small group (four to twelve participants

plus a facilitator) consultations used to explore peoples' attitudes, feelings, or

preferences, and to build consensus. Focus group work is a compromise

between participantobservation, which is less controlled, lengthier, and more

indepth, and preset interviews, which are not likely to attend to participants'own concerns.

y Force field analysis.A tool similar to one called "Story With a Gap," which engages people to

define and classify goals and to make sustainable plans by working on thorough

"before and after" scenarios. Participants review the causes of problematic

situations, consider the factors that influence the situation, think aboutsolutions, and create alternative plans to achieve solutions. The tools are based

on diagrams or pictures, which minimize language and literacy differences and

encourage creative thinking.

y Health-seeking behavior.A culturally sensitive tool for generation of data about health care and

healthrelated activities. It produces qualitative data about the reasons behindcertain practices as well as quantifiable information about beliefs and practices.

This visual tool uses pictures to minimize language and literacy differences.

y Logical FrameworkorLogFRAME.A matrix that illustrates a summary of project design, emphasizing the results

that are expected when a project is successfully completed. These results or

outputs are presented in terms of objectively verifiable indicators. The Logical

Framework approach to project planning, developed under that name by theU.S. Agency for International Development, has been adapted for use in

participatory methods such as ZOPP (in which the tool is called aprojectplanning matrix) and TeamUP.

y Mapping.A generic term for gathering in pictorial form baseline data on a variety ofindicators. This is an excellent starting point for participatory work because it

gets people involved in creating a visual output that can be used immediately to

bridge verbal communication gaps and to generate lively discussion. Maps are

useful as verification of secondary source information, as training and

awarenessraising tools, for comparison, and for monitoring of change.Common types of maps include health maps, institutional maps (Venn

diagrams), and resource maps.y Needs assessment.

A tool that draws out information about people's varied needs, raises

participants' awareness of related issues, and provides a framework forprioritizing needs. This sort of tool is an integral part of gender analysis to


27/30

develop an understanding of the particular needs of both men and women and

to do comparative analysis.y Participant observation

A fieldwork technique used by anthropologists and sociologists to collect

qualitative and quantitative data that leads to an indepth understanding of

peoples' practices, motivations, and attitudes. Participant observation entailsinvestigating the project background, studying the general characteristics of a

beneficiary population, and living for an extended period among beneficiaries,during which interviews, observations, and analyses are recorded and

discussed.

y Pocket charts.Investigative tools that use pictures as stimuli to encourage people to assess andanalyze a given situation. Through a "voting' process, participants use the chart

to draw attention to the complex elements of a development issue in an

uncomplicated way. A major advantage of this tool is that it can be put togetherwith whatever local materials are available.

y Preference ranking.Also called direct matrix ranking, an exercise in which people identify whatthey do and do not value about a class of objects (for example, tree species or

cooking fuel types). Ranking allows participants to understand the reasons for

local preferences and to see how values differ among local groups.Understanding preferences is critical for choosing appropriate and effective

interventions.

y Role playing.Enables people to creatively remove themselves from their usual roles andperspectives to allow them to understand choices and decisions made by other

people with other responsibilities. Ranging from a simple story with only a fewcharacters to an elaborate street theater production, this tool can be used to

acclimate a research team to a project setting, train trainers, and encourage

community discussions about a particular development intervention.y Seasonal diagrams orseasonal calendars.

Show the major changes that affect a household, community, or region within a

year, such as those associated with climate, crops, labor availability and

demand, livestock, prices, and so on. Such diagrams highlight the times of

constraints and opportunity, which can be critical information for planning andimplementation.

y Secondary data review.Also called desk review, an inexpensive, initial inquiry that provides necessary

contextual background. Sources include academic theses and dissertations,

annual reports, archival materials, census data, life histories, maps, projectdocuments, and so on.


28/30

y Semistructured interviews.Also called conversational interviews, interviews that are partially structured

by a flexible interview guide with a limited number of preset questions. This

kind of guide ensures that the interview remains focused on the development

issue at hand while allowing enough conversation so that participants can

introduce and discuss topics that are relevant to them. These tools are adeliberate departure from survey-type interviews with lengthy, predetermined

questionnaires.y Sociocultural profiles.

Detailed descriptions of the social and cultural dimensions that in combination

with technical, economic, and environmental dimensions serve as a basis for

design and preparation of policy and project work. Profiles include data aboutthe type of communities, demographic characteristics, economy and livelihood,

land tenure and natural resource control, social organization, factors affecting

access to power and resources, conflict resolution mechanisms, and values andperceptions. Together with a participation plan, the sociocultural profile helps

ensure that proposed projects and policies are culturally and socially

appropriate and potentially sustainable.

y Surveys.A sequence of focused, predetermined questions in a fixed order, often with

predetermined, limited options for responses. Surveys can add value when theyare used to identify development problems or objectives, narrow the focus or

clarify the objectives of a project or policy, plan strategies for implementation,

and monitor or evaluate participation. Among the survey instruments used in

Bank work arefirm surveys,sentinel community surveillance, contingentvaluation, andpriority surveys.

y Tree diagrams.Multipurpose, visual tools for narrowing and prioritizing problems, objectives,

or decisions. Information is organized into a treelike diagram that includes

information on the main issue, relevant factors, and influences and outcomes ofthese factors. Tree diagrams are used to guide design and evaluation systems, to

uncover and analyze the underlying causes of a particular problem, or to rank

and measure objectives in relation to one another.

y Village meetings.Meetings with many uses in participatory development, including informationsharing and group consultation, consensus building, prioritization and

sequencing of interventions, and collaborative monitoring and evaluation.When multiple tools such as resource mapping, ranking, and focus groups have

been used, village meetings are important venues for launching activities,

evaluatingprogress, and gaining feedback on analysis.


29/30

y Wealth ranking.Also known as wellbeing ranking or vulnerability analysis, a technique for therapid collection and analysis of specific data on social stratification at the

community level. This visual tool minimizes literacy and language differences

of participants as they consider factors such as ownership of or use rights to

productive assets, lifecycle stage of members of the productive unit,relationship of the productive unit to locally powerful people, availability of

labor, and indebtedness.y Workshops.

Structured group meetings at which a variety of key stakeholder groups, whose

activities or influence affect a development issue or project, share knowledge

and work toward a common vision. With the help of a workshop facilitator,participants undertake a series of activities designed to help them progress

toward the development objective (consensus building, information sharing,

prioritization of objectives, team building, and so on). In project as well aspolicy work, from preplanning to evaluation stages, stakeholder workshops are

used to initiate, establish, and sustain collaboration.

Source:

World Bank Participation Sourcebook.

NGO Roles in the Project Cycle

The following table provides a summary of the specific roles which NGOs can

play at various stages of a project cycle. Each of the tasks described belowrequires specific skills and competencies (eg.: participatory methodologies,technical knowledge etc.) and individual NGOs must be selected accordingly.

Levels of NGO experience vary significantly by country and sector, and

therefore, the ability of NGOs to fulfill the taskks described below must beverified on a case-by-case basis.

Stagein Project

CyclePotential NGO Involvement

ProjectIdentification

y provide advice/information on local conditionsy participate in environmental and social assessmentsy organize consultations with beneficiaries/affected

partiesy transmit expressed needs/priorities of local

communities to project staffy act as a source, model or sponsor of project ideas


30/30

y implement pilot projects

Project Design

y consultant to the government, to local communities orto the Bank

y assist in promoting a participatory approach to projectdesign

y channel information to local populations

Financing

y co-financier (in money or in kind) of a projectcomponent

y source of funds for activities complementary to theproposed Bank-financed project

Implementation

y project contractor or manager (for delivery of services,training, construction, etc.)

y promote community participation in project activitiesy financial intermediary roley supplier of technical knowledge to local beneficiariesy advisor to local communities on how to take advantage

of project-financed goods or servicesy implementor of complementary activitiesy beneficiary of an NGO funding mechanism established

by the project

Monitoring and

Evaluation

y NGO contracted to monitor project progress orevaluate project results

y facilitate participatory monitoring and evaluationy independent/unsolicited monitoring and evaluation

Source:

World Bank, Workingwith NGOs APractical Guide to Operational

Collaboration between theW

orld Bank and Non-GovernmentalOrganziations .Operations Policy Department, World Bank, 1995, pp.29.

evaluating programme performance

Documents