financial market development (fmd) activity …

FMD Activity MEL Plan, December 2017 1

FINANCIAL MARKET DEVELOPMENT (FMD)

ACTIVITY MONITORING, EVALUATION,

AND LEARNING (MEL) PLAN - REVISED

DECEMBER 2017 This publication was produced for review by the United States Agency for

International Development. It was prepared by Management Systems International, A

Tetra Tech Company, and the Trust for Democratic Education and Accountability.

FMD ACTIVITY MONITORING,

EVALUATION, AND LEARNING PLAN

DRAFT REVISION – DECEMBER 2017

DISCLAIMER

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency

for International Development or the United States Government.


ACRONYMNS

AR Activity Result (FMD level result)

AOR Agreement Officer’s Representative

CDNS Central Directorate of National Savings

COP Chief of Party

COR Contracting Officer's Representative

DCOP Deputy Chief of Party

DCA Development Credit Authority

DCM Debt Capital Management

DO Development Objective (USAID/Pakistan result)

DQA Data Quality Assessment

EBND Electronic Bond Trading Platform

EGA Economic Growth and Agriculture

EFW External Finance Wing

eWRS Electronic Warehouse Receipts

FMD Financial Market Development

GDP Gross Domestic Product

GOP Government of Pakistan

IMF International Monetary Fund

IP Implementing Partner

IR Intermediate Result (USAID/Pakistan result)

IST Indicator Summary Table

IT Information Technology

LOA Life of Activity

MIS Management Information System

MoF Ministry of Finance

M&E Monitoring and Evaluation

ME&L Monitoring, Evaluation & Learning

MEP Monitoring and Evaluation Plan

NBFI Non-Banking Financial Institution

NOC No Objection Certificate

NSS National Savings Scheme


OCAT Organizational Capacity Assessment Tool

PIRS Performance Indicator Reference Sheets

PMEX Pakistan Mercantile Exchange

PMP Performance Management Plan

PSX Pakistan Stock Exchange

PKR Pakistani Rupees

RF Results Framework

SBP State Bank of Pakistan

SECP Securities and Exchange Commission of Pakistan

SAR Sub Activity Result (FMD level result)

SIR Sub Intermediate Result (USAID/Pakistan result)

SME Small and Medium Enterprise

SOP Standard Operating Procedure

SPV Special Purpose Vehicle

TBD To Be Determined

USAID United States Agency for International Development

USD United States Dollars

USG United States Government


CONTENTS

ACRONYMNS ..................................................................................................................................................... 1

I. INTRODUCTION TO FINANCIAL MARKET DEVELOPMENT ACTIVITY (FMD)

MONITORING, EVALUATION, AND LEARNING (MEL) PLAN ............................................ 5

PURPOSE .............................................................................................................................................. 5

ACTIVITY INFORMATION AND CONTEXT ................................................................................ 6

THEORY OF CHANGE ....................................................................................................................... 8

II. ACTIVITY MEL PLAN ................................................................................................................................... 9

PERFORMANCE INDICATORS ....................................................................................................... 9

DATA QUALITY ASSURANCE AND DATA QUALITY ASSESSMENT

PROCEDURES ..................................................................................................................... 10

REPORTING OF DATA .................................................................................................................... 10

ROLES AND RESPONSIBILITIES AND DATA COLLECTION

METHODOLOGIES ............................................................................................................ 11

DATA COLLECTION METHODOLOGIES .................................................................................. 12

GENDER M&E ................................................................................................................................... 13

CALENDAR OF MEL EVENTS ....................................................................................................... 13

LEARNING .......................................................................................................................................... 13

ANNEX A: PERFORMANCE INDICATOR REFERENCE SHEETS (PIRS) .......................................... 14

ANNEX B: ASSURING DATA QUALITY ................................................................................................... 25

DATA QUALITY CHECKLIST AND RECOMMENDED PROCEDURES .............................. 25

List of Tables

Table 1: Project Summary ............................................................................................................................. 5

Table 2: Performance Indicator TrackingTable ..................................................................................... 9

Table 3: Performance Reports ................................................................................................................... 11

Table 4: Summary of FMD's indicator data colllection methodologies .................................... 12

Table 5: Calendar of MEL events .............................................................................................................. 13

List of Figures

FIGURE 1: FMD ACTIVITY REVISED RESULTS FRAMEWORK ........................................................... 7


ACTIVITY SUMMARY

Table 1 Summarizes basic information about the Financial Market Development Activity (FMDA)

TABLE 1: PROJECT SUMMARY

Title/Field Project/Activity Information

Contract/agreement numbers AID-391-C-15-00013

Contracting/agreement officer’s

representative (COR/AOR) Nadia Dawood

Start date March 01, 2016

Completion date February 28, 2019

Location Islamabad, Pakistan

Implementing partner(s) The Pragma Corporation

USAID/Pakistan Mission Strategic

Framework objectives addressed DO 2, IR2.2: Improved Business Enabling Environment

I. INTRODUCTION TO FINANCIAL MARKET

DEVELOPMENT ACTIVITY (FMD) MONITORING,

EVALUATION, AND LEARNING (MEL) PLAN

The goal of the three-year Financial Market Development Activity (hereinafter “FMD Activity,” “FMD,”

or the “Activity”) is to strengthen Pakistan’s debt financial markets, and contribute to the expansion of

financing for private sector investment and economic growth, through targeted policy reforms that helps

create a more open and free business enabling environment.

The FMD Activity seeks to target and effectively address critical factors that limit the deepening and

diversification of overall capital market financing, and that restrain the demand for innovative capital

market instruments in these areas. FMD will work with key public and private sector institutional actors,

under USAID guidance, to both prioritize and develop a consensus on how to effectively address these

key constraints, and support implementation of technical, training and outreach activities designed to

move critical capital market reforms forward.

Purpose

The purpose of this revised MEL plan is to describe how FMDA will monitor its activities:

• Presents the linkages between indicators and related USAID/Pakistan’s Mission Strategic

Framework (MSF) sub-intermediate results (IRs), IRs, and development objective (DO);

• Includes indicators to measure progress in achieving the expected activity results;

• Describes the processes that FMDA will use to conduct performance monitoring and collect,

manage, analyze, and report data;

• Defines the M&E unit’s staff structure and performance monitoring roles;

• Provides a timetable for performance reporting; and

• Describes the data quality control mechanisms.


Annexes A-C include Performance Indicator Reference Sheets (PIRS) and tools for conducting data

quality assessments (DQAs).

Activity Information and Context

Project Context

FMD is part of the Mission’s Economic Growth portfolio that is helping the Government of Pakistan

(GOP) implement key policy/regulatory and institutional reforms affecting trade, the financial sector, and

small- and medium- sized enterprise development. In particular, FMD directly contributes to

USAID/Pakistan Mission Development Objective Two (DO2): Improved Economic Status of Focus

Populations and Sectors. The activity will directly support DO2 Intermediate Result 2.2 Improved

Business Enabling Environment, and DO2 Sub-Intermediate Results 2.2.1 Improved Ability to

Develop and Implement Reform and 2.2.2 Strengthened Private Sector and Civil Society

Engagement in Policy-Making. For a graphic depiction of the DO 2 Results Framework, please see

Figure 1 below. The end goal of the FMD activity is a strengthened debt finance and capital market.

FMD Activity Description

The FMDA project started on March 2016, and the project will now continue through the revised end

date of February 28, 2019. FMDA contributes to USAID’s DO #2, IR2.2, “Improved Business Enabling

Environment.” The goal of the FMDA activities is to support policy/regulatory and institutional

interventions in the debt and market will help promote rationalization and diversification of debt product

offerings. Main components include:

• Strengthened Debt Finance and Capital Market

• Increased adoption of sound financial management practices by key financial institutions

The FMD project was de-scoped in August 2017. De-scoped components included SME financing enabling

environment reforms and support for the issuance and secondary market trading of domestic sovereign

debt. The original performance indicators and results statements for these de-scoped components are no

longer relevant and were removed. This revised MEL Plan serves to update the previously approved MEL

plan.

Activity Location and Institutional Context

Under this activity, FMD is working in the Islamabad Capital Territory, Khyber Pakhtunkhwa province,

Punjab province, and Sindh province.

The linkages between FMDA activities and the Project Goal, Purpose and Sub-Purposes are illustrated in

Figure 1.


FIGURE 1: FMD ACTIVITY REVISED RESULTS FRAMEWORK

MSF Goal: Increased stability, democracy and prosperity for the men and women of

Pakistan

Project Purpose - Intermediate Result 2.2: Improved Business Enabling Environment

Indicator 2.2a: Number of enabling environment policies analyzed, consulted on, drafted or revised,

approved, and implemented with USG assistance

Development Objective 2: Improved Economic Status of Focus Population

Project sub-purpose – IR 2.2.1: Improved

ability to develop and implement Reforms

of Policies, Laws and Regulations

Indicator 2.2.1a: Person hours of USG-supported

training completed in trade and investment

(Standard EG.2-1)

Indicator 2.2.1b: Number of persons receiving

USG supported training on policy development,

analysis, and implementation

Project sub-purpose – IR 2.2.2:

Strengthened Private Sector and Civil

Society Engagement in Policy-Making

Indicator 2.2.2a: Number of USG-assisted

organizations that participate in legislative

proceedings and/or engage in advocacy at all

levels

Indicator 2.2.2b: Number of events for

public/stakeholders consultation to enhance

sector governance or facilitate private sector

engagement in policy making as a result of USG

assistance


Theory of Change

The revised MELP incorporates guidance provided in a “De-Scoping Order” delivered to Pragma

Corporation in August 2017. In accordance with that De-Scoping Order, FMD indicators and scope of

MELP now cover the revised timeframe from September 2017 through February 2019.

As per its revised scope of work, FMD’s theory of change is aimed at effectively addressing critical

factors that are limiting the deepening and diversification, as well as the introduction of innovative capital

market financing solutions. FMD’s current approach is based on assisting key counterparts in their quest

for policy, regulatory, operational and technical reforms to (1) support the robust expansion of the

range of tradeable debt capital instruments; (2) improve the efficiency and cost-effectiveness with which

those instruments are traded; and (3) significantly enhance confidence in the transparency and integrity

of capital market trading and clearing/settlement operations.

At the same time, FMD’s impact through its scope of work is focused to enable mobilization of capital

for marginalized agricultural communities by introduction of international state of the art electronic

warehousing receipt system in Pakistan. FMD’s work with provincial governments for second tier debt

product issuance shall initiate the first ever provincial debt financing in Pakistan.


II. ACTIVITY MEL PLAN

Performance Indicators

The expected results of FMD’s activities are measured through performance indicators. The performance indicator tracking table below

illustrates how FMD will monitor its performance data. The table includes indicator baselines, annual targets, and target justifications.

TABLE 2: PERFORMANCE INDICATOR TRACKING TABLE

#

Indicator

Baseline

FY 2017

Target

FY 2018

Target

FY 2019, February

Target

LOA Target Justification

AR

1

2.2a: Number of enabling environment

policies analyzed, consulted on, drafted or

revised, approved, and implemented with

USG assistance (EGA, Modified F)

0 9 9 9 27 Assuming these policies include all policy level interventions

within the FMD approved workplan.

SAR

1.1

2.2.1a: Person-hours of USG-supported

training completed in trade and investment

(EGA, Standard F)

0

320

M: 256

F: 64

3840

M: 3072

F: 768

1920

M: 1536

F: 384

6080

Institutional and investor level capacity building initiatives of

FMD within the approved workplan Sub-targets for FY19

males/females will be established at the end of FY18.

2.2.1b: Number of persons receiving USG

supported training on policy development,

analysis, and implementation (EGA) 0

80

M: 64

F: 16

80

M: 64

F: 16

40

M: 32

F: 8

200

FMD’s capacity and consensus building outreach events with

key financial institutions. Sub-targets for FY19 males/females

will be established at the end of FY18.

SAR

1.2

2.2.2a: Number of USG-assisted

organizations that participate in legislative

proceedings and/or engage in advocacy at

all levels (EGA)

0 6 5 2

13 FMD’s ongoing technical assistance, as per the FMD workplan,

to SECP, PSX, PMEX, NCCPL & CDC.

2.2.2b: Number of events for public/

stakeholders’ consultation to enhance

sector governance or facilitate private

sector engagement in policy making

because of USG assistance (EGA)

0 19 8 3 30

For each of the legal/regulatory development

proposal/recommendation, FMD shall organize stakeholders’

feedback sessions followed by awareness sessions.


Data Quality Assurance and Data Quality Assessment Procedures

High quality data is the cornerstone of performance monitoring systems and evidence-based decision-

making, and FMD recognizes that a robust MEL system is required to provide evidence-based results to

implement the activities of partners and donors. USAID’s Automated Directives System (ADS) provides

the standards and criteria for data quality which include validity, integrity, precision, reliability, and

timeliness. Data quality assessments ensure that USAID Missions and implementing partners are aware

of the strengths and weaknesses of data, as determined by applying the ADS data quality criteria, as well

as the extent to which the data’s integrity can be relied upon to support management decisions.

FMD’s M&E manager develops the protocols and standard procedures to ensure that data is collected in

a consistent and technically sound manner, meets USAID’s data quality standards, and can be compared

over time. The FMD M&E Expert will be responsible for assessing the strengths and weaknesses of the

chosen indicators prior to submission of each annual report, and will use the updated USAID data

quality assessment checklist (including a follow-up action tracker) to conduct this internal assessment

(see Annex B). The FMD M&E Expert is also responsible for keeping a record of all issues related to

data quality, along with actions taken.

To manage performance data, all the required data will be collected on agreed indicators based on the

sources listed in the PIRS. The performance monitoring data will be quantitative as well as qualitative.

The following categories of data sources will be used:

• FMD maintained database on training participants and stakeholder engagement: FMD will be

responsible for designing a system for collecting training participant data and for making sure that

sign-in sheets are used at all FMD-supported trainings. FMD must also design and maintain a system

for documenting characteristics of all Activity stakeholders.

• International data and statistics: The FMD Activity will take into account international research

and statistics regarding Pakistan’s economic rankings, potential, and competitiveness in capital and

debt markets development. These will be used to provide contextual information regarding the

financial market and inform dialogue with other donors and development partners. Contextual data

may help the FMD team understand trends and pattern as they fine-tune strategies and

interventions. Additionally, the Activity’s ME&L Plan will provide insight to the possible contribution

of FMD interventions to the changes in data gathered through these sources.

Reporting of Data

In order to provide regular updates to USAID and key stakeholders of the FMD Activity, various types

of reports will be generated and shared internally and externally, as agreed upon with USAID in

advance. Reports will include:

• Activity Meeting Reports: Including quarterly progress sharing meeting reports (these reports will

be shared with the COP and the senior management to keep them informed of progress on the

Activity’s results and to enable them to determine program priority areas)

• Activity Reports/Plans:

o FMD Work Plan

o Monitoring reports and/or site visit reports of M&E Unit

o Quarterly Performance Monitoring Reports, which will include progress against each ME&L

Plan indicator


o Quarterly reporting, on the USAID Pakistan “Pak Info” platform, of the indicators linked to

the EGA results framework

o Annual Performance Monitoring Reports, which will include progress against each ME&L

Plan indicator

o End of Activity Report.

FMD will submit quarterly and annual reports to USAID, as indicated in table 3. FMD will organize and

update the quarterly performance data and review the data with the COR. At the end of the fiscal year,

FMD will submit an annual performance report that will include a compilation of the year’s actual results

versus targeted results for each indicator and an explanatory narrative. FMD will submit all reports to

the COR for approval.

TABLE 3: PERFORMANCE REPORTS

Performance Report Content

Quarterly

Quarterly accomplishments; lessons learned/challenges faced and solutions

proposed; planned activities for the next quarter; performance indicator

data versus targets; analytical narrative for results under each activity;

results achieved and means of verification for each result; and photos

Weekly Weekly updates on implementation of activities.

Annual Annualized information included in the quarterly performance reports.

Roles and Responsibilities and Data Collection Methodologies

The FMD chief of party (COP) has overall responsibility for assuring that the work of the M&E unit meets

the needs of the project and USAID reporting requirements. The M&E expert is responsible for all M&E

activities and reports directly to the COP.

Implementing the MEL Plan

The M&E expert is the primary person responsible for implementing the MEL plan and is also

responsible for:

• Overseeing and providing technical guidance with respect to MEL approaches, practices, and

tools;

• Meeting with the USAID/Pakistan COR on performance management; and

• Submitting performance data and reports to USAID/Pakistan.

Data Review Process

FMD has instituted a comprehensive data review process that assesses the validity and accuracy of the

data at successive levels. Reviewed and verified data is reported on PakInfo USAID.

Reporting

FMD gives data management special consideration to produce error-free and timely reporting. The M&E

expert is responsible for providing the verified data which is then consolidated into high-quality quarterly

performance reports and annual reports by FMD and submitted to USAID/Pakistan for review.


TABLE 4: SUMMARY OF FMD'S INDICATOR DATA COLLLECTION

METHODOLOGIES

INDICATOR DATA COLLECTION METHODOLOGY

2.2a: Number of enabling environment policies analyzed,

consulted on, drafted or revised, approved, and implemented

with USG assistance Collected by the FMD technical staff

2.2.1a: Person-hours of USG-supported training completed in

trade and investment FMD training data collection forms

2.2.1b: Number of persons receiving USG supported training

on policy development, analysis, and implementation FMD training data collection forms

2.2.2a: Number of USG-assisted organizations that participate

in legislative proceedings and/or engage in advocacy at all levels FMD data collection forms

2.2.2b: Number of events for public/ stakeholders’ consultation

to enhance sector governance or facilitate private sector

engagement in policy making as a result of USG assistance

FMD data collection forms

Data Collection Methodologies

Performance monitoring plays a central role throughout the project cycle. The M&E expert is responsible

for data collection.

Planning

Specific data collection tools to be used by FMD include: training attendance sheets, training participant

feedback questionnaires, and pre- and post-training assessments; and customized data collection tools for

other performance indicators. Most data collection instruments can be modified from tested tools to

reduce cost and biases, and increase reliability.

Data Analysis

Analysis of indicator performance data is critical for continuous learning and to determine whether targets

are being met. Data is disaggregated in accordance with USAID requirements for each indicator. Common

disaggregates are by geographical region, project theme, age group, and male/female (for all indicators that

measure number or percentage of citizens). The M&E expert reviews the quality of the data received

based on the five standards of data quality. The analysis also provides evidence-based guidance for

monitoring and taking corrective action when necessary.

Various tools will be used for the analysis of data. FMD will maintain an Excel database of stakeholders

and their characteristics, and training participants and their characteristics to facilitate analyses across

organizations and other disaggregation criteria. Every effort will be made to quantify data that can be

quantified. Beyond the standard use of charts, graphs and tables, where possible, the team will use creative

ways to present qualitative data to highlight patterns and relationships, such as mappings and network

diagrams.


Gender M&E

FMD MEL plan contains two performance indicators that are gender sensitive, and will be disaggregated

by sex. These indicators specifically target women’s participation in capacity building and skill development

trainings. All gender sensitive indicators will be analyzed to assess the different impact on men and women.

2.2.1a Person hours of USG-supported training completed in trade and investment (EG.2-1)

2.2.1b Number of persons receiving USG supported training on policy development, analysis, and

implementation

Calendar of MEL Events

FMD will submit to USAID/Pakistan quarterly and annual performance reports, and a final closeout plan

and report.

TABLE 5: CALENDAR OF MEL EVENTS

Performance Report Calendar Year Date Comments

Quarterly

July 31, 2016

January 31, 2017

April 30, 2017

July 31, 2017

January 31, 2018

April 30, 2018

July 31, 2018

The Quarterly Report due to

be submitted at the same time

as the Annual Report will be

combined with the Annual

Report.

Annual

October 31, 2016 (2 quarters only)

October 31, 2017

October 31, 2018

The report will aggregate

annual data on indicators and

other results along

with the 4th quarter report.

Final

No later than 90 days, following the

completion date of the agreement.

February 28, 2019

The final report will cover the

entire period of the USAID-FMD

agreement

Learning

FMD prides itself on being a committed learning organization that relies on performance data to make

evidence-based decisions on programming. Its performance monitoring system provides the platform to

facilitate learning and adaption throughout activity implementation. FMD envisages a monitoring,

evaluation and learning system that helps identify promising interventions in advance so that they

can effectively be implemented during the course of the activity, to generate approved anticipated

results. Having data available about how well a particular intervention or policy works will provide

useful information for formulating and justifying resource requests. This will facilitate judicious

allocation of resources to undertake interventions that will provide the strongest impact.


ANNEX A: PERFORMANCE INDICATOR REFERENCE

SHEETS (PIRS)

PERFORMANCE INDICATOR REFERENCE SHEET (PIRS)

Indicator: 2.2-a Number of enabling environment policies analyzed, consulted on, drafted or revised, approved, and implemented with USG assistance

Development Objective (DO): 2 Improved Economic Status of Focus Populations and Sectors

Intermediate Result (IR): 2.2 Improved Business Enabling Environment

Sub-Intermediate Result (Sub-IR): N/A

Relationship between the Sub-IR and IR or IR and DO: Enter the explanation of the linkage between the lowest level of result represented by the indicator, and the next level of result up; address the “so what?” question to move from outputs to outcomes, or outcomes to impact; explain in terms of the development hypotheses, do not simply restate the structure of the Results Framework.

An improved business enabling environment will allow enterprises to achieve their potential, thereby increasing incomes and employment, because such an improvement indicates that major legal, policy, regulatory, and administrative barriers that inhibit their performance are no longer in effect. The indicator measures the number of policies/ regulations/ administrative procedures in the various stages of progress towards an enhanced enabling environment for agriculture whose sub-elements are specific policy sectors. This indicator is easily aggregated upward from all operating units.

INDICATOR DESCRIPTION

Precise Definition(s): Enter the precise definition of the indicator so it can be operationalized; define all terms, elements, implied actions and calculations; [for example, “farmers using better production techniques” – define “better production” and “techniques”. Describe how this will be determined – e.g. Index, scale, standards]. For indicators that are percent or proportions explain how it will be calculated and what will serve as the numerator and denominator. If the indicator is cumulative, made up of stages or phases, or is a yes-no, please specify this and explain the stages/phases or how it is cumulative. If it is a Standard Program Structure (“F”) Indicator, use and if necessary, refine the standard definition.

The indicator counts the number of economic growth policies related to the institutional architecture for improved policy formulation, the enabling environment for private sector investment, agricultural trade, agriculture input provision, land and natural resource management, or food and nutrition that have completed one or several of the following 5 steps or processes: 1. Underwent analysis (review of existing policy and/or proposal of new policy); 2. Underwent public debate and/or consultation with stakeholders on the proposed new or revised policy. This can also include proposed repeal of an existing policy; 3. Were newly drafted or revised; 4. Received official approval (legislation/decree) by the relevant authority (legislative or executive body) of a new, revised, or repealed policy; 5. Were fully and effectively implemented by the relevant authority (this includes USG support to implementing the effective repeal of a policy). Policies can include laws, legal frameworks, regulations, administrative procedures, or institutional arrangements. Note that the indicator has been revised to acknowledge that these processes are not always linear: newly drafted laws can be defeated by a legislative body and require redrafting or new analysis; or approved regulations can prove difficult to implement and need to be revised. Because of this non-linear approach, double counting is no longer a concern and is in fact appropriate: Partners should indicate if multiple processes/steps were completed in a given year, as this more accurately represents work under a given activity. The disaggregate “Total policies passing through one or more processes/steps of policy change” will count the total number of policies that completed any process/step, regardless of the number of processes/steps each policy completed during the reporting year.


Full and effective implementation must meet the following criteria: (1) The policy must be in force in all intended geographic locations and at all intended administrative levels with all intended regulations/rules in place (“full”); (2) Any ongoing activities or tasks required by the policy (e.g., various kinds of inspection, enforcement, collection of documents/information/fees) are being executed with minimal disruptions (“effective”). For example, a new business registration procedure that has been rolled out to just four of six intended provinces would not meet these criteria (not full), nor would a new customs law that is on the books but is not being regularly enforced at the border (not effective). For regional missions, approval (step 4) counts any regionally agreed policies that have been regionally approved (i.e., reached the minimum number of signatory countries to be passed) during the reporting year. Full and effective implementation (step 5) would count any regionally agreed policy for which all countries falling under the policy’s jurisdiction have fully and effectively implemented the policy. To capture individual countries’ progress toward full and effective implementation of regional policies, use FTFMS-only indicator 4.5.1 (TBD9). This is a modified form of standard indicator “EG.3.1-12 Number of agricultural and nutritional enabling environment policies analyzed, consulted on, drafted or revised, approved, and implemented with USG assistance” where it covers broadly Economic Growth and Agriculture Program of USAID/Pakistan

Unit of Measure: Type of Indicator: Category: Desired Direction:

Enter unit of measure (e.g. “number of___”, “percent of

___” etc.)

Enter “output”, “outcome” or “impact”.

Enter “Standard F” or “Custom”

If “Standard F indicator”, enter

the number

Enter “increasing”, “decreasing” or ‘static” to indicate the direction of

success result.

Number of policies Output (Stages 1 & 2) Outcome (Stages 3-5)

Custom (Modified)

EG.3.1-12 Increasing

Aggregation Process: If indicator will be collected by more than one source, explain how the data will aggregate across these multiple sources (e.g. in the case of # of jobs, demonstrate how data definitions for what is counted as a “job” is consistently interpreted across sources and specify that he data reported by each partner will be added together for a combined total; or in the case of a stage of phase indicator, state how data from different partners will combine into one final data). Also specify the timeline for aggregation (e.g. all sources will be added together each quarter). All data are added together across all data collection/reporting entities at respective stage. Formula of aggregation for each stage will be: Ps1,1 + Ps1,2 + Ps1,3 + . . . + Psi,j = Ps1

Ps2,1 + Ps2,2 + Ps2,3 + . . . + Psi,j = Ps2

Ps3,1 + Ps3,2 + Ps3,3 + . . . + Psi,j = Ps3

Ps4,1 + Ps4,2 + Ps4,3 + . . . + Psi,j = Ps4

Ps5,1 + Ps5,2 + Ps5,3 + . . . + Psi,j = Ps5

Ps is Policy stage, i is subscript for stage and j is subscript for partners/projects.

Disaggregates: Enter all disaggregation titles/ categories and values (e.g. title: Household Head Type; values: Female no Male Adult households, Male no Female Adults households, Male and Female Adult households, Child no Adult households.)

Policy Stages Process/step: analysis; stakeholder consultation/public debate; drafting or revision; approval (legislative or regulatory); full and effective implementation; Total policies passing through one or more processes/steps of policy change

DATA COLLECTION, STORAGE, and ANALYSIS

Name of IP/ Responsible Party for Data Collection:

Frequency of data collection: Enter how often the data will be collected (Weekly, Monthly, etc.)

PACE, PEEP, PREIA, Satpara Development Project, Firms, PSSP, FMD

Annually

Data Source: Enter where IP obtains data (e.g. self-collected, GOP records or private sector).

Data Entry Frequency into PakInfo: Enter the anticipated frequency of regular data entry into PakInfo (e.g. Quarterly, Annually, etc)

Responsible Party for Data Entry into PakInfo: Enter who will be responsible for inputting and submitting data via PakInfo.


Partner will collect data as regular M&E activities

Annually PACE, PEEP, PREIA, Satpara Development Project, Firms, PSSP, FMD

Data collection method: Enter the tools and methods to be used for data collection and indicate for each method who (IP, USAID or third party) will collect the data. (e.g. telephone survey of household sample, reading assessment administered by third-party, sign-in sheets of training participants by IP)

Project will recode progress of their policy work and collect data about number of policies on specific stage of development through regular monitoring exercise.

Data Analysis Plan: Enter how the data will be analyzed, including description of methodology (e.g. descriptive, comparative, qualitative or quantitative) as well as who will participate in the data analysis process (e.g. activity manager, chief of party, other stakeholders, GOP representatives, etc.)

Descriptive, comparative (pre/post activity, sector wise, grants wise), quantitative (numbers) Simple count of reports and documents by monitoring &evaluation team.

DATA QUALITY

Data Quality Assessment (DQA): Enter the date the DQA was conducted and the person who conducted the DQA

Date: (MM/YY) DQA completed by:

10/2016 Third Party (Monitoring and Evaluation Project with MSI)

Key Data Quality Limitations (if any) and Actions Planned to Address Those Limitations: Enter data limitations identified in the data quality assessment process related to the five quality standards, namely validity, integrity, precision, reliability and timeliness; discuss the significance of data weakness that may affect the conclusions about the extent to which performance goals have achieved; describe corrective actions planned or taken for addressing data weakness.

CHANGES & UPDATES

Date (MM/YY)

Name Enter who made

updates Change or Update Made:

Reason for Change or Update:

08/2016 Nazim Change in Definition and language of indicator

New F-Framework

12/12/17 McHugh Slight modification to correct partner/mechanism name

Slight correction



Indicator: 2.2.1a Person-hours of USG-supported training completed in trade and investment



Sub-Intermediate Result (Sub-IR): 2.2.1 Improved Ability to Develop and Implement Reform of Policies, Laws, and Regulations


Improving the ability of government and professional bodies to develop and implement reforms of policies, laws, and regulations will result in an improved business environment, provided that those bodies are willing and able to make productive use of such improvements.



This indicator uses the following equation to express the number of USG-supported training hours that were completed by training participants: Hours of USG-supported training course x Number of people completing that training course Support from the USG: This indicator counts training hours that were delivered in full or in part as a result of USG assistance. USG support may include provision of funds to pay teachers, providing hosting facilities, or other key contributions necessary to ensure training was delivered. This indicator does not automatically count any course for which the USG helped develop the curriculum, but rather focuses on delivery of courses that was made possible through full or partial funding or in-kind assistance from the USG. People: Only people who complete the entire training course are counted for this indicator. Training: Training is defined as sessions in which participants are educated according to a defined curriculum and set learning objectives. Sessions that could be informative or educational, such as meetings, but do not have a defined curriculum or learning objectives, are not counted as training. Trade and investment is defined as the policies, laws, regulations, and administrative practices affecting international trade and investment and the public and private sector institutions that support sustained, locally driven improvements in the trade environment that benefit both men and women; and the collection of services, technologies, equipment, and techniques used to enhance private sector response to international trade and investment opportunities.



___” etc.)




the number


success result.

Number of days Output Standard EG.2-1 Increasing

Aggregation Process: All data are added together across all data collection/reporting entities.


Disaggregates: Gender (Male, Female)




PACE, PREIA, PEEP, FMD Quarterly


Data Entry Frequency into PakInfo: Enter the anticipated frequency of regular data entry into PakInfo (e.g. Quarterly, Annually, etc.)


Sign in sheets and M&E forms recording training events

Quarterly PACE, PREIA, PEEP, FMD


Specific M&E tools (trackers/templates, attendance sheets, training review forms and reports on status/progress on activities will be filled by partners.

Data Analysis Plan: Enter how the data will be analyzed, including description of methodology (e.g. descriptive, comparative, qualitative or quantitative) as well as who will participate in the data analysis process (e.g. activity manager, chief of party, other stakeholders, GOP representatives, etc.) Data will be analyzed using comparative and descriptive statistics and presented in the forms of tables, graphs and reports.

DATA QUALITY



TBD


OTHER NOTES / NEXT STEPS

If the indicator is pending, explain why and expected date when collection will begin. As appropriate, indicate any other important information about the indicator and/or its data collection as well as actions needing to be taken.

CHANGES & UPDATES

Date (MM/YY)

Name Enter who made



6/2016 Nazim Ali Drafted as new PIRS


Slight correction



Indicator: 2.2.1b Number of persons receiving USG supported training on policy development, analysis, and implementation


Intermediate Result (IR) - # and Title: 2.2 Improved Business Enabling Environment

Sub-Intermediate Result (Sub-IR): 2.2.1 Improved Ability to Develop and Implement Reform of Policies, Laws, and Regulations

Relationship between the Sub-IR and IR or IR and DO: Enter the explanation of the linkage between the lowest level of result represented by the indicator, and the next level of result up; address the “so what?” question to move from outputs to outcomes, or outcomes to impact; explain in terms of the development hypotheses, do not simply restate the structure of the Results Framework. Improving the ability of government and professional bodies to develop and implement reforms of policies, laws, and regulations will result in an improved business environment, provided that those bodies are willing and able to make productive use of such improvements.



Indicator counts persons undergoing training relating to policy development, analysis, and implementation. Beneficiaries may include government officials trained on policy development or implementation, journalists trained on reporting on economic policy issues, and researchers trained on selecting and carrying out applied research on topics related to the policy-making process. This indicator does not automatically count any course for which the USG helped develop the curriculum, but rather focuses on delivery of courses that was made possible through full or partial funding from the USG. People: Only people who complete at least 70% of the entire training course are counted for this indicator. Training: Training is defined as sessions in which participants are educated according to a defined curriculum and set learning objectives. Sessions that could be informative or educational, such as meetings, but do not have a defined curriculum and learning objectives are not counted as training.



___” etc.)




the number


success result.

Number of persons Output Custom Increasing

Aggregation Process: If indicator will be collected by more than one source, explain how the data will aggregate across these multiple sources (e.g. in the case of # of jobs, demonstrate how data definitions for what is counted as a “job” is consistently interpreted across sources and specify that he data reported by each partner will be added together for a combined total; or in the case of a stage of phase indicator, state how data from different partners will combine into one final data). Also specify the timeline for aggregation (e.g. all sources will be added together each quarter).

All data are added together across all data collection/reporting entities.


Gender; male, female





PACE, PREIA, PEEP, PSSP, DOC632b, Trade Project

Quarterly




Sign in sheets and M&E forms recording training events

Quarterly PACE, PREIA, PEEP, PSSP, DOC632b, Trade Project


Specific M&E tools (trackers/templates, attendance sheets, training review forms and reports on status/progress on activities will be filled by partners.


Data will be analyzed using comparative and descriptive statistics and presented in the forms of tables, graphs and reports.

DATA QUALITY







CHANGES & UPDATES

Date (MM/YY)

Name Enter who made





Indicator: 2.2.2-a Number of USG-assisted organizations that participate in legislative proceedings and/or engage in advocacy at all levels


Intermediate Result (IR) - # and Title: 2.2 Improved Business Enabling Environment

Sub-Intermediate Result (Sub-IR): 2.2.2 Strengthened Private Sector and Civil Society Engagement in Policy-Making


In order to improve the business enabling environment, it is necessary that the private sector and civil society be aware of policy-related issues and able to engage with policy-makers to make their views on these issues known. Such activity forms the “demand” side of the policy development process that results in good policy outcomes.



The number of organizations (NGOs) that participate in sessions of federal or regional parliaments (or their committees) or engage in advocacy activities in the public domain that have received USG assistance on advocacy and civic participation. For an organization to be counted, it must be on record as having participated in such parliamentary sessions or engaged in advocacy activities, such as participating in private or public meetings with policy-makers, publishing opinion pieces in widely circulated newspapers or Web sites, engaging in public (e.g., televised or broadcast) debates, and so on. It must also have received USG assistance (e.g., participated in training events) aimed specifically at building its capacity to engage in such activities.



___” etc.)




the number


success result.

Number of organizations Output Custom Increasing




Non




PACE, PREIA, PEEP, SDP, PSSP, TAP, DOC, FMD

Quarterly





Partners’ reports. Quarterly PACE, PREIA, PEEP, SDP, PSSP, TAP, DOC, FMD


Qualitative and quantitative analysis; specific M&E tools (trackers/templates); through beneficiary registration, attendance sheets and training evaluation forms.


Mixed methods using quantitative and qualitative data. Automated generation of output/ summary tables.

DATA QUALITY



10/2016 Third Party MSI


CHANGES & UPDATES

Date (MM/YY)

Name Enter who made



1/2017 Nazim Ali Updated partners reporting and DQA dates, removed disags


Slight correction



Indicator- # and Title: 2.2.2b. Number of events for public/stakeholders consultation to enhance sector governance or facilitate private sector engagement in policy making as a result of USG assistance



Sub-Intermediate Result (Sub-IR): 2.2.2 Strengthened Private Sector and Civil Society Engagement in Policy-Making

Relationship between the Sub-IR and IR or IR and DO: Enter the explanation of the linkage between the lowest level of result represented by the indicator, and the next level of result up; address the “so what?” question to move from outputs to outcomes, or outcomes to impact; explain in terms of the development hypotheses, do not simply restate the structure of the Results Framework. In order to improve the business enabling environment, it is necessary that the private sector and civil society be aware of policy-related issues and able to engage with policy-makers to make their views on these issues known. Such activity forms the “demand” side of the policy development process that results in good policy outcomes. This measures the drafting of critical policy/procedural improvements and presenting them for broad stakeholder consultation, which help to further sector reform efforts in a way to build broad consensus and thus help ensure reforms are successful and sustainable.



This indicator counts number of events held to consult/present a broadly representative cross-section of stakeholders (e.g. private sector and civil society) on policy issues as a result of USG assistance. These events include conferences, town hall meetings, debates, advocacy events which are arranged to present current policy status of sector/subsector, policy analysis, and proposed changes in policy.



___” etc.)




the number


success result.

Number of events Output Custom Increasing




District; EGA Type of Event (Conference, Workshop, Seminar)




PACE, PEEP, PREIA, GAIN, PSSP, DOC632b, USDA632b, FMD

Quarterly





Partners’ reports Quarterly PACE, PEEP, PREIA, GAIN PSSP, DOC632b, USDA632b, FMD


Qualitative and quantitative analysis; specific M&E tools (trackers/templates); through beneficiary registration, attendance sheets and training evaluation forms.


Mixed methods using quantitative and qualitative data. Automated generation of output/ summary tables.

DATA QUALITY







CHANGES & UPDATES

Date (MM/YY)

Name Enter who made



1/2017 Nazim Ali Updated partners reporting and DQA dates


Slight correction


ANNEX B: ASSURING DATA QUALITY

Data Quality Checklist and Recommended Procedures

USAID Mission Name:

Title of Performance Indicator:

Linkage to Foreign Assistance Standardized Program Structure, if applicable (i.e. Program Area,

Element, etc.):

Result This Indicator Measures (i.e., Specify the MSF Development Objective, Intermediate Result,

etc.):

FMD Data Source(s):

Partner or Contractor/ Source Who Provided the Data:

Period for Which the Data Are Being Reported:

Is This Indicator a Standard or Custom Indicator? __ Standard Foreign Assistance Indicator

___ Custom (created by USAID/Pakistan; not

standard)

Data Quality Assessment methodology:

Date(s) of Assessment:

Assessment Team Members:

USAID/Pakistan Verification of DQA

Team Leader Officer approval

X_______________________________________

S. No Elements of Data Quality Acceptable?

Comments Yes No

VALIDITY – Data should clearly and adequately represent the intended result.

1

Does the information collected measure

what it is supposed to measure? (E.g., A

valid measure of overall nutrition is

healthy variation in diet: Age is not a

valid measure of overall health.)

2

Do results collected fall within a plausible

range? (E.g., A plausible range for height

would not be 8 feet 11 inches.)



Comments Yes No

3

Is there reasonable assurance that the

data collection methods being used do

not produce systematically biased data

(e.g., consistently over- or under-

counting)?

4

Are sound research methods being used

to collect the data? (E.g., if the result is a

change in perception, a participant sign-

in sheet would not be a sound research

method to collect the data.)

INTEGRITY – Data collected should have safeguards to minimize the risk of transcription error or

data manipulation.

1

Are procedures or safeguards in place to

minimize data transcription error? (E.g.,

entering the same dataset twice; training

of data staff; double-keying; rechecking

by a supervisor; taking samples and

comparing the entered data to the

original source.)

2

Is there independence in key data

collection, management, and assessment

procedures? (E.g., if the same staff

collects the data, enter it into MIS, and

review and analyze the data, then there is

no independence in the procedures, and

bias may be an issue.)

3

Are mechanisms in place to prevent

unauthorized changes to the data? (E.g.,

can any staff access the database or

public drive and change the data?)

PRECISION – Data have a sufficient level of detail to permit management decision making; e.g. the

margin of error is less than the anticipated change.

1

Has the margin of error been reported along

with the data? (Only applicable to results

obtained through statistical samples.)

2

Is the margin of error less than the expected

change being measured? (E.g. If a change of

only 2% is expected and the margin of error

in a survey used to collect the data is +/- 5%,

then the tool is not precise enough to detect

the change.)



Comments Yes No

3

Is the data collection method/tool being used

to collect the data fine-tuned or exact

enough to register the expected change?

(E.g. A yardstick may not be a precise

enough tool to measure a change of a few

millimeters.)

4

Is the detail of the data aligned with

management decision making

requirements? (E.g., A percentage score

of 60.43527 may not be necessary when

analyzing data on citizen perceptions,

but it might be critical when looking at

water quality.)

RELIABILITY – Data should reflect stable and consistent data collection processes and analysis

methods over time.

1

Is a consistent data collection process used

from time to time, location to location, data

source to data source (if data come from

different sources)?

2

When the same data collection method is

used to measure/observe the same thing

multiple times, is the same result

produced each time? (E.g., A ruler used

over and over always indicates the same

length for an inch.)

3

Are data collection and analysis methods

documented in writing and being used to

ensure the same procedures are followed

each time? (This could be included in the

PIRS, the M&E plan, or in some other

handbook or guide (e.g., a guide for

uploading data into a database/MIS

system).

4 Are data limitations and quality problems

clearly described in final reports?

TIMELINESS – Data should be available at a useful frequency, should be current, and should be timely

enough to influence management decision making.

1 Are data available frequently enough to

inform program management decisions?

2 Are the data reported the most current?



Comments Yes No

3 Are the data reported as soon as possible

after collection?

SUMMARY

Based on the assessment relative to the five standards, what is the overall conclusion regarding the

quality of the data?

Significance of limitations (if any):

Actions needed to address limitations prior to the next DQA (given level of USG control over data):

IF NO DATA ARE AVAILABLE FOR THE

INDICATOR

COMMENTS

If no recent relevant data are available for this indicator, why not?

What concrete actions are now being taken to collect and report these data as soon as possible?

When will data be reported?

Recommendations for Conducting Data Quality Assessments

1. Data Quality (DQ) assessor should make sure that they understand the precise definition of the

indicator by checking the Performance Indicator Reference Sheet. Please address any issues of ambiguity

before the DQA is conducted.

2. DQ assessor should have a copy of the methodology for data collection in hand before assessing the

indicator. For USAID Missions, this information should be in the PMP’s Performance Indicator Reference

Sheets for each indicator. Each indicator should have a written description of how the data being

assessed are supposed to be collected.

3. Each implementing partner should have a copy of the method of data collection in their files and

documented evidence that they are collecting the data according to the methodology.

4. DQ assessor should record the names and titles of all individuals involved in the assessment.

5. Does the implementing partner have documented evidence that they have verified the data that has

been reported? Partners should be able to provide USAID with documents (process/person conducting

the verification/field visit dates/persons met/activities visited, etc.) which demonstrates that they have

verified the data that was reported. Note: Verification by the partners should be an ongoing process.


6. The DQ assessor should be able to review the implementing partner files/records against the

methodology for data collection laid out in the PMP (for USAID Missions only). Any data quality

concerns should be documented.

7. The DQ should include a summary of significant limitations found. A plan of action, including timelines

and responsibilities, for addressing the limitations should be made.

financial market development (fmd) activity …

Documents