southern african development community [sadc] capacity ... · one of the key results areas for the...
TRANSCRIPT
Southern African Development Community [SADC]
Capacity Building in Open and Distance Learning [ODL] Project
Draft
SADC ODL Monitoring and Evaluation [M&E] Framework
Directorate of Social and Human Development and Special Programmes
[SHD&SP]
July, 2010
2
Table of Contents
1. Introduction ……………………………………………………… 3
2. Background ……………………………………………………… 3
3. Rationale for a SADC ODL M&E Framework ……………….. 4
3.1 Summary of Assessment Findings on the Status of
ODL M&E Systems in the SADC Region ………………………5
4. The Framework …………………………………………………… 6
4.1 Current ODL Indicators being Tracked by SADC MS ………. 6
4.2 Gaps in Current ODL Indicators ………………………………..11.
4.3 Criteria for Selecting Core ODL Indicators …………………… 11
4.4 Core ODL Indicators ……………………………………………. 12
5. Data Management and Utilisation …………………………….. 18
6. Domestication of the ODL M&E Framework ………………… 19
6.1 Role of SADC Member States ……………………………… 19
6.2 Role of SADC Secretariat …………………………………… 19
7. Reporting ………………………………………………………….. 19
3
1. Introduction
This is a Southern African Development Community (SADC) Regional Monitoring and
Evaluation (M&E) Framework for Open and Distance Learning (ODL). The regional
framework has seven sections, that is, introduction; background; rationale for a regional
ODL M&E framework; the framework; data management and utilisation; reporting; and
Operationalisation of the framework.
The background section presents the role of ODL programmes in the SADC region and
introduces the SADC Capacity Building Project in Open and Distance Learning that is
being funded by the African Development Bank. This is followed by the rationale for a
regional framework. This section provides reasons why it is important to have a regional
M&E framework and further describes the process that was used in developing the
framework. The framework and the Indicator Matrix are presented in section 4. Data
generated by Member States based on the core indicators contained in the framework
have to be analyzed, managed, disseminated and utilized to inform planning and
development of effective ODL programmes. These issues are presented in section 5 of
this document. The process of reporting data by Member States (MS) to the SADC
Secretariat and feedback to MS by the SADC Secretariat is described in section 6. The
last section describes the process that is required for the effective Operationalisation of
the framework.
2. Background
The SADC region is an economic grouping of fifteen Member States. These are Angola,
Botswana, Democratic Republic of Congo [DRC], Lesotho, Madagascar, Malawi,
Mauritius, Mozambique, Namibia, Seychelles, Swaziland, South Africa, United Republic
of Tanzania, Zambia and Zimbabwe. The raison d’être for the economic block is to
facilitate regional economic growth and integration and ultimately reduction of poverty.
One of the key results areas for the SADC region is education and the goal for the
education sector is to increase access to education by the population of the region.
The most common mode of education delivery in the region has been conventional.
However, available evidence shows that this mode of education delivery cannot meet
the educational demands in the region. In order to increase the supply of education in
response to increased demand, SADC MS started independently to develop and deliver
education through the Open and Distance Learning mode. The SADC Secretariat, with
financial support from the African Development Bank embarked on the implementation
of a Capacity Building in Open and Distance Learning (ODL) Project in 2006. The
4
purpose of this project is, to facilitate the development and delivery of harmonised and
effective ODL programmes in the region. It is hoped that the SADC ODL project would
contribute to increased access to education in general and specifically, harmonize the
development and delivery of ODL programmes.
3. Rationale for a SADC ODL M&E Framework in the SADC Region
It has been indicated above that SADC MS introduced ODL programmes independently,
and these MS are tracking and reporting on different indicators, a situation that makes it
difficult for the SADC region to monitor progress being made in the delivery of ODL
programmes in the region.
The education sector has global, continental and regional commitments with specific
indicators and targets. All the SADC MS are signatories to these commitments and
contribute to the realization of the targets through the delivery of education using both
the conventional and ODL modes. However, the contribution of ODL towards these
targets is hardly measured. Thus, the progress in implementing commitments is being
under-estimated. This calls for the development of an ODL specific M&E framework for
the region that will objectively measure progress in the implementation of ODL
programmes.
Available evidence shows that there are negative perceptions towards ODL. ODL is
perceived to be inferior and as the last resort for those who fail to proceed with their
studies through the conventional systems. These negative perceptions are largely due
to the fact that the progress and achievements made by ODL programmes have not
been documented and disseminated. An ODL M&E framework with clearly defined core
indicators can go a long way in informing the general populace of the SADC region
about benefits associated with ODL
In order for ODL programmes to have sustainable impact at the regional level, there is
need to harmonise both the design and deployment of ODL programmes. Thus,
development of an ODL policy framework and a regional ODL M&E framework would,
contribute in important ways to the harmonization process. All the issues raised in this
section point to the need for a regional ODL M&E framework with clearly articulated
indicators that will be tracked and reported on by all SADC MS. The SADC ODL Project
made a provision for the development of such an M&E framework.
5
3.1 Summary of ODL M&E Assessment Findings
As already alluded to above, in order to objectively develop an ODL M&E framework for
the region, an assessment was conducted to establish the status of ODL M&E systems
in SADC MS. The assessment was conducted between January and March 2010. This
section summarises the key findings that informed the content of this M&E framework.
Monitoring and Evaluation Policies: At Member States level, it was found that there
were no Monitoring and Evaluation policies in all the countries visited during the
assessment with the exception of Mozambique. On the other hand, at institutional level,
all the institutions visited had ODL Monitoring and Evaluation policies.
Monitoring and Evaluation Indicators: It was found that there were six main
components that were monitored and evaluated by ODL programmes in the region.
These were basic information or data, programme process, programme performance,
staff, output and examination management. Impact components such as further
studies, promotions, and community participation were not monitored and evaluated.
Institutional Collaboration: Few ODL institutions in the SADC Region collaborated in
the development and delivery of ODL programmes. All the institutions visited,
expressed the wish to collaborate in staff-student exchange and training in ODL M&E.
Reporting Mechanism: Currently there is no formal reporting of national ODL progress
by MS to the SADC Secretariat. All the MS visited expressed the need for establishing a
formal reporting mechanism between the SADC Secretariat and Member States.
According to MS, this reporting system should have a feedback loop. To that end, the
Ministries of Education expressed a willingness to identify focal point persons with the
responsibility to submit the national ODL reports to the SADC Secretariat..
Challenges: SADC Member States face challenges in ODL Monitoring and Evaluation.
These include inadequate resources, financial, human and material; limited capacity in
ODL M&E, hence the need for training in ODL M&E; lack of skills in identifying what to
monitor and how to monitor and evaluate ODL programmes; and inadequate ICT
equipment and materials
.
6
4. The Framework
This section presents indicators that are currently being tracked by ODL programmes in the SADC region. It further describes that gaps that are there in terms of the indicators that need to be addressed. This is followed by a section that presents the criteria used to select core indicators that are contained in the current framework. The last sub-section presents the core indicators to be tracked and reported on by all SADC MS. Table 4.1 below shows the components, sub-components that are currently being monitored and evaluated. It further presents whether the extend to which the components and sub-components are disaggregated.
4.1 Current ODL Indicators being Tracked by SADC MS
Table 4.1: ODL Components and Sub-Components being Monitored by SADC MS
7
Component
Sub-component Sources of Data Desegregation by
Sector
SE* TE HE TVET
Basic
information/data
Age Review of records √
Gender Review of records √ √ √ √
Occupation Review of records √ √ √ √
Enrolments by
programme
Review of records √ √ √ √
Initial course
choice
Review of records √ √ √ √
Sponsorship
profile
Review of records √ √ √ √
Programme
process
Course approval External assessor reviews, syllabi review
committees, college-based assessments
- √ - -
Curriculum Visits to centres, through assignments, tests
and assignments, examinations, interim visits,
internal and external examiners
- √ √ -
Quantity of
modules to be
produced
Physical checks, determined by enrolments,
needs analysis data
- - - -
Quantity of
modules produced
Enrolment projections - √ - -
Targets of course
materials
Check with learners, needs assessment - - - -
Quality of course
materials produced
Editing by experts, content reviews, learners’
inputs, proof reading, analysis of syllabi and
reviews
√ - - -
Course material
distribution
Through regional centres, college registers,
enrolment lists, check with dispatch policies,
learners physically collect from college due to
lack of funds or unreliable postal services
√ √ √ √
Comprehension by
learners
Learner-tutor feedback, quality of assignments,
performance in examinations
√ √ √ √
Praxis Assignments, learner-tutor feedback, end of
course examinations
- √ - -
Use of ICT by staff Citing websites, downloading, communicating
through ICT, reports, interviews, completion of
ED 46 forms, monitor through passwords,
- - - -
Use of ICT by
learners
Password based logging, learner feedback, e-
mails, cell phone communication where
applicable, computer application tests, interim
- - - -
9
Completing of
assignments by
learners
Records of assignments, carrying out audits,
check registers during interim visits and
external examinations, learners’ portfolios,
checking deadline compliance, using learner
information management data base, feedback
from learners
√ √ √ √
Tutor comments
on assignments
During assignment moderation, feedback from
tutors, during external examining
√ √ √ √
Marking
assignments by
tutors
Feedback from learners, random physical
checks
√ √ √ √
Assignment
deadlines
Review of records √ √ √ √
Turn around time Physical inspection, academic boards, during
panel marking
- - - -
Learner
reproduction of
assignments
Using learner information management data
base, physical checks, giving different
assignments to learners, heavy penalties on
cheating
- - - -
Gender sensitivity
in learning
materials
Review of ODL learning materials, editing - - - -
HIV and AIDS in
learning materials
Compulsory HIV and AIDS course, external
assessment, incorporates in syllabi, reflected in
health and life skills course
- - - -
Tutor-learner
tutorial attendance
Registers, roll call, sessions are compulsory for
both learners and lecturers
√ √ √ √
Programme
performance
Availability of
learning centres
Central/regional records √ √ √ √
Use of facilities at
learning centres
Review of regional centre reports - - - -
Attendance at
learning centres
Attendance registers - - - -
Use of library Review of records √ √ - -
Staff
Staff-learner ratios Monthly staff returns, records on websites,
national and regional records, interim visits, ED
46 returns
√ √ √ √
Staff qualifications Ensuring appropriate entry qualifications at
recruitment or entry level, submission of
certificates or transcripts, staff qualification
audits, use of ED 46 forms, checks during
interview visits, evaluation by national quality
assurance
√ √ √ √
10
Staff training in
ODL and M & E
On entry, recruitment stage, staff audit, ED 46
forms
- √ - -
Staff attendance of
weekend sessions
Attendance register, weekly reports, interim
visits, learners’ feedback, regional learning
centre management reports
√ √ √ √
Facilitations of
tutorials at learning
centres
Attendance registers, weekly reports, interim
visits, students feedback, regional centres
management reports
√ √ √ √
Delivery of tutorials Attendance registers, interim visits and reports,
regional centres management reports
√ √ √ √
Staff research
output
No M&E done by all institutions - - - -
Resources
Adequacy of
human resources
Annual appraisals, ED 46 regional centre
reports
√ √ √ √
Utilisation of
resource materials
Physical counts of learners reports, visits to
centres, interim visits, regional centre reports,
ED 47 returns, asset management and finance
committee reports
- √ - -
Equipment supply
and utilisation
Checking inventory records, financial resource
management reports, on site visits, regional
centre reports, ministry of education audits
- √ - -
Financial resource
management
Finance committee reports, ministries of
education audits
- - - -
Examination
management
Setting
examinations
Moderation panels, moderation committees - - - -
Selection of
examination
moderators
Academic board, quality assurance unit
committees
- √ - -
Use of external
examiners
Checking on external examining - √ - -
Standardisation of
examinations
Quality assurance, examination committee,
feedback from lecturers and learners
- - - -
Examination
malpractices
Through lecturers, academic committee,
feedback from lecturers and learners
- - - -
Programme
Outputs
Dropout rates ED 46 forms, check final enrolment against
initial enrolment, teaching practice reports,
interim visits and reports, faculty reports,
regional centre reports
√ √ √ √
Progression rates Attendance registers, learner information
management data base, faculty and
departmental records, feedback from learners
√ √ √ √
11
Completion rates Learner information management data base,
annual departmental/college reports, ED 46
forms, interim visits and reports
√ √ √ √
Pass rates Examination pass lists – end of semester, term,
year and final year results
√ √ √ √
* SE – Secondary Education; TE – Teacher Education; HE – Higher Education; TVET – Technical and Vocational Education and Training.
The assessment revealed that currently MS in the SADC region are monitoring and evaluating seven components of their ODL programmes. These are basic information, programme process, programme performance, staff, resources, examinations management and programme outputs. Each of these components has specific sub-components that are monitored and evaluated. However, an analysis of the information provided by MS during the assessment reveals that with the exception of programme outputs, no specific indicators have been defined.
4.2 Gaps in Current ODL Indicators
The major gap that has been observed is the absence of specific indicators across all components that are currently being monitored and evaluated with the exception of programme outputs. This situation makes it difficult to standardize measurement across MS in the region. Furthermore, no all MS are monitoring and evaluating all the seven components presented in Table 4.1 above. There is therefore a need to first agree on the major components that are important to track across all MS. When these components have been agreed on, then a few important indicators have to be defined for each of the components.
,
4.3 Criteria for Selecting Core ODL Indicators
A number of factors were taken into consideration in choosing the ODL core indicators
that are contained in Table 4.2 below. There are global, continental and regional
education commitments that SADC MS are signatories to. Although progress in the
implementation of these commitments is being measured for the conventional education
mode, ODL programmes also contribute towards the progress in the commitments. The
non-measurement of these indicators has underestimated the progress that MS have
made individually and collectively as a region. Relevant indicators to measure progress
towards implementation of these commitments have been included in this ODL M&E
framework.
Second, there are indicators that are currently been tracked by most of the SADC MS.
Key indicators in this category have been included in the framework.
12
Also included are indicators that are not currently being tracked by MS but the majority
of MS perceive these to be critical for a better understanding of progress being made by
ODL.
The SADC ODL project has a logical framework that clearly spells out the results that
have to be achieved and the indicators that are expected to measure realization of
these results. These indicators were also considered in selecting the core regional ODL
indicators.
In selecting indicators, it is important to consider the ease with which they are
measured. It is not helpful to include indicators that the current M&E systems are not
ready to measure. This factor was taken into consideration in selecting indicators that
are included in this framework.
4.4 Core ODL Indicators
Table 4.2 below presents the SADC ODL core indicators that must be tracked and reported on by all SADC Member States. The indicator matrix has seven components, namely, basic information, programme process, resources, programme performance, ICT/Multimedia, Information, Education and Communication (IEC) and Gender Mainstreaming.,
Table 4.2: Core Indicator Matrix for the SADC ODL Programme
Component Sub-component
Indicators Definitions Data Source
Frequency
S/ED
TE
HE
TVET
1. Basic nformation/
Data
Gender
Gender Parity index
Percent of foreign learners enrolled at ODL institutions disaggregated by sex
Numerator No. of female learners enrolled Denominator No. of male learners enrolled Numerator Number of foreign learners enrolled during a given academic year Denominator Total number of learners
Institution records
Institution records
Annual
Annual
√ √
√ √
√ √
√ √
13
Component Sub-component
Indicators Definitions Data Source
Frequency
S/ED
TE
HE
TVET
enrolled at ODL institutions in a given academic year
2. Programme
Process
Curriculum Research
Curriculum implementation
Learner Support Services Module distribution
% of prescribed curricula which were informed by needs assessment % of prescribed curriculum subjected to formative evaluation
% of ODL institutions with Learner Support Units
% of learners with prescribed
modules
Numerator No. of prescribed curricula informed by a needs assessment Denominator Number curriculum research was conducted
Numerator
No. of prescribed curricula subjected to formative evaluation
Denominator No. of curricula
Numerator Number of ODL institutions with learner support Units Denominator Number of ODL institutions in the Country Numerator Number of learners with all prescribed modules Denominator
Institution records
Institution records
Institution records
Institution records
Annual
Annual
Annual
Annual
√ √ √ √
√ √ √ √
√ √ √ √
√ √ √ √
14
Component Sub-component
Indicators Definitions Data Source
Frequency
S/ED
TE
HE
TVET
Student attendance at learning centres
Student completion of assignments HIV and AIDS issues in the curriculum
Gender issues in the curriculum
% of learners who attended all prescribed sessions
% of learners who submitted all prescribed assignments
% of curricula addressing HIV and AIDS issues
% of curricula addressing gender issues
Total number of learners enrolled in a given year Numerator No. of learners who attended all prescribed face-to-face sessions Denominator All students enrolled during the year Numerator Number of learners who submitted all prescribed assignments during the year Denominator Number of learners enrolled during the year Numerator Number of curricula addressing HIV and AIDS issues Denominator Total number of curricula Numerator Number of curricula addressing gender issues Denominator Total number of curricula
Institution or centre records
Institution records
Institution records
Institution records
Annual
Annual
Annual
Annual
√ √ √ √
√ √ √ √
√ √ √ √
√ √ √ √
15
Component Sub-component
Indicators Definitions Data Source
Frequency
S/ED
TE
HE
TVET
Curriculum examination
Curriculum review
% of learners who wrote all prescribed examinations % of curricula subjected to summative evaluation
Numerator Number of learners who wrote all examinations Denominator Total number of learners enrolled during year Numerator No. of curricula subjected to summative evaluation Denominator Number of curricula ready for summative evaluation
Institution records
Institution records
Annual
Annual
√ √
√ √
√ √
√ √
3.0 Resources
Human
Financial
% of trained teaching staff
Staff Student ratios
% of education sector budget
Numerator No. of
teaching staff at an ODL institution
trained in ODL
Denominator Total number of teaching staff at ODL institutions
Numerator No. of learners Denominator No. of teaching staff at ODL institutions Numerator Amount of
Institution records
Institution records
Institution records
Annual
Annual
Annual
√ √ √
√ √ √
√ √ √
√ √ √
16
Component Sub-component
Indicators Definitions Data Source
Frequency
S/ED
TE
HE
TVET
Student placement
allocated to ODL % of students able to find placement
money allocated to ODL institutions Denominator Total education budget in a given year Numerator No. of students who are able to find placement Denominator Total number of students eligible for placement
Institution records
Annual
√
4.0 Programme Performanc
e
Progression
Completion
Dropouts
Progression rates
Completion rates
Dropout rates
Numerator No. of learners who progress to the next level
Denominator Total no. of learners at a given level
Numerator No. of learners who complete their studies to the next level
Denominator Total No. of learners at the level Numerator No. of learners who drop out of a programme
Denominator No. of learners enrolled in a
Institution records
Institution records
Institution records
Annual
Annual
Annual
√ √ √
√ √ √
√ √ √
√ √ √
17
Component Sub-component
Indicators Definitions Data Source
Frequency
S/ED
TE
HE
TVET
Passes
Failures
Further studies
Promotions
Pass rates
Failure rates
% of learners who complete a given level and proceed to the next higher level
Promotional rates
programme Numerator No. of learners who successfully complete their programme of study
Denominator Total number of learners enrolled in that programme at the beginning of the programme
Numerator No. of learners who fail to complete their studies
Denominator No. of learners who enrolled in a given programme at the beginning of the programme
Numerator No. of learners who successfully complete a given level and proceed to do further studies
Denominator No. of learners who complete a given level Numerator No. of ODL learners who
Institution records
Institution records
Institution records
Employment records
Annual
Annual
Annual 5
years
√ √ √ √ √ √
√ √ √ √ √ √
√ √ √ √ √ √
√ √ √ √ √ √
18
Component Sub-component
Indicators Definitions Data Source
Frequency
S/ED
TE
HE
TVET
Employment Creation
Percent of ODL TVET graduates running their own businesses
obtain promotion at their workplaces
Denominator No. of ODL learners who are employed Numerator Number of ODL TVET learners who graduated in the past three years who are running their own businesses Denominator Number of ODL TVET learners who graduated in the past three years
Tracer Surveys
& above
5
Years
√
5. ICT /multimedia
National ICT policy
ICT Infrastructur
e
Availability Computers
No. of MS with ICT Policy Percentage of ODL institutions with internet connectivity
Learner computer ratio
ICT course
Numerator No. of ODL institutions with internet connectivity Denominator Total no. of ODL institutions
Numerator No. of learners enrolled
Denominator No. of computers at ODL institutions
Numerator
MS ODL Reports
Institution records
Institution
Annual
Annual
Annual
√ √ √
√ √ √
√ √ √
√ √ √
19
Component Sub-component
Indicators Definitions Data Source
Frequency
S/ED
TE
HE
TVET
Availability of library
utilization
% of ODL institutions with library facilities
No. of ODL institutions offering ICT courses Denominator Total no. of ODL institutions Numerator No. of ODL institutions with library facilities Denominator Total no. of ODL institutions
records
Institution records
Annual
√
√
√
√
5. Data Management and Utilisation
Data generated by MS on the core indicators must be analysed in order to generate information that will guide Member States to improve the design and delivery of effective ODL programmes. The data will also assist the SADC Secretariat to produce trends on some of the key ODL education outcomes and at the same time identify bottlenecks in the implementation of ODL programmes in the region.
The utilisation of the data may be enhanced if Member States put in place mechanisms
to share national ODL reports with a wide group of ODL practitioners and stakeholders.
On the other hand, the SADC Secretariat will organize regional meetings to provide
feedback to Member States on the progress being made by the region in realizing key
ODL outcomes.
6. Implementation Mechanisms
In order for the M&E Framework to be effectively implemented, there are roles that
should be played by both SADC MS and the SADC Secretariat. These roles are
explained below.
6.1 Role of SADC Member States
20
In order for the Regional M&E Framework to be effectively implemented, MS are
expected to do the following:
Disseminate and advocate for the use of the M&E Framework;
Domesticate the M&E Framework so that it becomes part and parcel of the
National Education Management Information Systems (EMIS);
Facilitate national training on the M&E Framework;
Collect data on the core indicators and submit annual national ODL reports to the
SADC Secretariat;
Provide resources to support effective implementation of the M&E Framework;
and
Utilise data generated by the M&E Framework for planning purpose at the
national level
62. Role of SADC Secretariat
The SADC Secretariat is expected to do the following in order to support the
implementation of the Regional M&E Framework:
Facilitate training of Trainers on the M&E Framework at the regional level;
Support national level training on the M&E Framework and on monitoring and
evaluation in general;
Monitor implementation of the M&E Framework at the national level;
Prepare regional ODL annual reports on the basis of national ODL reports
submitted to the SADC Secretariat by Member States;
Validate the quality of data submitted by MS on the core indicators;
Disseminate the information on the regional progress in implementation of ODL
programmes to stakeholders; and
Periodically review the M&E Framework to accommodate any emerging issues
7. Reporting
All SADC Member States will track all the core indicators contained in this framework
and report on then annually. Each MS will prepare a national ODL report that will be
guided by a report format that will be developed by the SADC Secretariat and agreed on
21
by all MS. These national reports will be submitted to the SADC Secretariat on 30
March every year to allow the Secretariat to generate a regional ODL Report. The
SADC Secretariat will prepare a regional ODL report and share it with Member States
for comments and validation purposes on 30 April every year. Member States are
expected to share their comments with the SADC Secretariat by 30 May every year to
allow the SADC Secretariat to finalise the report by mid-June.
In order to make the reporting effective and facilitate Member States to provide quality
data, there is need for MS to put in place structures for data validation before national
ODL reports are submitted to the SADC Secretariat.