From evaluation to reflection-on-action: Lessons learnt from the impact of a distance

education programme

NADEOSA Conference (24 – 25 June 2014) Ruth Aluko

Introduction• Quality assurance at various levels in HE, but

a gap evaluation dimension (Mizikaci, 2010)• Evaluation in HE (Hall & Hall, 2004) focus – aims of a program & to what extent tool for decision-making & assessing quality• Purpose to maintain / improve quality of products &

processes• Tool reflection

2

Background• ACE (EM) - Continuing professional

qualification• Paper-based DE program • Elongated impact study (2007 – 2012)Mixed-methods enquiry (surveys &

interviews)Surveys: 300 graduates (2004-2006 cohorts)

3

Background 128 principals Interviews (2 individual & 10 focus group)30 teachers20 principals Kirkpatrick’s & Baldwin & Ford’s Training

Evaluation ModelsEvidence of value added to individual lives &

workplaceSuggestions were proffered

4

Reflection & Reflective PracticeBy three methods we may learn wisdom:

first, by reflection, which is noblest; second, by imitation, which is easiest; and third, by

experience, which is the bitterest. (Confucius)

• Experience, meaning, learning & application

• Reflection-in & on-action• Trigger for reflection

5

Cyclic Process of ReflectionContent

Return

Feelings & Emotion

Re-evaluation

Possible explanation

s

Outcome

Intelligent action

6

Reflection Questions• What valuable lessons can be learnt from the

impact of the discontinued ACE: (EM) programme on the professional development of graduates and why?

• How could these lessons apply to other distance education programmes at the University of Pretoria?

• What possible value could these lessons add to the practices of other distance education providers?

7

6 Major Lessons1. Clear institutional policy on quality

assurancepersistent negative perceptions of DE

programmes (within & outside)Quality – a priority & an iterative processAwareness among stakeholdersPossible relationship between student

attrition rates & quality

8

Table 1: Non-completion rates of distance education students at UP (All programmes)

Year Students who discontinued

their studies

Percentage of total enrolment

2006 419 4%2007 650 4%2008 584 3%2009 762 4%2010 430 2%2011 650 3%2012 689 4%2013 336 2%Source: University of Pretoria (UDE) (2006–2013)

9

2.Practices guided by a policy on distance educationTraditional Education policy does not necessarily

apply to DEImportance of policy document

3. On-going monitoring of the DE students’ profileNecessity for quality data to make data-driven

decisions regarding programme improvement

10

Examples of data: age, technology, work & geographical profiles, and graduation & retention rates.

11

Table 2: Technology profile of students who enrolled for the first time between 2004 and 2013

Year 2004 2006 2007 2008 2009 2010 2012 2013

Number of students 3 187 5 087 5 643 8 011 6 102 5 675 3 354 2 599

Cell phone use 98% 99% 99% 99% 99% 99% 100% 100%

Internet use 0% 2% 1% 3% 8% 13% 25% 29%

Source: University of Pretoria (between 2004 & 2013)

4. Student support structuresall activities beyond the production and delivery

of course materialsMost students from TE backgroundExamples: contact sessions, tutorial letters,

assignments, SMS & an academic enquiry service

5. Programme designContinual reviewIn spite of IT, most countries in Africa are still

trapped in the first-generation mode of delivery

12

6. Research focused on DEDedicated research unitOperational & academic researchPurpose is to inform practiceunderstanding trends and issues in terms

of topics & methodsE.g. mixed-methods

13

Concluding Thoughts• Quality and quality management are

organisational obligations that lie vested in management’s commitment towards an understanding of quality (Aluko, Fraser, and Hendrikz 2008)

• improved performance of distance education students attested to by scholars if quality improves

• The necessity for continual evaluation of DE programs & reflection-on-action

14

Bibliography• Aluko, R. 2009. The impact of an Advanced Certificate in Education (ACE) programme on the professional

practice of graduates. The International Review of Research in Open and Distance Learning , IRRODL, 10 (4) www.irrodl.org

• Aluko, FR, WJ Fraser and J Hendrikz. 2008. Some interfaces in conventional and distance education programmes in a postmodern context. South African Journal of Higher Education, 22(3): 484–497.

• Aluko, FR, WJ Fraser and J Hendrikz. 2011. Transactional Distance Theory and Total Quality Management in contact and distance learning. Africa Education Review, 8(1): 115–132.

• Commonwealth of Learning. 2004. Planning and implementing open and distance learning systems: A handbook for decision makers. Vancouver: Commonwealth of Learning. Available at: www.col.org (accessed 7 April 2011).

• Hall I. and D. Hall. Evaluation and social research: introducing small-scale practice. United Kingdom: Palgrave Macmillan, 2004.

• Lee, Y, MP Driscoll and DW Nelson. 2004. The past, present, and future of research in distance education: Results of a content analysis. American Journal of Distance Education, 18(4): 225–241.

• Lockee, B., Moore, M., & Burton, J. (2002). Measuring success: Evaluation strategies for distance education. Educause Quarterly, 1, 20-26.

• Mizikaci, F. Total quality management in higher education: An evaluation model for practitioners. Germany: LAP Lambert Academic Publishing, 2010.

• Pulsipher S. 2009. Preparing quality teachers: Tracking a cohort of students from pre-admission to graduation. Available at: http://education.byu.edu/cites/documents/Student%20ProfileSusan%20Pulsipher.pdf (accessed 7 January 2013).

• Rubin, F. 1995. A basic guide to evaluation for development workers. Oxford: Oxfam Publications.• Wilson-Strydom, M. 2004. Programme evaluation and its role in quality assurance. Vancouver:

Commonwealth of Learning. Available at: http://dspace.col.org/handle/123456789/351

15

Thank You!ruth.aluko@up.ac.za