Sarah GoodierROER4D Evaluation Advisor

UCT Teaching & Learning Conference30 March 2016

User focused evaluation: Feedback on research capacity building webinars

3/30/20161

OutlineThe project being evaluated: ROER4DThe evaluation work: what & howThe key evaluation questionFindingsRecommendations & next stepsLessons learnt

THE ROER4D PROJECT

in the Global South

In what ways, and under what circumstances can the adoption of OER address the increasing demand for accessible, relevant, high-quality and affordable education and what is its impact in the Global South?

Research on Open Educational Resources for Development

http://roer4d.org/

ROER4D has 86 researchers across 18 sub-projects in 26 countries across 16 time zones

EVALUATING ROER4D

1. Build an empirical knowledge base on the use and impact of OER in education2. Develop the capacity of OER researchers3. Build a network of OER scholars4. Communicate research to inform education policy and practice5. Curate output as open content

ROER4D Key Evaluation Areas

EVA

LUAT

ION

Using the utilization focused evaluation (UFE) framework

What is utilization focused evaluation (UFE)

UFE is centered around intended use by intended users

- What do the users (key stakeholders) want to know?

- How will answers to the evaluation questions potentially help to improve the project?

UFE in 12 steps:Steps are iterative,

not linear

Utilization Focused Evaluation Framework (Adapted from Ramirez & Brodhead, 2013)

1. Build an empirical knowledge base on the use and impact of OER in education2. Develop the capacity of OER researchers3. Build a network of OER scholars4. Communicate research to inform education policy and practice5. Curate output as open content

ROER4D Key Evaluation Areas

Webinars

Webinar evaluation timeline

2014 series (Feb – Oct) 2015 (Feb – July)

EvaluationData presented today

This 12-session series focused on the harmonisation of survey questions that were to

be used across the different ROER4D sub-projects

Iterative evaluation

work during this series

of webinars

Key Evaluation QuestionTo what extent have ROER4D Network

Hub webinars been a mechanism to potentially grow the research capacity of researchers working on the ROER4D project?

[Has the process worked & how can we improve it?]

To answer the question, focused on:◦Recorded attendance (taken at each

webinars)◦Survey results of the sub-project researchers

regarding the webinars (sent after the completion of the webinar series)

FINDINGS

2014 Session Attendance

• Overall, the trend was a decrease in attendance over the course of the sessions

2014 Session Attendance

• For all but one of the last five sessions, the network hub team made up half or more of the attendees

2014 Session Attendance

• Approximately half of the ROER4D network didn’t attend any of the sessions;

• Of those who did attend at least one session, most attended only up to six sessions

Research Capacity and Question Harmonization Survey Findings

Key findings:

• Those who did attend/view the session found them useful• Scheduling issues were a factor for those who didn’t attend any

of the live sessions. • The fact that these sessions were voluntary was flagged as a less

valuable point about the question harmonization process. This was seen to potentially create gaps in terms of research capacity across the project.

• Having recordings and the research questions and concept definitions available was useful.

In summary, this first webinar process has experienced varying degrees of success:◦Useful for those who attended/viewed the

sessions◦Several barriers (e.g. timezones) flagged by

the researchers & engagement in the webinars was sporadic across the sub-projects

RECOMMENDATIONS & NEXT STEPS

Evaluation recommendations on webinars

– Encourage each subproject to have at least 1 researching member attend the live webinar

– Keep recording webinars and point the subprojects to the recordings after the session as a reminder

– Have a forum for questions/discussion (before and) after the webinar so that those who don’t attend live can also benefit from shared discussion

– Have session repeated (2x) to incorporate different time zones (if there is capacity)

*Update: several of the recommendations were picked up in the 2015 series (e.g. having repeat webinars scheduled at different times)

Constant dialogue with the ROER4D team and users as the evaluation work continues

Next steps

Some lessons from evaluating ROER4DFind a framework for your evaluation (e.g. UFE):

valuable guiding structure

For both internal evaluation and external evaluations, never underestimate the importance of iterative engagement :

more engagement = better understanding of project & what matters to users

This work is licensed under a Creative Commons Attribution 4.0 International License.

Website: http://roer4d.org

Contact author: Sarah Goodier sarah.goodier@uct.ac.za @SarahGoodier

Excluding images, screenshots and logos and/or unless otherwise indicated on content

Thank you!

Further reading:About ROER4D:• Hodgkinson-Williams, C. (2013). Research on Open Educational

Resources for Development in Post-secondary Education in the Global South (ROER4D) - Scoping Document. Available online: http://hdl.handle.net/11427/8430 [Last accessed 10 February 2015].

• Hodgkinson-Williams, C. and Cartmill, T. (2014). Research on Open Educational Resources for Development in the Global South: 1st Technical Report 23 June 2013 to 27 August 2014. Available online: http://hdl.handle.net/11427/9695 [Last accessed 11 February 2015].

About evaluation:• Patton, M. Q. (2008). Utilization-focused evaluation. California: Sage

Publications Inc.• Ramirez, R. and Brodhead, D. (2013). Utilization Focused Evaluation: A

Primer for Evaluators. Penang: Southbound.• Scriven, M. (1991). Evaluation Thesaurus. California: Sage Publications

Inc.