creating a culture - neon · creating a culture. creating buy-in/raising awareness •unpicking...
TRANSCRIPT
Creating and Embedding an Evaluation Culture in WP Project Work
Lydia Redican – Widening Participation Project Officer University of Liverpool [email protected]
Creating a culture
Creating buy-in/raising awareness
• Unpicking previous evaluation work that had been done
• Understanding who is a resource or a source of support
• Guidance from OfS should now give a focus with the new self assessment toolkit
• Number of new Evaluation Officer posts has grown and has created
Initial research
• Use the OfS Guidance, Standards of Evidence and Standards of Evaluation (Crawford et al 2017).
• Other sources of material
• Has anything else been written about your programmes, for example Programme Reviews, Focus Groups, Strategic Plans etc.
Resources and capacity
• Once the framework is set it impacts less on capacity
• Divide up areas of focus between teams
• Time for Evaluation is set in PDR
Embedding Evaluation
Logic Modelling
The ‘Liverpool Model’: methodology
Activity mapping
Practitioner intuition
Professional perspectives
Milestone data
Creating evidenced attribution
Survey, focus groups, etc.
Insight; moving to strategic inquiry
Dialogue, critical reflection, reporting, evaluation refinement, hypothesis, plausible scenarios
Moving to second phase and ‘2nd tier rigour’Case studies, control-based comparison studies, refined focus, extended inquiry etc.
Creating tentative attribution
First phase and ‘1st tier rigour’
Decide what is your core and enhanced evaluation• We decided what was our core evaluation
that could be measured consistently over an extended period of time
• Core• UG Surveys
• First Year Success
• Set framework meetings
• Enhanced: • Focus groups, academic interviews, practitioner
insight, student surveys.
Regular meetings with key stakeholders to develop and discuss the focus
• Who are your key stakeholders? • Admissions, Strategic Planning, Academics
with education/evaluation research experience, Student Recruitment, WP Leads in faculties, Student support/Careers and Employability, your students, parents and teachers.
• What is your focus? • Reviewing your core evaluation and
deciding how to tackle your enhanced
Map out your evaluation framework
• Identify your stakeholders and set your meetings and activities, have it planned at least a year in advance.
• Set your mile stones for evaluation activity, surveys, focus groups, data analysis.
• Where does the evaluation go?
• What do you do with the evaluation? Your programmedesign!
Liverpool Scholars Evaluation Framework
• Core Evaluation • Focus Groups
• Academic Interviews
• End of Programme Survey
• Undergraduate Survey for qualitive Impact
• Undergraduate data analysis for quantativeimpact
• Monitoring
• Enhanced – events, academic skills, experience, UCAS, BAME, pastoral support.
Focus Groups – Liverpool Scholars Example
• Every year we identify an area of focus to discuss with our students
• How we delivered the focus groups
• Our Expected Assumptions from the questions
• The Emerging Assumptions from the feedback
• Paper discussing both sets of assumptions and finish with an Action Plan
