Making Assessment Count Consortium(e-Reflect)

Presenting today: Sian Lindsay and Kate Reader (City University London) and Gunter Saunders (University of Westminster)

The Team: Mark Clements (University of Westminster), Mark Gamble (University of Bedfordshire),Rae Karimjee (City University London), Mark Kerrigan and Simon Walker (University of Greenwich), Loretta Newman-Ford (University of Wales Institute, Cardiff), Maria Papaefthimiou (University of Reading)

Making Assessment Count Consortium

• an informal group that has a collective interest in benefiting from, and further developing and adapting, best practice in the use of technology to enhance the effectiveness and use of feedback;

• Realise the benefits of the JISC funded projects such as Making Assessment Count and other projects the partners have or are involved in.

The Original Making Assessment Count ProjectWhat did we do?

Sought to address issues around feedback by:–Developed a framework for action on feedback called the SOS model;

• To encourage students and staff to work through a process designed to promote action on feedback

• To link action on feedback to the personal tutorial system–Developed a ‘small piece of technology (software we call e-Reflect)’ to:

• Help the student strategically reflect on their feedback• Connect the student’s feedback and their reflections to

the personal tutor

SUMMARY OVERVIEW DIAGRAM OF MAKING ASSESSMENT COUNT

Encouraging students to reflect on their feedback

– e-reflect questionnaire encourages students to reflect on key aspects of their performance

– In response to selected answers, the tool automatically provides additional feedback (student generated) and prompts to stimulate deeper reflection

– Questionnaires can be authored and aligned for specific types of coursework

Benefits to staff, benefits to students, benefits to the institution

– Staff • personal tutors provided with more timely information on

their tutees overall performance;• Clear framework for provision of feedback.

– Student• Additional layer of feedback facilitating strategic reflection;• Strategy for using and acting on feedback.

– Institution• Raising of the profile of feedback (what it’s for and how it

should be exploited within a broader assessment strategy);• Support for shifting of emphasis of personal tutorial to

academic performance.

Making Assessment Count Consortium Progress

• UWIC – piloted with up to 90 Sports Medicine undergraduates

• Reading – planned pilot with over 100 final year project students

• Westminster – used now to support feedback on written exams

• City – planned pilot incorporating use of Moodle with Politics undergraduates

Making Assessment Count Consortium UWIC

• Students in a recent focus group very positive• Students said the process had made them

think about: – their own strengths and weaknesses– how they approached the assessment task– areas for future development – usefulness of tutor feedback

Making Assessment Count Consortium Westminster

• Piloting use of the MAC process with exams• 2 stage process

– Questionnaire immediately after the exam, students predict their mark and reflect on why they think this

– Subsequently given an opportunity to see and annotated copy of marked script

– Then complete a second questionnaire and add to the content of their learning journal

Kate Reader Dr Sian Lindsay

Senior Educational TechnologistSchools of Arts and Social SciencesCity University London

Lecturer in Learning DevelopmentLearning Development CentreCity University London

Encouraging our students to reflect and learn from their feedback

May improve student retention To engage more students with the personal

tutorial system at City Encouraging students to take a feedforward

approach to work

We took an ‘integrative’ evaluation approach (Cook, 2002)whereby we looked at how eReflect could be best integrated with

otherresources available to our users. In our case, the resource was our

newvirtual learning environment Moodle.

Students – how easy is eReflect to use? How enjoyable is it to use? Does eReflect feel personalised to me and meet my individual needs? Will eReflect help me to perform better in my studies?

Personal Tutors (teaching staff) – how easy is eReflect to use? How much additional work will I have to do over what I am doing already with my personal tutorials to use it effectively? Does eReflect look professional and does it integrate with the course content available on Moodle?

Support Staff – what are the hardware implications of eReflect? Who hosts it? What are the staff development and training implications of eReflect?

Developers – can eReflect be customised? Is it open-source and can we adapt it to better suit our needs? How do we report problems with bugs in the code?

Managers – how does eReflect fit within the institutional strategy? Can it enhance the status and attractiveness of the institution? What are the costs of running eReflect?

Strenghts Weaknesses Opportunities Threats

Easy interface- user friendly

New system – training needs?

Open Source – possible future integration

Reliant on Westminster IT infrastructure for support

3 years of experience and results at Westminster

Doesn’t utilise single sign on

No IT central support at City – not scalable for future?

No integration = tutors don’t get holistic view of students work

Difficulties engaging staff and student in new tech so soon after moodle launch?

Inflexible = staff can not amend to fit with current feedback/tutorial processes

Plain text editor only, not suitable for math/formula based feedback

No clear linkage of diagnostic questionnaires to broader module content – student error easy

Strengths Weaknesses

Opportunities Threats

Successfully piloted for 3 years at Westminster

Dependent on staff and students being comfortable with an electronic interface

No resistance to moodle: Staff are highly engaged with the moodle project, are actively using it to embed T&L activities, all staff had recent training.

Staff and students would need to realise the benefits in order to actively engage in the model.

The Model could integrate well with our current tutorial and feedback processes.

100% online electronic submission already in place, already implemented models for online feedback via moodle.

The model has not been piloted with different disciplines, no evidence this will work with Social Sciences.

Staff are currently engaged with a blended learning model, due to the moodle roll out.

If we use moodle, all students automatically receive an induction and support/help for students is already in place.

Students are already familiar with the Quiz and assignment submission features of moodle.

The model could improve student retention, and enhance our current feedback and tutorial models.

Small scale pilot within the Politics Department in September 2011.

Staff in the politics department have amended the SOS model to fit with their current personal tutoring system.

If successful possible roll out of the model in other departments from September 2012.

Conclusions and Future Work

• The MAC process can benefit students• Brings them into greater/better ‘contact’ with their

feedback and tutors• Variations of the MAC process can emerge to suit

departmental/institutional context• MAC process not limited to being used with marked

coursework• Generally institutions would prefer to see the MAC

process happen within their VLE

Acknowledgements and Thanks

To all the staff and students who participated across all institutions

To JISC for supporting the work

To you all for listening.

Making Assessment Count