The OLDS MOOC Evaluation Plan and an Experiment in Recognising Learning usingBadges

Slides by Simon Cross and Rebecca GalleyInstitute of Educational Technology, The Open University

Presented by Simon Cross as part of:Cross, S. and Mor, Y. (2013) OLDS MOOC: design, evaluation and badges, eLearning Community Event: MOOCs at the Open University: past, present and future, 13 March 2013, OU, Milton Keynes

For more on the event visit: http://cloudworks.ac.uk/cloud/view/8043

1. OLDS MOOC Evaluation Plan(themes, structure, questions)

2. OLDS MOOC Badge Strategy(what are badges, what are they for, and how could we use them)

3. Use and user comments(examples of twitter posts that

comment on role of badges + Bonus slide on expectations)

Presentation overview

OLDSMOOCEvaluationStrategy

Questions onRegistration

Prior understandingand knowledge

Previous Experienceof MOOCsQuestionnaire

n=1400+n=35

MOOC ‘EvaluationPanel’ recruited before start for longitudinal study and to ensure weare likely to interviewsome people at endwho ‘dropped out’

n=15

September 2012

April 2013

Expectations andMotivationsQuestionnaire: oneweek prior to start

n=60+

Weekly manual recording of usestatistics from Cloudworks andBibsonomy

Weekly capture andarchive of all tweetsmade across thechannels

Recording of datarelating to badge approvals

Feedback form submissions

n=25 at present n=80+ and rising

MOOC ExperienceQuesitonnaire - opento all

Opens March

Telephone interviewswith Panel and othersthought to beinteresting cases

Opens March

MOOC STARTS

MOOC ENDS

MOOC Evaluation Themes

Motivations

Expectations

Quality / Quantity

Groups & Support

Badges / Reward

MOOC Makers

Expectations and Motivation in MOOCs A review of pre-course survey results and user posts will provide insight in to what people expected of the MOOC and why they were engaging with it.

What variety of expectation is there?

How does this compare to the reality of the MOOC experience?

Who and when could we use this type of MOOC structure/format?

How is content and structure informed by the aims and intentions of the designers?

Use of Badges and approach to reward

Nine badges have been developed in conjunction with a badging strategy. A method of creating and awarding badges added to Cloudworks.

What role did the badges have in motivating participants and preventing withdrawal?

What can we learn about using badges to support learners and promote learning?

1. What are badges?

2. What are they for?

3. How could we use them?

… September 2012

… 1. What is a badge?

My working definition:

“A badge is visual public symbol that communicates to others a particular quality, achievement or affiliation possessed by the owner. It is one ofa variety of ways that such recognitioncan take a material form.”

… 2. What are they for?

The Role of the badge for receiver and issuer…

As a motivatorAs an engager As a meaning makerAs a signifier of learning objectivesAs a valuerAs a symbol of identityAs a means of associationAs an empowerAs an entrencher

http://www.olds.ac.uk/blog/moocbadgingandthelearningarc

http://www.olds.ac.uk/blog/moocbadgingandthelearningarc

… 3. How can we use them?

Additional Cloudworks technical development undertaken:

•Badge unload functionality•Define approval criteria•Apply for badge•Approval (two processes)•Display on personal profile•Aligned to Mozilla badging

… 3. How can we use them?

Additional Cloudworks technical development

Endeavour and Progress

Achievement

Practice

http://www.olds.ac.uk/blog/olds-moocbadgingstrategy

OLDS MOOC Badges

http://www.olds.ac.uk/blog/moocprofiles

OLDS MOOC Participant Profile

From a sample of registrations

Study planning

[Am] interested in gaining badges, and working with colleagues in a local study group.

[My personal aims]: 1. completing one week of the MOOC and applying for a badge (maybe not getting the badge but at least applying for one week:)

I like the Badges, works as motivation for me. [Will] chose to get [the] Endeavour [ones] due [to] time constraints. Maybe Achievement.

Badges and Participation

122012201310

Week 1 Week 3

340

100

200

3. Week 3 Cloudscape

4. Sharing LD representations Cloud (Activity 6-7)

1. Introduce yourself cloudscape

2. Dream (Project) Bazaar cloudscape

1 2 3 4

FollowingFollowingContributing / DoingUnique viewsBadge

A bit of fun ... even a surprise

@[B]: Earned my Week 1Badge from #oldsmooc ! Unexpectedly pleased to have a reward for efforts! What fun and motivates …

Wooh [I] too have got a badge from #oldsmooc_w1 how #awsome Just applied for my #oldsmooc week 1 badge. Not that it matters but .... if I get it I will print it out (in colour) and pin it on my lapel

First #oldsmooc badge...what fun, feel like a Girl Guide again

Thursday motivation - got my 3 weeks #oldsmooc badge” @[T] Well done and snap! Delighted to get my 3 weeks badge too - likin' those badges more than expected!

Confidence, collecting, something to show

Guiding reflective learning

Hooray for my first @openbadges!

... and badges def. do work. I'm bagsing as many as I can to show that I'm learning loads… I love #openbadges. See my #oldsmooc badges [x]

Impressed #oldsmooc badge linked to evidence of reflective blogging #openbadges > me too, I got a badge :-)

@[R] I love the #oldsmooc badges http://t... - keep up the reporting on your learning, love it

Reason to ask for peer review

Facilitator: OER developer badge is crowdverified, you need to get your peers to approve your application. don't be shy to ask!

Have applied for OER developer badge, pls verify http://t...

Hi Folks Just applied 4 my #oldsmooc collaborator badge http://t.co/[x] Could you do the honours?

Supporting sense of community

Just applied for Resource Gatherer Badge...anyone else? thanks to @[D]'s reminder I have got another #oldsmooc badge http://t.co/[X]...

I got another #oldsmooc badge!! #oldsmooc_wk4 looks puzzling though …See how we go :)

Yay! #cloudworks #badges now link to "Who has been awarded this badge?”

… Just need to figure out my #mozilla backpack now :)

and checking progress

‘Own Path’ participants

I haven't qualified for a badge yet as I'm pursuing a somewhat singular path through #oldsmooc -- I'm OK with that

“Oh - and a final word: don't let badge envy ruin your day :)”

“I’m starting to really enjoy OLDSMOOC, I’ve acceptedthat I’m no longer on the approved route but wanderingabout reading, thinking about stuff and chatting withlike-minded people is great learning anyway. Having said that, I got my Week 1 badge today!”

Concluding remarks

• Promoting deeper and more expansive learning?

• Promoting richer learning experience?

• Alternatives to summative and formative assessment?

• Appeal to broader range of study motivations / practice?

• Help students better value their study?

• Help students better communicate their achievements?

Concluding remarks

Design considerations

• Overjustification• Reflective practice / Affirmation• Status / Reputation• Overlearning• Goal setting / Identity• Community endorsement • Whenever• Fun and enjoyment• Collecting …

Immersion in an online social environmentBrushing up on skills or knowledgeLearning new skills, knowledge, participation skillsInternational working with many people from many different countries Local working with colleaguesStructured format for thinking about and designing learningBeing challengedTaking time out to reflect on practiceNew networks Trying a MOOC / a (c)MOOC / new technologiesAchieving objectivesCompleting the course Completing the subject of the course - a learning projectExploring or gaining exposure to new resources / design tools / teaching tools or technologyExperiencing a MOOC and seeing how it worksEvaluating the potential of the MOOCDocumenting experience to publish, for own reflection or for studentsResearch case study or for research dataTeam work skills required / seeing how team working is made to workPeer discussion of issues around the subjects with peersPeer feedback on own learning projects and ideasCollaboration with others on developing a courseSharing and aggregating toolsComplementing formal study (post-graduate / graduate courses)Gain badgesGroup working with local colleaguesLearning how to exploit technology for teachingOverview of field Extending existing learnt knowledge in area previously studiedCutting edge ideas and research

What do you want to get out of it?

Just 8 of 39 mentioned completion

As a bonus, here is a summaryof just some of the reasons participants gave for wanting todo the MOOC. More to follow inthe full project evaluation

Please use an appropriate reference or acknowledgement when

citing this presentation.

For further information please contact:

simon.j.cross@open.ac.ukrebecca.galley@open.ac.ukyishay.mor@open.ac.uk

This presentation was prepared on the eve of the final weekof the OLDS MOOC. A full evaluation will take place betweenMarch and May 2013 so watch out for more detailedanalysis and review at:

OLDS MOOC Blog: http://www.olds.ac.uk/blog

And personal reflection at:

Simon’s Blog: http://latestendeavour.wordpress.com/