MOOCs and Quality Issues - QAQE Steering Group

Quality Assurance and Quality Enhancement in E-learning Special Interest Group (QE-SIG)

Helen Barefoot University of Hertfordshire

Jon RosewellThe Open University

Session Overview

•Definition of MOOCs

•Online learning

•The Quality Code

MOOCs and Quality Issues

3

So what is a MOOC?•Massive•Open•Online•Course

MOOC background information•Term coined in 2008•Connectivism and Connective Knowledge▫25 tuition paying students▫2300 other students from the general public taking

course online and free of charge

Providers

Context

What are the key challenges associated with assuring the quality of MOOCs?

MOOCs and Quality Issues

8

Why bother with quality?•Students – know what they are committing to•Employers – recognition of content and skills •Authors – personal reputation, 'glow' of success • Institutions – brand reputation •Funders – philanthropic, venture caps, governments

•Quality agencies – on behalf of above

MOOCs and Quality Issues

9

TensionsDelivery: face-to-face . distanceAccreditation: credit . nonePrice: cost . freeEntry: selective . openScale: personal . massiveSupport: intensive . not supportedPedagogy: constructivism . transmissionTeacher: star . anonymous

Massive• Issue is not large size, but scale independence

•Aspects:▫Financial▫Technical▫Pedagogical

Learning design at scale• Individual learning•Small group collaboration• Large communities

Open•Different meanings:▫Open = free = gratis▫Open = free = libre▫Open entry▫Open = transparency

Course•What distinguishes MOOC from OER or learning

object?▫Size?▫Goal, learning outcomes?▫Measure of completion / achievement / attainment?▫Structure

As narrative (xMOOC)? Negotiated experience (cMOOC)?

▫Time?

Quality and learners“What are MOOCs actually aiming at? “Can the quality of MOOCs be assessed in the same way as any defined university course with traditional degree awarding processes? “Or do we have to take into account a different type of objective with MOOC learners? Are the learners mostly interested in only small sequences of learning, tailored to their own individual purpose, and then sign off and move to other MOOCs because their own learning objective was fulfilled?”Ulf-Daniel Ehlers, Ebba Ossiannilsson, Alastair Creelmanhttp://mooc.efquel.org/

Quality Assurance Agency “Our job is to safeguard quality and standards in UK universities and colleges, so that students have the best possible learning experience” Quality Code for Higher Education sets out the expectations all providers of UK higher education are required to meet. It gives all higher education providers a shared starting point for setting, describing and assuring the academic standards of their higher education awards and programmes and the quality of the learning opportunities they provide.

QAA event• Quality Code“Factors which apply to all learning opportunities regardless of location, mode of delivery, academic subject; MOOCs are no exception to that”

Anthony McClaran, QAA event (July, 2013)

The Quality Code

The Quality CodePart A: Setting and maintaining threshold academic standardsPart B: Assuring and enhancing academic qualityPart C: Information about higher education provision

The Quality CodePart A: Setting and maintaining threshold academic standards

Part B: Assuring and enhancing academic quality

• A1: The national level• A2: The subject level and qualification

level• A3: The programme level• A4: Approval and review• A5: Externality• A6: Assessment of achievement of

learning outcomes

• B1: Programme design and approval• B2: Admissions• B3: Learning and Teaching• B4: Enabling student development and

achievement• B5: Student engagement• B6: Assessment of students and the

recognition of prior learning• B7: External examining• B8: Programme monitoring and review• B9: Academic appeals and student

complaints• B10: Managing HE provision with others• B11: Research degrees

Part C: Information about higher education provision

Considering MOOCs through the lens of the Quality Code

B1: Programme design and approval• Procedures for design and approval for

MOOC courses?• Are external partners involved in the

design and approval?• Assessing the effectiveness of the

course and processes for monitoring and reviewing the course?

Course approval

B3; Learning and Teaching• Constructive alignment• Levelness• Teaching methods▫ Videos▫ Embedded questions ▫ Tutorials▫ Discussion forums▫ Wikis▫ Assessments

Considering MOOCs through the lens of the Quality Code

•B5; Student engagement• “improving the motivation of students to engage in

learning and to learn independently▫data

• the participation of students in quality enhancement and quality assurance processes, resulting in the improvement of their educational experience”.

Considering MOOCs through the lens of the Quality Code

Source: edX – Presented by Lori Breslow – Enhancement Themes conference, June 2013

“the participation of students in quality enhancement and quality assurance processes, resulting in the improvement of their educational experience”

• B6; Assessment of students and the recognition of prior learning

Considering MOOCs through the lens of the Quality Code

Moz

illa

badg

e in

fras

truc

ture

http

://o

penb

adge

s.or

g

http://www.dmlcompetition.net/

A badge is a validated indicator of accomplishment, skill, quality or

interest that can be earned in any environment

http://scouts.org.uk/

Back to basics…•Are students learning?▫Self assessment▫Quizzes ▫Peer assessment▫Analytics▫Survey evaluation

Monitoring and evaluation

Completion rates•Reported completion may be very low (1-10%)•Does that matter?▫With very large starting numbers, there are still many

learners completing▫Maybe learners achieve personal goals even if they

don’t complete▫Can MOOCs encourage access to HE if >90% have an

experience which is a ‘failure’?

ProvenanceReputation

Brand

ProvenanceReputation

Brand

creation use

user recommendationpeer review

MOOCMOOC

Quality points

checking

Quality Dimensions

ContentAccuracyCurrencyRelevance

Pedagogic EffectivenessLearning objectivesPrerequisitesLearning designLearning stylesAssessment

Reusability & opennessFormat & interoperabilityLocalisationDiscoverability: metadataDigital preservationAccessibility

Ease of useClarityVisual attractiveness, engagingClear navigationFunctional!

Benchmarks / indicators(NB: an unashamed plug!)•Conventional HE benchmarks not be appropriate •E-learning benchmarks a better starting point▫E-xcellence NEXT

http://e-xcellencelabel.eadtu.eu/

•MOOC benchmarks better!▫OpenupEd / E-xcellence MOOC (to come…)

http://openuped.eu/

OpenupEd www.openuped.eu • ‘Reflecting European values such as equity, equality

and diversity’•Courses should show features:▫Openness to learners▫Digital openness▫Learner-centred approach▫Media supported interaction▫Recognition options▫Quality focus▫Spectrum of diversity

UK’s first credit rated MOOC• Level 4• 20 credit points• Taught as MOOC for

undergraduate students and as open access

• 1000 students• 200 students going for credit• Assessed by a

• 10-minute podcast presentation (40%)

• 1500-word critical blog (60%)

And the latest…• University of California, Irvine, • “Tackle serious scientific issues,

related to events in the show”▫ Physics- "science behind decay“▫ Public health department- "What

can we learn from past epidemics?“

▫ Maths - "post-calculus maths can be used to model population and epidemic dynamics"

• TV series - audience of 10 million • Student numbers for online

course expected to be in hundreds of thousands