Tracking and Assessing Vocational Qualifications

John GordonOpus Learning

21st November 2013

Opus Learning Ltd, Edinburgh

• We develop online open courses to SQA HND standards

– Direct to students who are enrolled with Opus– A white label service for students enrolled with

other providers.

• We provide the learning environment, and deliver the learning content and assessments.

History of Opus

• A spinout from CAPDM.com

• Developing and delivering Masters programmes for over 17 years

History of Opus

• We deliver highly interactive courses, delivered via an online platform such as Moodle.

• With tools to manage and monitor learning with or without tutor support.

Structure of our programmes

• No longer standalone University Masters Modules

• Now we build an integrated framework of Units for an SQA HND

• Within a framework of commercial discipline.

Today

In this afternoon’s talk, I hope to indicate how we have:– Benefited by using technology to support and

manage vocational learning;– Managed downwards the risks and costs of using

technology.

The problems

Online Learning does not change the problems of assessing learning– Tracking learning– Authentication– Identifying problems in learning– Recognising need for intervention

How do students interact when learning

Student

Environment

Content Tutor

Student/Student

Content/ContentTutor/Tutor

Environment/Environment

Student/Tutor

Tutor/Environment

Student/Environment

Student/Content

Content/Environment

Tutor/Content

Based on diagram inAnderson 2003

Interactions and MonitoringType of Interaction Tracking FunctionHuman to human Within Forums

Via a ticket system for formal student to tutor interactions

Student to Learning Platform Via built in tracking tools on Platform

Student to contentStudent and formative assessmentsStudents and personal reflections

Our in-built tracking and monitoring tool

Interacting with Content Objects

Learner interacting with content, leaving traces

Student interacts with content

Makes notes, responds to quiz, send messages, etc

Capture notesStore in MLE

Print off record of learning The Digital Workbook

The Digital Workbook is

• A portfolio embedded in the content

• A dashboard for learning • Rich seam of data and

behaviours for mining• Ongoing formative

assessment, and potential summative assessment

• A pdf for the student

Assessment In theSQA environment

All units are stand alone, they can be delivered in a very flexible manner, but there are very clear pointers to how assessment should be carried out.

It is no longer appropriate to base the assessment of competence on a final 3 hour written examination.

Assessment and Verification what happens

Candidate Requests

Assessment Instrument from VLE

Assessment sent to

Candidate – time limited

PDF

Candidate responds to Assessment

Assessor reviews

Assessment Attempt Makes

comments

Assessment pdf

Candidate Attempt

Satisfactoryyes

No

Assessor sends comments to

candidate

Assessor comments

Candidate Resubmits

Resubmit?

yes

Candidate attempt

NoSign assessment record detailing re-assessment

Sign assessment record detailing

Satisfactory

Candidate Assessment

Record

Send assessment record to

candidate – complete record in VLE/SQA Connect

Verifier consultation

Verifier consultation

Case study HND Business Unit by Unit Assessment

In a typical instance of the HND Business each student has to:– Sit 6 formal examinations– Deliver 6 documented portfolio exercises , and – 22 other assessments - reports with a total word

count of 34,500.

A university Dip HE might only have 12 modules with 12 final examinations or courseworks

Managing Assessment LoadIntegration of Assessment

In classroom, all students work in synchpossibilities of integrating assessmentchanging assessments due to the dynamics of

teaching.The assessor and IV can• consider integration of assessment • develop a more holistic view of assessment.

Assessment Integration ScenarioFrom the classroom

Assessment Units Nature of integration Additional comments

Simulated exercise

Communication

Business Accounting

IT Applications Software 1 IT in Business: Spreadsheets

A meeting could be held to discuss financial performance of a business. The production of an accounting statement using, for example, spreadsheets could form the basis for an agenda item.

The emphasis is upon the development of skills that are applicable to a business environment. As a result, it is important to provide candidates with the opportunity to generate evidence through simulated business activity where possible. For example, candidate may participate in a business meeting, they may produce a profit and loss account for a business, they may produce business documentation to support business

One instrument providing a framework for the assessment of 4 units

Open Distance Learning

The Tensions in Technology Based Open and Distance Learning

• Who is actually at the end of the screen?• What is the level of Granularity for assessment?

– We want freedom for student to learn what they want when they want

– We want to integrate assessment across a group of units

• What is the cohort size?– We want individualised learning– We want efficiency in the use of tutor support

Who is at the end of the screen?

Opus Policy is:

Digital Authentication

+Physical Authentication

Digital Authentication

The Physical Presence Requirements of a Programme

In the HND there are various points where physical presence is required by candidates.

Closed book instrument of assessmentProctored examination, eg graded unitPresentation by candidate, eg in a unit

using powerpointPresence at meeting, eg unit on

communications

The Opus Authentication TableAction Point Action Problem with ID Action on problem

with idSign on of candidate

Assign secure login to candidate to VLE, assign to tutor

Username and password wrong

Re-issue username and password

Physical presence point

Invigilator/proctor checks photo-id

Photo-id missing Do not proceed with assessment

Photo-id not accepted

Report to Head of Centre, who will decide whether to proceed with assessment

Digital Check via workbook

Tutor monitors Candidate progress via Digital workbook

Lack of progress Discuss with candidate

Change of personality, or suspicion of personating

Report to Head of Centre.

Digital Check via VLE

Tutor monitors Candidate activity through activity logs

Lack of activity Discuss with candidate

Managing Learning with Technology

• At Opus we track student behaviour. • We use 2 levels of tracking

– Moodle Based (the VLE) and – Content Based.

Moodle Based Tracking

• A good example:

• https://workspace.nottingham.ac.uk/download/attachments/170312943/Monitoring+students+-+NOOC+PDF.pdf?version=1&modificationDate=1372952991000

• Nottingham University experiment tracking 800+ students

Nottingham U. Sample Screen

Tracking through the learning content

• Opus content is massively connected to the students workbook

• Every reflection, exercise, quiz, short response and so on are logged within the learning content and in the workbook

• The tutor can view, track and compare workbooks

DWB Examples – Source Page

DWB Dashboard –Managing groups of students

Some conclusions from tracking

• Tracking appears to:– Provide the potential of personalised learning

within mechanical presentation of study– Assist formative assessment in the content driving

tracking and the experience of learning progress– Provide feedback on Tutor effectiveness and the

effectiveness of the pedagogy underpinning the content

The Commercial Imperative

We wish to – drive down cost of assessment– Drive up the quality and efficiency of assessment

within the context of a well resourced learner.

We have to manage costs, and use of human resource

Assessor/Tutor Activity Model in Distance Learning

1. Monitor VLE activity, identify any candidate with lack of progress and contact course leader for action.

2. Monitor Digital Workbook (if used) and identify any outliers in progress, contact course leader with details.

3. Respond to direct tickets from candidate, making use of FAQs for responses, if new response is required, copy response to course leader.

4. Responding to assessments; remediation advice

Managing Resource

• We can time and cost these elements• Times and costs vary with cohort size

Timing Model for HND Business

Assuming a cohort size of 20-30 students, averaged over the HND

Assumptions minutes Assessor time for single credit assessment

22

Assessor time for double credit assessment

26

Verifier Overhead 7 Exam Marking Time 45 Verifier minutes

per script20

Graded Unit Marking time

90 Verifier minutes per project

40

Timing Model vs Cohort size

HND Business Timings Assumptions

Students per assessor

1 5 10 15 20

Average timing per unit for HND Business - minutes

93 85 77 70 64

Learning Centre Cost Model

• Based on the previous models– Time per tutor/unit/student can be estimated– Payment to Tutor/assessor is known– Viability of the Distance Course can be calculated.

• We need to know these numbers and to watch them

Potential Cost Model

Possible costs/charges per studentawarding body charges = £250;tutoring = £1000; tech + online content licence = £300; agent fees = £250Centre overheads per student = £500

Total cost per student = £2300

Potential Centre Surplus

• In UK Charge to student £3500-£4500• In India Charge to student £1500-£2500

We cannot serve one of largest markets in the world, with this model.

We have to be able to manage costs.

Learning from MOOCS

• Massive size of cohort• Highly programmed week by week• Clearly signposted weekly targets• Using the student body as the main tutor

toolset• Building a community of activists and lurkers• Managing and controlling costs• Generate revenue by conversions

Learning from MOOCS

• Do not do assessments – Until students join full programme

• Keep the granularity of the course fairly small – not full degrees

• Set up an honour system – make the students police themselves.

Applying all these lessons

• Manage the cohort size• Make sure investment reflects assessment and

tutor support provided – the added value of a non-free course

• Integrate assessments across units; manage the assessment overload

• Make assessments social, encourage group help. Track this activity!

Opus StrategySmall Private Open Courses

• Organise cohorts in groups of 10s, not groups of 10k’s or 100k’s

• Localise tutors and assessors• Build communities including assessors• Track activity and behaviour at keyboard level and at

cognitive level• Record activity – perhaps award a badge for partially

completed activity• Guide the learners towards completion, intervene as

necessary

Creating the Social Learner

• Our DWB is part of a social learner strategy– Rich interaction spaces– Personal interaction with the content (DWB)– Interaction on specific issues (traversing the web)– Social learner (Forums, Facebook, etc.)

• Manage forum and virtual space activity

The Future

• The DWB is sharable, including – with the Tutor – with other students, groups

• In the future it will– support many content types– integrate with other portfolio components

• The DWB will be our main assessment toolset• The DWB will be our main behaviour manager

Conclusion

• Do not just set exams and give badges– Track and Assess– Intervene and support– Audit and learn lessons

• Build a community for the whole qualification• Record learning and make available for future

generations• Continually improve learning content.

The End

John GordonOpus Learning

jg@opuslearning.com

Thank You