implemententing analytics part 1 - niall sclater

Implementing analytics:Learning analytics and business intelligenceNiall Sclater, Consultant

Session outline

» Overview of the Learning Analytics Service » Overview of the Business Intelligence and Analytics Labs

project» Presentation 1: Moriamo Aduyemi, head of corporate

information systems, Abertay University» Presentation 2: Niall Sclater, learning analytics consultant,

Jisc

Effective learning analytics challenge

Rationale» Universities and colleges wanted help to get started and

have access to a standard set of tools and technologies to monitor and intervene

Priorities identified» Code of Practice on legal and ethical issues» Develop a basic learning analytics service including an app

for students» Provide a network to share knowledge and experienceTimescale» 2015-2016 - Test and develop the tools and metrics» 2016-2017 - Transition to service (Freemium)» Sept 2017 - Launch. Measure impact on retention and

achievement

What do we mean by Learning Analytics?

» The application of big data techniques such as machine based learning and data mining to help learners and institutions meet their goals:

» For our project: › Improve retention (current project)› Improve attainment (current project)› Improve employability (future project)› Personalised learning (future project)

Toolkit and community

» Blog: http://analytics.jiscinvolve.org

» Reports› Code of practice for learning

analytics› The current state of play in UK

higher and further education› Learning analytics in Higher

Education: a review of UK and international practice

» Mailing: analytics@jiscmail.ac.uk » Network meetings

Learning analytics architecture

Current engagement

» Expressions of interest: 85» Engaged in activity: 35» Discovery to Sept 16: agreed (28), completed (18),

reported (17)» Learning analytics pre-implementation: (12)» Learning analytics implementation: (7)

Current engagement

» From Sept 2016» “Readiness Toolkit” with a diagnostic set of questions and

support materials leading to implementation» Start-up guidelines to get ready for learning implementation

Further details will be announced via analytics@jiscmail.ac.uk

HESA and Jisc Business Intelligence

Heidi Plus The new business intelligence service for UK Higher Education Replaces Heidi (which will be decommissioned in November

2016) Launched in November 2015 offering:

Improved data content and functionality Delivery of data sets through commercial data explorer tool New visualisations and dashboards New training programme and support materials

Available to HE institutions with a full HESA subscription Over 80% of current Heidi subscribers have started the Heidi Plus

application process (40% completed)

Heidi Lab overview

Take a thin vertical sliceDon’t try to waterfall each stepWant quick release of product and steer

Dev Team1

Dev Team 2

Dev Teams

Dev Teams

Dev Teams

Dev Teams

Analysis

Data orders

User stories

Dashboards

Process overview

Scrum team activity

Library Analytics labs

» Teams working on Library BI Stories at 0.2 FTE, total estimated effort 15 days from July - Oct 2016

» Both product owners and sector data eperts invited:› Product owner from the sector to steer which stories are

of interest› Sector experts to understand what data sources are

available and what is in the data› Jisc Contracted data transformation specialist (CETIS)› Jisc Agile scrum master and Tableau user

» Teams receive experience and guidance of Agile working» Option for Tableau desktop training to help with creating

visualisations» Apply at http://bit.ly/jisc_library_data_labs_applications» Queries to siobhan.burke@jisc.ac.uk or

myles.danson@jisc.ac.uk

Learning analytics

Retention

» 178,100 students aged 16-18 failed to finish post-secondary school qualifications they started in the 2012/13 academic year› costing £814 million a year - 12 per cent of all

government spending on post-16 education and skills (Centre for Economic and Social Inclusion)

» 8% of undergraduates drop out in their first year of study › This costs universities around £33,000 per student

» Students with 340 UCAS points or above were considerably less likely (4%) than those with fewer UCAS points (9%) to leave their courses without their award

Attainment

» 70% of students reporting a parent with HE qualifications achieved an upper degree, as against 64% of students reporting no parent with HE qualifications

» In all disciplines except Computer Science, Medicine and Dentistry, and Physical Science, students with a parent with an HE qualification were more likely to have achieved an upper degree

» Overall, 70% of White students and 52% of BME students achieved an upper degree

Marist college – Academic early alert system

Approach» Supported by Bill and Melinda Gates Foundation.

Investigated how use of Academic Early Alert systems impact on final course grades and content mastery

Outcome» Analysis showed a statistically significant positive

impact on final course grades» The most important predictor of future academic

success was found to be partial contributions to the final grade

» The predictive models developed at one institution can be transferred to very different institutions while retaining most of their predictive abilities

» Simply making them aware that they are at risk may suffice

Signals at Purdue University

Approach» Signals’ predictive algorithm is based on performance,

effort, prior academic history and student characteristics

Outcome» Problems are identified as early as the second week in

the semester» Students are given feedback through traffic lights – and

from messages tailored by their instructors» Students using Signals seek help earlier and more

frequently» One study showed 10% more As and Bs were awarded

for courses using Signals than for previous courses which did not use Signals

UK Open University – developing an analytics mind-set

Approach» Investing in strategic learning analytics programme to

enhance success of over 200,000 studentsOutcome» Availability of data – macro-level aggregation of all data

relating to student learning and experience» Analysis of that data – multiple dashboards and tools

under development, including student views» Interventions and processes that enhance success –

including input into learning and assessment design at module level

Predictive analytics at Nottingham Trent University

Approach» Most prominent UK: university-wide dashboard for

30,000 students, enhancing retention, community belonging, and attainment

Outcome» ‘Doubters’ less likely to complete targeted via tutor and

peer support» 27% of students changed learning behaviour because of

analytics feedback» Engagement in studies shown better predictor of final

grade then entry qualifications or demographics

25

Jisc’s effective learning analytics programme

ECAR Analytics Maturity Index for Higher Education

UK Learning Analytics Network

analytics@jiscmail.ac.uk

26

ECAR Analytics Maturity Index for Higher Education

Group Name QuestionMain type

Importance Responsibility

2 Consent Adverse impact of opting out on individual

If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress?

Ethical 1 Analytics Committee

7 Action Conflict with study goals

What should a student do if the suggestions are in conflict with their study goals?

Ethical 3 Student

8 Adverse impact

Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances?

Ethical 1 Educational researcher

86 issues in 9 groups

Available from Effective learning analytics blog: analytics.jiscinvolve.org

Group Name Question Main type Importance Responsibility2 Consent Adverse impact of

opting out on individual

If a student is allowed to opt out of data collection and analysis could this have a negative impact on their academic progress?

Ethical 1 Analytics Committee

7 Action Conflict with study goals

What should a student do if the suggestions are in conflict with their study goals?

Ethical 3 Student

8 Adverse impact

Oversimplification How can institutions avoid overly simplistic metrics and decision making which ignore personal circumstances?

Ethical 1 Educational researcher

86 issues

jisc.ac.uk/guides/code-of-practice-for-learning-analytics

jisc.ac.uk/guides/code-of-practice-for-learning-analytics

jisc.ac.uk/guides/code-of-practice-for-learning-analytics

Times Higher, 25 Feb. 2016

32

Jisc’s effective learning analytics programme

ECAR Analytics Maturity Index for Higher Education

Discovery Phase

34

ECAR Analytics Maturity Index for Higher Education

Architecture

Project partners

Learning Analytics architecture

Unified data definitions

The case for Learning Analytics

Service: dashboards

Visual tools to allow lecturers, module leaders, senior staff and support staff to view: » Student engagement» Cohort comparisons» etc…

Based on either commercial tools from Tribal (Student Insight) or open source tools from Unicon/Marist (OpenDashBoard)2/03/2016

The case for Learning Analytics

Service: Alert and intervention system

Tools to allow management of interactions with students once risk has been identified:

» Case management» Intervention management» Data fed back into model» etc…

Based on open source tools from Unicon/Marist (Student Success Plan)

2/03/2016

Service: Student app

Comparative – social – gameified – private by default – usable standalone - uncluttered

Profile Aim Activity Data sourcesRussell Group Retention of widening

participation + support for students to achieve 2.1 or better

Discovery + Tribal Insight + Learning Locker

Moodle + Student Records

Research led University

Retention, improve teaching, empowering students

Discovery + OpenSource Suite + Student App

Moodle + Attendance+ Student Records

Teaching led University with WP mission

Retention - requirement to make identifying students more efficient so they can focus on interventions

Tribal Insight + Learning Locker

Blackboard + Attendance + Student Records

Research led University

Student engagement Discovery + Student app + Learning Locker

Moodle + Student Records

Teaching lead Understanding of how Learning Analytics can be used

Discovery + Technical Integration

Moodle

jisc.ac.uk

05/02/2023 48

Niall SclaterConsultant, learning analyticsniall.sclater@open.ac.uk