scaling up learning analytics

Scaling Up Learning Analytics

Rebecca Ferguson, The Open University, UK

The Open University (OU)

• The Open University: largest in UK• Distance university• Making use of big data for 45 years• Informal learning: iTunes, YouTube…• MOOCs on FutureLearn,

OpenLearn and elsewhere• Learning analytics research and events• LACE project

2

open.ac.uk

http://www.laceproject.eu

Learning analytics

3

solaresearch.org

…the measurement, collection, analysis and

reporting of data about learners and their contexts,

for purposes of understanding and optimizing

learning and the environments in which it occurs.

Educators use analytics to…

• Monitor the learning process• Explore student data• Identify problems• Discover patterns• Find early indicators for success• Find early indicators for poor marks or drop-out• Assess usefulness of learning materials• Increase awareness, reflect and self reflect• Increase understanding of learning environments• Intervene, supervise, advise and assist• Improve teaching, resources and the environment

4

Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U. (2013). Supporting Action Research with Learning Analytics. Paper presented at LAK13.

Learners use analytics to…

• Monitor their own activities and interactions• Monitor the learning process• Compare their activity with that of others• Increase awareness, reflect and self reflect• Improve discussion participation• Improve learning behaviour• Improve performance• Become better learners• Learn!

5

Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U. (2013). Supporting Action Research with Learning Analytics. Paper presented at LAK13.

Analytics at scale: UK schools

6

• Aligned with clear aims

• Huge and sustained effort

• Agreed proxies for learning

• Clear and standardised visualisation

• Driving behaviour at every level BUT

• Stressed, unhappy learners• Analytics with little value for learners or educators• Omission of key areas, such as collaboration

Analytics at scale: Course SignalsDeveloped at Purdue University, USA

7

Arnold, K. E., & Pistilli, M. (2012). Course Signals at Purdue: Using Learning Analytics To Increase Student Success. Paper presented at LAK12, Vancouver, Canada.

Developing institutional strengthsThe OU is developing its capabilities in 10 key areas

8

The university needs world class capability in data science to continually mine the data and build rapid prototypes of simple tools, and a clear pipeline for the outputs to be mainstreamed into operations

We need to ensure we have the right architecture and processes for collecting the right data and making it accessible for analytics

– we need a ‘big data’ mind-set

Benefits will be realised through existing business processes

impacting on students directly and through enhancement of

the student learning experience – we will develop an ‘analytics

mind-set’ in these areas

The strategic roadmap will build these

capabilities prioritised using the indicators and

drivers of student success

Easily accessible OU dataLearning design and analytics at the OU

9

Relating design and outcomesLearning design and analytics at the OU

10

Innovaating in technology-enhanced learning (TEL)

The TEL Complex

11

The many elements of the ‘TEL Complex’ must all be taken into account

as an innovation is designed, developed

and embedded

Scanlon, E., Sharples, M., Fenton-O'Creevy, M., Fleck, J., Cooban, C., Ferguson, R., Cross, S. & Waterhouse, P. 2013. Beyond Prototypes. London: TEL Programme.

Rapid Outcomes Modelling Approach (ROMA)

The ROMA Framework

12

Ferguson, R., Macfadyen, L., Clow, D., Tynan, B., Alexander, S., & Dawson, S.. (2015). Setting learning analytics in context: overcoming the barriers to large-scale adoption. Journal of Learning Analytics, 1(3), 120-144.

Adapted from: Young, J., & Mendizabal, E. (2009). Helping researchers become policy entrepreneurs: How to develop engagement strategies for evidence‐based policy‐making. ODI Briefing Papers. London, UK: ODI.

Define (and redefine)

your policy objectives

What does success look like?

13

Academic analytics can guide future change

Student perspectives●Overall, I am satisfied with the quality of this module●Overall, I am satisfied with my study experience●I would recommend this module to other students●I was satisfied with the support provided by my tutor on this module●I enjoyed studying this module●This module met my expectations

Academic perspectives●The students were well prepared●The students met specified learning outcomes●The students defined and achieved their own learning goals

University perspectives●The module enhanced the university’s reputation●The module aligned well with others●The module generated income

What does success look like?

●Students demonstrate the skills necessary to network, collaborate, browse and reflect

●Students show progress towards defined learning outcomes●Students communicate well… when asked to collaborate●Students access and share links… when encouraged to browse●Students return to materials... when encouraged to reflect●Students engage with course content●Students seek out new challenges●Students persist when the work is challenging●Students persist in the face of failure●Students ask for help… when they are stuck

after several attempts●Students compare their learning strategies with those of experts●Students adapt their learning strategies to resemble those of experts

14

Learning analytics help to identify appropriate interventions

Policy objectivesOU Strategic Analytics Investment Programme

15

VisionTo use and apply information strategically to retain students and enable them to progress and achieve their study goals.

This vision requires• Discursive changes

to the communication of data and analytics

• Procedural changes in how learners are supported

• Behavioural changes associated with sustainable change in learner support.

Define (and redefine)

your policy objectives

Political contextMapping people and processes

16Tynan, B. & Buckingham Shum, S. (2013). Designing systemic learning analytics at the Open University.

http://www.slideshare.net/sbs/designing‐systemic‐learning‐analytics‐at‐the‐open‐university

Key stakeholdersOU Strategic Analytics Investment Programme

17

Define(and

redefine) your policy objectives

A community of stakeholders working in different areas:• Intervention and Evaluation• Data Usability• Ethics Framework• Predictive Modelling• Learning Experience Data• Professional Data• Student Tools

Key stakeholders are• University administrators• Students• Educators

Desired behaviour changesOU Strategic Analytics Investment Programme

18

Define(and


VisionTo use and apply information strategically to retain students and enable them to progress and achieve their study goals.

Desired behaviour changes• Staff will use and apply

information strategically• Students will extend their

learning journeys• Students will complete their

learning journeys• Students will set learning goals• Students will work effectively

towards study goals

Engagement strategyOU Strategic Analytics Investment Programme

19

Define(and


• Data in action is provided to stakeholders through a live portal, enabling them to understand learner behaviour and make adjustments and interventions that will have an immediate positive impact.

• Data on action is a more reflective process that takes place after an adjustment or intervention.

• Data for action takes advantage of predictive modelling and innovation in order to isolate particular variables and make changes based on a variety of analysis tools.

Internal capacity to effect changeOU Strategic Analytics Investment Programme

20

Define(and


Includes• Recruitment• Capacity building• Developing an ethical

framework for the use of learning analytics.

MonitoringOU Strategic Analytics Investment Programme

21Tynan, B. & Buckingham Shum, S. (2013). Designing systemic learning analytics at the Open University.

http://www.slideshare.net/sbs/designing‐systemic‐learning‐analytics‐at‐the‐open‐university

Policy objectivesUniversity of Technology, Sydney, Australia

22

VisionA university where staff and students understand data and, regardless of its volume and diversity, can use and reuse it, store and curate it, apply and develop the analytical tools to interpret it.

Teaching and learningEnsure that all stakeholders have the capacity to understand and interpret contemporary data‐rich environments.ResearchEnable researchers to think and act differently when designing their research methodologies and practices.AdministrationIdentify opportunities to obtain, generate, visualize, and communicate data and analyses that can improve core business outcomes.UniversityMine existing institutional data to identify areas that can provide direct evidence or assistance to staff and students.

Political contextUniversity of Technology, Sydney, Australia

23Ferguson, R., Macfadyen, L., Clow, D., Tynan, B., Alexander, S., & Dawson, S.. (2015). Setting learning analytics in

context: overcoming the barriers to large-scale adoption. Journal of Learning Analytics, 1(3), 120-144.

• Project initiated and led by Deputy Vice‐Chancellor and Vice‐President (Teaching and Learning)

• Pilot projects were completed successfully to secure ongoing funding.

• Critical to the success of the initial pilot projects was the existence of an Advanced Analytics Institute with internationally regarded researchers in big data, data sciences and analytics sciences.

• This enabled the establishment of a Connected Intelligence Centre.

Key stakeholdersUniversity of Technology, Sydney, Australia

24

Define(and


• 190 staff attended a one‐day ‘Data Intensive University Forum’, thus beginning a university‐wide conversation.

• Working party included Deputy Vice‐Chancellors, a senior member of library staff, and representatives of all faculties and administrative areas with relevant expertise

• Stakeholder buy‐in and ongoing participation in the project have been critical to its success.

Desired behaviour changesUniversity of Technology, Sydney, Australia

25

Define(and


• Provide information that can be used to decrease student attrition

• Provide a more detailed understanding of factors affecting low pass rates in subjects with very high failure rates over time

• Provide students with more information about their own study and engagement patterns

• Enable a more fine‐grained understanding of the influences of a range of possible interventions on pass rates and completions

• Provide valuable input to learning futures projects

Engagement strategyUniversity of Technology, Sydney, Australia

26

Define(and


• Give attention to institutional culture, ensuring engagement and buy‐in from key stakeholders through good communication and governance

• Invest in pilot projects of significant concern to the university and reporting of outcomes

• Invest in infrastructure: tools, applications, services

• Invest in expertise: recruitment of critical staff

• Provide leadership and engage institutional leaders

Internal capacity to effect changeUniversity of Technology, Sydney, Australia

27

Define(and


• Students and staff must be sufficiently numerate and equipped to make use of the analyses that analytics projects produce.

• A subject has been developed and trialled with staff

• The course develops students’ ability to engage with complex, extended arguments underpinned by numerical data as a key to participation as informed citizens in issues of significance to our culture and society

• The course available as an elective and will become compulsory

MonitoringUniversity of Technology, Sydney, Australia

28

• UTS has been engaged in a variety of learning analytics projects to assess scale and impact

• For example, the Outreach Programme rings as many commencing undergraduate students as possible. Early results consistently show a significant decrease in attrition in the group of students contacted.

Define(and


29

Slides online at www.slideshare.net/R3beccaF

Rebecca Ferguson @R3beccaFhttp://r3beccaf.wordpress.com/

Join the LACE project at the LACE SoLAR Flare on 9 October in Milton Keynes, UK, or online

lanyrd.com/2015/laceflare/

scaling up learning analytics

Education