cemca edtech notes: learning analytics for open and distance education

1CEMCA EdTech Notes

A topical start-up guide series on emerging topics on Educational Media and Technology

2 CEMCA EdTech Notes

IntroductionIntroduction

Learning analytics make use of large datasets inorder to improve learning and theenvironments in which it takes place. Students

and educators can use these analytics to reviewwork that has been done in the past, to supportcurrent study and to access recommendations forfuture activities. In the context of open and distance

education, analytics are able to draw oninformation about online learning activity in orderto support teachers and to help guide learners.Many of the major learning management systems(LMSs) used to support education worldwidecurrently have basic-level learning analytics builtinto them, and new tools are currently underdevelopment that will expand the use of analyticswithin open and distance education.

ContextThe recent development of learning analytics isassociated with several factors (Ferguson, 2012).The first of these is the increasing availability of‘big data’ – datasets so complex that they requirenew approaches to management and analysis.Within education, particularly within onlinedistance learning, Learning Management Systems(LMSs) collect and store enormous amounts ofadministrative, systems, personal and academicinformation. Institutions running these systemshave an interest in finding ways of extracting valuefrom their big sets of learning-related data.

At the same time, access to devices with Internetaccess has grown and there has been an

increase in the take-up of online learning.This offers many benefits for learners and

extends the availability of expertise andresources, but also poses challenges.Teachers who lack the visual cues thatare available in a face-to-faceclassroom may struggle to recognisewhen online students need morechallenge, when they are confused,overwhelmed or failing to engage.Learners, particularly those workingalone with open educationalresources, can find themselves adriftin a sea of information. Both learners

and teachers need tools that can helpthem to optimise opportunities for

learning.

In the wider context, educational institutionsand regions are increasingly seeking to measure,

compare and improve educational performance.Governments and local administrative bodies arelooking for ways of identifying best practice in order

to optimise learning and educational results atnational and international levels.

There is therefore a demand at national level, atlocal level and at the individual level of studentsand teachers for analytics that can not only makesense of the increasing amount of educational databut can also use it to support learning and teaching.

Learning AnalyticsThese various interest groups have made differentdemands on the big educational datasets that arenow available. This has resulted in the developmentof several related areas, with overlapping interestsand similar concerns. Broadly speaking, EducationalData Mining focuses on managing and making senseof data, while Academic Analytics deal withinstitutional and national priorities such asretention and success rates.

Learning Analytics are defined by the Society forLearning Analytics Research (SoLAR) as ‘themeasurement, collection, analysis and reporting ofdata about learners and their contexts, for purposesof understanding and optimizing learning and theenvironments in which it occurs’. A key element ofanalytics is that they are not primarily concernedwith reporting activity or with theoretical insights.Instead, they focus on the provision of ‘actionableintelligence’ that can provoke or encouragepractical action. In an educational setting, analyticsallow learners and educators ‘to increase the degreeto which our choices are based on evidence ratherthan myth, prejudice or anecdote’ (Cooper, 2012).

A review of key learning analytics tools (Dyckhoff,Lukarov, Muslim, Chatti, & Schroeder, 2013) has

Teachers who lackthe visual cues thatare available in a

face-to-faceclassroom may

struggle to recognisewhen online studentsneed more challenge,

when they areconfused,

overwhelmed orfailing to engage

3CEMCA EdTech Notes

Learning Analytics inUseThe range of available learning analytics tools andapproaches is constantly expanding. Here, threeexamples are used to give a flavour of how learninganalytics are currently being used. For a moredetailed overview of the field, see the freely availablereport on analytics and infrastructure produced byJISC CETIS (Kraan & Sherlock, 2013).

Course Signals at Purdue University

The Signals project, developed over a decade atPurdue University, applies statistical tests to large

highlighted ways in which they can be used byeducators and learners:

Educators can use learning analytics to

• monitor the learning process

• explore student data

• identify problems

• discover patterns

• find early indicators for success, poormarks or drop-out

• assess usefulness of learning materials

• increase awareness, reflect and self reflect

• increase understanding of learningenvironments

• intervene, supervise, advise and assist, and

• improve teaching, resources and theenvironment.

Learners can use learning analytics to

• monitor their own activities, interactionsand learning process

• compare their activity with that of others

• increase awareness, reflect and self reflect

• improve discussion participation,learning behaviour and performance

• become better learners, and

• learn.

Figure 1: Course Signals in use at Purdue University, USA

A ‘traffic signal’status display tells

students when thingsappear to be goingwell (green), when

the system hasdetected some causefor concern (amber)and when they havebeen classified as at

high risk (red)

datasets in order to predict, while courses arein progress, which students are in danger offalling behind. A ‘traffic signal’ statusdisplay tells students when things appearto be going well (green), when the systemhas detected some cause for concern(amber) and when they have beenclassified as at high risk (red). Thesignal colour is associated with adviceon action that the students can take toget back on track. The university reportsimproved retention levels when CourseSignals is used. Students in experimentalgroups seek help earlier than those in acontrol group, the percentage of studentsawarded A and B grades rises, and thepercentage awarded D and F grades falls (Pistilli& Arnold, 2012).

Social Networks AdaptingPedagogical Practice

A challenge within distance learning is to monitorand support activity within online discussionforums. The freely available SNAPPtool was developed to enableeducators to visualise the developmentof relationships within theseenvironments. SNAPP makes it clear


Mini CaseDr Patel teaches Arts 101, an online course at a largeCanadian university. Arts 101 has been designed to be acollaborative course, with the 200 students guided by tutorsas they engage in discussion and build a sharedunderstanding of course material. However, feedback fromprevious years suggests that some students struggle to engagein the discussion forums.

In a face-to-face setting it would be relatively simple toidentify and support students who are isolated andunengaged. In the standard forum view, it is difficult to do thiswithout looking at the contributions of every studentindividually.

Dr Patel installs the freely available SNAPP tool, developed foruse with standard learning management systems. This allowsher to visualise her forums as networks in which individualsare represented as nodes, linked by lines representing theconnections they have made as they have responded todiscussions. This view shows potential problems, allowing herto take action to support learning.

One group is in a characteristic ‘wagon wheel’ formation. Atthe hub of this network is the teacher – her connectionsradiate out in all directions, but the students arenot talking to anyone else. At the start of acourse, this is a typical formation, but by this

time students should be engaging in discussion with eachother. Changing behaviour here is easy, a prompt from thetutor encourages students to engage more widely.

Some isolated nodes in the main group are of more concern.These are students who have posted in the forum but havereceived no replies, so they appear as individuals with noconnections. Clicking on these nodes provides a link to basicdata. Worryingly, many of these students are receiving lowgrades, and are among those most in need of support. Acheck with the course team reveals these students are askingbasic factual questions and tutors are leaving it to otherlearners to respond. The problem is that other learners arenot responding. A two-pronged approach works here – thetutors identify and work with the isolated students, and alsoexplain to all participants the advantages of answeringquestions and addressing confusion.

Later in the course, the network has taken on a differentform, with two large groups that are not talking to eachother. Students focused on a history assignment are notengaging with others who are focused on literature. This lackof links means that the two groups are not sharing ideas andinformation. Again, the SNAPP visualisation allows Dr Patelto identify the problem and she is able to establish linksbetween the two groups.

when students have posted but received no reply, orwhen cliques are forming and dominating a forum.It shows when a teacher is taking the lead, whengroups are working together, or if a teacher isworking only with specific groups. Being able to seethese representations in real time, rather than when

the course has finished, makes it possible to takeaction. Where appropriate, isolated students can besupported and encouraged, groups reorganized, andeducator effort refocused (Bakharia & Dawson,2011).

Figure 2: Interpreting a SNAPP network diagram ofstudent interactions

Figure 3: Comparison of threaded forum views and SNAPP visualisationsof forum activity

5CEMCA EdTech Notes

Making Effective Useof Learning AnalyticsTools are not the only key element of learninganalytics. They form part of a system in whichhuman decisions and actions are also of keyimportance. Without attention to context, to peopleand to learning design, learning analytics cannotfunction effectively.

Context includes the problems that can arise whena large-scale learninganalytics systemundertakes real-timeanalysis of varied types ofdata in order to producereports. Unless the systemis confined to an LMS,data it employs are likelyto be generated fromsystems that were notdesigned with a view tosuch integration and areunlikely to offer dataconsistency. In addition,while using data from

different live operational systems, analytics mustnot limit the operational effectiveness of any ofthese systems. This may necessitate thedevelopment of a data warehouse where data can bebrought together and processed without interferingwith activity on the live system.

In order to use a learning analytics systemeffectively, people require support. Staff will need tobe able to select and interpret suitable metrics andto engage in good quality dialogue about how andwhy these are used. As learners and teachers arelikely to have limited experience of datavisualisation, this needs to be considered when

Identifying andSupporting Students‘At-Risk’ StudentsIn distance education it is sometimes difficult tofind and support students who are struggling on acourse. The JISC-funded RETAIN project at the UKOpen University uses online data to predict whichstudents are at risk of dropping out. The projectfound that variability amongst learners makes itdifficult to build the profile of an average student.Instead, the system looks for individuals whochange their patterns of activity. Online behaviourof learners is usually consistent until they hit aproblem. At that point, they may reduce usagebecause they are discouraged, or increase usagebecause they are searching for a solution. Analysisof their activity can be presented to tutors in a

dashboard, where it is possible to drill down formore data, and to record action taken (Wolff &Zdrahal, 2012).

Figure 4: Identifying at-risk students. RETAIN analytics based onengagement, submission and grade for tutor-marked assessments (TMAs)


ChallengesAlthough the use of student data to help with theachievement of desired learning outcomes is avaluable activity, it raises problems in terms ofapproaches to teaching and learning and in terms of

ethics and data management.

Analytics cannot be understood simply astools to support teaching and learning,

they carry with them assumptions abouthow learning takes place and abouthow it can be managed. In countriessuch as the UK and Australia, the useof analytics related to nationalschool tests has been associatedwith state management of educationand with enhanced regulation of theteaching and learning environment.There is always a danger thatanalytics will focus attention on the

achievement of a constrained set ofmeasurable criteria, and will distract

attention from other aspects of learning.Educators need to consider and be aware of

the models of teaching and learning thatunderpin any set of analytics, and the categories of

action and interaction prioritised by those models(Griffiths, 2013).

They also need to be aware of ethical issuesassociated with learning analytics. These includeconsideration of how data will be collected andstored, how it will be classified and managed, howand when it will be anonymised, and how it will beinterpreted and presented. Educational institutionsare likely to need a comprehensive data governancestructure, drawn up with reference to locallegislation, to help them to deal with these issues(Slade & Prinsloo, 2013). Procedures will berequired that will discourage people from providingfalse data in order to game the system, and that willhelp educators and students challenge findings if itappears that analytic tools are producing biasedresults or reinforcing stereotypes. In addition,institutions should always set time aside to testthat every predictive model employed is capable ofgenerating useful and valid predictive insights inthe local context. Being aware of potential problemsand planning to reduce them and deal with them iskey to the successful use of learning analytics.

designing dashboardsand developing trainingprogrammes. Theemotional elements ofassessment and feedbackmust also be taken intoaccount, developingeffective strategies forsharing data withlearners in ways that areboth supportive andsensitive.

Learning design is theprocess of putting inplace practices, activities,resources and tools thatwill allow learners toachieve specified learning outcomes in a givensituation. This helps interpretation of the data that

learner environmentsgenerate by linking theactivities of learners withthe outcomes they shouldbe working to achieve(Lockyer, Heathcote, &Dawson, 2013). It alsohelps to move the analyticfocus away fromsimplistic presentation ofdata relating to click ratesand test results towardslearning and teachingconcerns – ‘Whichelements are my studentsstruggling with?’ ‘Whatmisconceptions have theyshown? ‘How can

analytics help us to achieve the desired learningoutcomes?’

Procedures will berequired that willdiscourage people

from providing falsedata in order to gamethe system, and thatwill help educators

and studentschallenge findings if

it appears thatanalytic tools areproducing biased

results or reinforcingstereotypes

7CEMCA EdTech Notes

Finding Out MoreThe Society for Learning Analytics Research (SoLAR)organises both national and internationalconferences, including the annual LearningAnalytics and Knowledge conference (LAK);publishes the Journal of Learning Analytics; andruns an online doctoral school (SoLAR Storm) forstudents around the world.http://www.solaresearch.org/The CETIS Analytics Series of reports, available freeof charge online, explores key issues around thestrategic advantages and insights that analyticsbring to the education sector.http://publications.cetis.ac.uk/c/analyticsThe Educause Library brings together analyticsreports and case studieshttp://www.educause.edu/library/learning-analytics

The FutureTo date, a major focus of learning analytics hasbeen on their use in LMSs, where large amounts ofstudent data are readily available. This has provedfruitful in universities running such systems, incountries where regular and reliable Internet accessis easily available to students. In countries wherethese conditions are not met, learning analytics aremore difficult to implement (Faridhan, 2012).

New developments in learning analytics arereported in the proceedings of the annualinternational Learning Analytics and Knowledgeconference organised by SoLAR. Work is currently inprogress to develop analytics that can be used onmassive open online courses (MOOCs). This form ofopen learning has recently seen huge internationalengagement, and is capable of providing structuredresources and discussion to tens of thousands oflearners on a single course. MOOC analytics offerthe potential of providing support andrecommendations to learners who do not haveaccess to a local educational institution or whoonly have intermittent Internet access.

Other innovations currently under development arerecommender systems and evidence hubs that willhelp learners to search online educationalresources quickly and effectively. Data fromhandheld devices is being used to underpin appsthat can support learning and can offer suggestions

SummaryLearning analytics make use of ‘bigdata’ in order to understand anddevelop both learning and thecontexts in which it takes place.They focus on ‘actionableintelligence’ _ information,recommendations and prompts thatcan be used to increase studentengagement and learning. This is arelatively new field, but it draws onextensive work in related areas. As aresult, tools and methods are alreadyavailable that can support both learners andeducators. The majority of current tools arebest suited to students and teachers withreliable access to an online LMS. Toolscurrently under development will provide moresupport for learners in other contexts,including those engaged in open education,those with limited access to onlineenvironments and those learning with the helpof mobile phones.

and prompts via mobile phones. On every continent,researchers and educators are working to developanalytics that are sensitive to local needs andcontext.

Other innovationscurrently under

development arerecommender systems

and evidence hubsthat will help learners

to search onlineeducational resources

quickly andeffectively


References

Bakharia, A., & Dawson, S. (2011). SNAPP: A Bird’s-Eye View of Temporal Participant Interaction. Paper presentedat the LAK11: 1st International Conference on Learning Analytics and Knowledge (27 February – 1 March), Banff,Canada.

Cooper, A. (2012). What Is Analytics? Definition and Essential Characteristics. Bolton: JISC CETIS.

Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U. (2013). Supporting Action Research withLearning Analytics. Paper presented at the LAK13: 3rd International Conference on Learning Analytics &Knowledge (8 – 12 April), Leuven, Belgium.

Faridhan, Y. E. (2012). Exploring Data from Existing Systems That Are Useful in Assisting Student Learning: AnIndonesian Perspective. http://epress.lib.uts.edu.au/ocs/index.php/SoLAR/SSFC12/paper/viewPaper/440

Ferguson, R. (2012). Learning analytics: drivers, developments and challenges. International Journal of TechnologyEnhanced Learning (IJTEL), 4(5/6), 304-317.

Griffiths, D. (2013). The Implications of Analytics for Teaching Practice in Higher Education. Bolton: JISC CETIS.

Kraan, W., & Sherlock, D. (2013). Analytics Tools and Infrastructure. Bolton: JISC CETIS.

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: aligning learning analytics withlearning design. American Behavioral Scientist, 57(5), doi: 10.1177/0002764213479367.

Pistilli, M., & Arnold, K. (2012). Course Signals at Purdue: Using Learning Analytics To Increase Student Success.Paper presented at the LAK12: 2nd International Conference on Learning Analytics and Knowledge (30 April – 2May), Vancouver, Canada.

Slade, S., & Prinsloo, P. (2013). Learning analytics: ethical Issues and dilemmas. American Behavioral Scientist,57(5), doi: 10.1177/0002764213479366.

Wolff, A., & Zdrahal, Z. (2012). Improving retention by identifying and supporting ‘at-risk’ students, EDUCAUSEReview Online. Retrieved from http://www.educause.edu/ero/article/improving-retention-identifying-and-supporting-risk-students.

CEMCA EdTech Notes is a topical start-up guide series onemerging topics in the field of educational media andtechnology. New titles are published every year.

Series Editor: Sanjaya MishraDesigner: Sabyasachi Panja

Copyright © CEMCA, 2013. CEMCA EdTechNotes: Learning Analytics for Open and

Distance Education, is made available under a CreativeCommons Attribution 3.0 License (international):http://creativecommons.org/licenses/by-sa/3.0/

Views expressed in the CEMCA EdTech Notes are that of theauthor, and do not necessarily reflect the views of CEMCA/COL. All products and services mentioned are owned by their

Dr. Rebecca Ferguson is Research Fellow in the Institute of Educational Technology,The Open University, UK. She works at the intersection of complex social andtechnological change and cross-disciplinary research into Educational Futures.She fosters’ Open University research that is (re)conceptualising contemporaryand next-generation educational practices and technologies. She can be reachedat Rebecca[dot]Ferguson[at]open[dot]ac[dot]uk

respective copyrights holders, and mere presentation in thepublication does not mean endorsement by CEMCA/COL.

CEMCA in an international organization established by theCommonwealth of Learning, Vancouver, Canada to promotethe meaningful, relevant and appropriate use of ICTs toserve the educational and training needs of Commonwealthmember states of Asia. CEMCA receives diplomaticprivileges and immunities in India under section 3 of theUnited Nations (privileges and immunities) Act, 1947.

Printed and published by Mr. R. Thyagarajan,Head (Administration and Finance),CEMCA, 13/14 Sarv Priya Vihar, New Delhi110016, INDIA.Website: http://www.cemca.org.in