d3.3 content and - european data science...

Projectacronym: EDSA

Projectfullname: EuropeanDataScienceAcademy

Grantagreementno: 643937

D3.3ReportontheEvaluationofCourseContentandDelivery1

DeliverableEditor: Aba‐SahDadzie(OU)

Othercontributors:InnaNovalija(JSI),ErikNovak(JSI),RémiBrochenin(TU/e),JoosBuijs(TU/e),AlexanderMikroyannidis(OU)

DeliverableReviewers: AngiVoss(Fraunhofer)/SimonBullmore(ODI)

Deliverableduedate: 31/07/2016

Submissiondate: 29/07/2016

Distributionlevel: Public

Version: 1.0

ThisdocumentispartofaresearchprojectfundedbytheHorizon2020FrameworkProgrammeoftheEuropeanUnion

ChangeLog

Version Date Amendedby Changes

0.1 27/05/2016 Aba‐SahDadzie ToC&analysisofEDSALRM

0.2 03/06/2016 InnaNovalija JSIVideoLecturesData,DataAnalysisforVideoLectures,StatisticalAnalysis,VisualExploration&Analysis,PrototypeInterface,

PrototypeImplementation

0.3 06/07/2016 Aba‐SahDadzieRémiBrochenin

ToCedits,content

0.4 01/07/2016 Aba‐SahDadzie restructuring&consolidation

0.5 15/07/2016 Aba‐SahDadzie addressingreviewcomments

0.6 22/07/2016 ALL addressingreviewcommentsandfinalisingdeliverable

0.7 25/07/2016 Aba‐SahDadzie submissionversionforscientificreview

0.8 28/07/2016 ElenaSimperl Scientificreview

1.0 29/07/2016 AnetaTumilowicz FinalQA

D3.3ReportontheEvaluationofCourseContentandDelivery1Page3of71

2016©Copyrightlieswiththerespectiveauthorsandtheirinstitutions.

TableofContents

ChangeLog..................................................................................................................................................................................2

TableofContents......................................................................................................................................................................3

ListofTables..............................................................................................................................................................................5

ListofFigures............................................................................................................................................................................5

1. ExecutiveSummary......................................................................................................................................................6

1. Introduction.....................................................................................................................................................................7

1.1 Outline.......................................................................................................................................................................7

2. OnlineLearningEventDatasets..............................................................................................................................8

2.1 EDSAProjectData:LearningLocker.............................................................................................................8

2.1.1 DataPublicationPlan...............................................................................................................................11

2.2 OtherDatainusebytheEDSAConsortium............................................................................................11

2.2.1 JSIVideoLecturesData.............................................................................................................................11


2.2.3 TU/eMOO......................................................................................................................................................13


3. DataAnalysis................................................................................................................................................................15

3.1 LearningAnalyticsTaskemployingVideoLectures.............................................................................16

3.1.1 StatisticalAnalysis.....................................................................................................................................16

3.1.2 VisualExploration&Analysis................................................................................................................18

3.1.3 TheLandscape.............................................................................................................................................19

3.1.4 TheVideoLecturesLearningAnalyticsDashboard........................................................................23

3.1.5 Summary.......................................................................................................................................................28

3.2 LearningAnalyticsTaskemployingtheEDSALearningLocker....................................................29

3.2.1 BasicStatisticalAnalysis.........................................................................................................................30

3.2.2 VisualExploratoryAnalysis....................................................................................................................33

3.2.3 Course‐centricPerspective......................................................................................................................33

3.2.4 FocusonaSingleUser..............................................................................................................................34

3.2.5 Summary.......................................................................................................................................................35

3.3 LearningAnalyticsTaskemployingProcessMining...........................................................................36

3.3.1 Preprocessing:BuildinganEventLog................................................................................................36

3.3.2 VisualisationofLearningBehaviour...................................................................................................37

3.3.3 QuantificationofLearningBehaviour................................................................................................39

3.3.4 Summary.......................................................................................................................................................42

3.4 FocusontheEDSAOnlineCourse“FoundationsofBigData”.........................................................42

3.4.1 Statistical&VisualAnalysis...................................................................................................................43

of71EDSAGrantAgreementno.643937

3.4.2 ProcessMining............................................................................................................................................49

3.4.3 Normativemodel........................................................................................................................................50

3.4.4 Analysis..........................................................................................................................................................50

4. Discussion......................................................................................................................................................................52

4.1 InitialOverviewsofStudentOnlineLearningBehaviour..................................................................52

4.2 TriangulationofResultsforEDSACourse“FoundationsofBigData”.........................................53

4.3 ContributiontoFace‐FaceAndBlendedCoursesinEDSA................................................................54

4.4 SynergywithEDSADemandAnalysisTask............................................................................................54

4.5 InitialProposalfora‘LearningAnalyticsFramework’forEDSA...................................................55

5. Conclusions...................................................................................................................................................................57

5.1 NextSteps..............................................................................................................................................................57

6. References.....................................................................................................................................................................60

7. Appendices....................................................................................................................................................................61

7.1 AppendixA.‐ListofQueries..........................................................................................................................61

7.1.1 FollowaNamedUser................................................................................................................................61

7.1.2 ResultsSnapshot.........................................................................................................................................62

7.1.3 CaptureUserActivities.............................................................................................................................67

7.1.4 GroupbyEventwithinaCourse............................................................................................................67

7.2 AppendixB.‐SampleSnapshotsofEventDataforCourse"FoundationsofBigData".........68

7.3 AppendixC.‐ThirdPartyAPIsandAnalysisToolsUsed...................................................................71

7.3.1 VideoLecturesLearningAnalyticsDashboard.................................................................................71



ListofTablesTable4.1:Topfivecoursesbyactivityandenrolment...

Table4.2:Fromthemostactive,thetop30...

ListofFiguresFigure3.1:TheEDSALearningLockerdashboard.

Figure3.2:SnapshotofaVideoLecturesrawlogfile...

Figure3.3:SnapshotofaVideoLecturesrangeslogfile...

Figure4.1:SearchcloudatVideoLectures.NETportal...

Figure5:SearchoptionsforVideoLecturesExplorer.The...

Figure4.3:Thelandscapeofdatamininglectures,...

Figure4.4:Thelandscapeofdatabaselectures,...

Figure4.5:Thelandscapeofpythonlectures,...

Figure4.6:Theadditionalinformationwindow.Createdusing...

Figure4.7:VideoLecturesViewTrends.

Figure4.8:VideoLecturesDownloadTrends.

Figure4.9:VideoLecturesSearchTrends.


1. ExecutiveSummary

ThisdeliverablereportsthestartofworkinEDSATask3.4:theinitialevaluationofcoursecontentanddelivery, by studying student interactionwith online coursematerial. This is to obtain insight intostudent engagement and behaviour, and the impact of student behaviour, demographics and other(external)factorsoncoursecompletionandsuccess.

Deliverable3.3reportsexploratoryanalysisofeventdatafrom(i)theEDSAproject’sportalprovidingaccesstoopen,online,self‐studycoursesinDataScience;(ii)aCourseraMOOConaselectedtopicinDataScience;(iii)aportalcontaininglearningresourcescapturedonvideoonabroadvarietyoftopicsincludingDataScience.Eachdataset isanalysed independentlyusingoneormoreof threedifferentanalysis techniques ‐ statistical analysis, visual analysis and process mining. All three techniqueshighlightedcommoninterestinorpopularityofspecificresourcesandcoursesoverall;tovalidateboththe approach taken and the appropriateness of each technique employed for the datasets and theanalysisrequiredweapplythethreetechniquestothemostpopularcourseintheEDSAOnlineCoursesportal(byeventandenrolment).

Triangulatingourresultsweseebothoverlappingfindingsandinformationrevealedonlyusingasingleapproach,eachofwhichtakesaparticularperspectiveonthedata.Overall,studentactivitypatternsshowintermittentaccesstocoursesovertime,withhighattritionshortlyafterenrolment,andgradually,butnotuniformly,decliningtowardthelatterstagesofacourse.Accesstocoursematerialvaries,withsomeresourcesseeingsignificantlymoreaccess thanothers.Onlysomecourses includeassessmentmaterial; further analysis as Task 3.4 progresseswill examine inmore detail the impact of studentbehaviourandotherfactorsonperformanceinbothindividualcoursesandingeneralforonlineself‐study.Weaimalsotoidentifypotentialcriteriaforassessingwhatlearninghasoccurredwhereexplicit,objectiveassessmentand/orcoursegradesarenotavailable.

TheoutcomesofLearningAnalyticsinWorkPackage3aretoleadultimatelytoevidence‐basedbestpracticethatwillguide(re)designofcoursecontentanddeliveryandprogrammecurriculainthefieldofDataScience,andtoaidwherenecessarytailoringoflearningmaterialtofitparticularcontexts.Weenvisagealsothatourfindingswillcontributetoguidingstudentsinselectingcoursesthatmeettheirrequirementsforself‐improvementandaspartoftheprocessesofskillstrainingandjob‐seeking.

Becausetheanalysisreportedhereispreliminaryweindicatemainlyfurtherdirectionsforresearchandanalysisbasedonpatternsandtrendsrecognisedaswecarriedouttheexploratoryanalysisofthethree datasets. Where strong evidence exists for making early recommendations we suggest alsopotentialpathstoexplorefurthertoreinforcethese.



1. Introduction

TheobjectiveofWP3inEDSAisto:

1. Deploy the course material developed in WP2 for different target groups and in differentenvironments:webinars,videolecturesandface‐to‐facetraining.

2. Gatherfeedbackabouttheeffectivenessoflearningfromthesecourses.3. Analyse feedback and other data generated during course delivery, to feed into improving

contentand/orformofdeployment,aswellasintothedesignofnewcourses.WorkPackage3(WP3)producestwoseriesofdeliverables.Thefirstseriesaddressesobjectives1and3andthesecondobjectives2and3.Deliverable3.3(D3.3)isthefirstinthesecondsetofdeliverables,within Task 3.4 (T3.4), and focuses on initial analysis of the content of and interactionwith onlinelearningmaterialproducedwithintheEDSAprojectandbyprojectpartnersontopicsrelatedtoEDSA’sremit.

Thedeliverabledescribesfirstthethreeeventdatasetsoncoursesbeingdeliveredandrelevantlearningmaterial‐bothinprogressandcomplete‐andtheanalysisapproachesemployedtoobtainanoverviewofthedata,followedbymorein‐depthanalysisofselectedcoursesandstudentpaths.Theultimateaimistoprovideapictureofstudentbehaviouracrossallandindifferentcoursesandhowthisrelatestocompletion,timetocompletionandstudentgrades.

Wetriangulatetheresultsobtainedfromthedifferentanalysisapproachestoobtainbroaderreachingandmorereliableconclusionsonstudentbehaviour.Weaimtomeasuretheimpactoftypeoflearner(e.g.,professionalorpractitionervs. collegeoruniversity student), courseandcontentdeliveryandtopic,amongothers,onstudentsuccess.Basedontheresultsofouranalysisandprojectdiscussionsfeeding into the deliverable we propose a foundation for a structured process to guide LearningAnalytics(LA)withinEDSA,toensure:

1. event (user interaction) data collection captures all aspects of student activity required toobtainasufficientlycomprehensivemapofbehaviourthatallowseffectiveprovisionoffeedbacktobothstudentsandcourseinstructors/owners,

2. learninganalyticswithintheprojectcontributestothestateoftheartinthefield,with,wherepossible,usecasesandbenchmarksthatcanbereleasedforfurtherreuse,asLinkedOpenData,

3. the outcomes of research on learning analytics for online, self‐study courses inWP3 cross‐fertiliseotherrelatedworkinEDSA,keybeing(i)theanalysisofface‐faceandblendedcourses,and (ii) Data Science job demand and skill analysis, to identifywhere and how skill‐driventrainingcanaidtheclosingoftheskillsgaprecognisedintheproject.

1.1 OutlineSection2providesdescriptionsof the three eventdatasets analysed in thisdeliverable, licenses forreuseandthedatapublicationplanfordataproducedwithintheEDSAproject. Section3detailsthedataanalysiscarriedoutforthethreedatasets,usingoneormoreofthethreeapproaches:statisticalanalysis,visualexploratoryanalysisandprocessmining.Allthreeapproachesarethenappliedtothecourse in theEDSALearningManagementSystemthat saw thehighestenrolmentandrecorded thelargestnumberofeventsoverall‐“FoundationsofBigData”.Triangulatingtheresultsobtainedhere(section4.1.2)allowsustovalidateboththeanalysisapproachesemployedandbegintopointtothe


formalisationofthelearninganalyticsprocesstobefollowedwithinEDSA.Thedeliverableconcludeswithadiscussionofthefindingsofthisinitialexploratoryanalysis(section4)andplans(section5.1)forlearninganalyticsforthesecondhalfoftheproject.

2. OnlineLearningEventDatasets

TheanalysiscarriedoutforthefirstphaseofTask3.4looksatthreemaindatasets:

1. theEDSAprojecthostsaLearningManagementSystem(LMS)thatcollectseventdatageneratedthroughinteractionwiththecoursesofferedfromtheEDSAonlinecoursesportal1,

2. theVideoLectures.NET2portalhostedbyJSIthatprovidesopenaccesstolearningmaterialasonlinevideo,

3. theCourseraMOOC‘‘ProcessMining:DataScienceinAction”deliveredbyTU/e.Thissectionprovidesadescriptionofeachdataset,detailingeventdatacollected,addressingofprivacyconcernsandlicensesforreusewithintheprojectandbythirdparties.

2.1 EDSAProjectData:LearningLocker

AsdescribedinD2.4,thelearningmaterialproducedbytheproject isdeliveredviatheEDSAonlinecoursesportal1.TheportalisanLMSbasedontheopen‐sourceMoodle3software.Ithoststhefullsetoflearningmaterial(presentations,webinars,text,quizzes,etc.)fortheEDSAself‐studycourses,towhichlearnerscanenrol andstudyat theirownpace.Learnersmayregister toaccess theportalusinganexistingsocialmediaaccount,suchasGoogle,FacebookandLinkedIn,orrequestanEDSAaccount.TheportalalsolistsothertypesofEDSAcourses,bothonlineandnot‐MOOCs,blendedandface‐to‐face(f‐f) courses.Foreachcourse there isabriefoverviewanda link to thededicatedcoursepageon thewebsiteoftheEDSApartnerorassociateEDSApartnerthatoffersthecourse;thisdataisalsoincludedwithcorrespondingeventlogs.

InordertocollectdataforLearningAnalyticsfromtheportal,wehavedeployedanopen‐sourceMoodleplugin4 that conformswith the xAPI (or Tin Can API) specification. xAPI provides a framework forcapturingstatementssuchas“someonedoesanactionto/withsomething”,e.g.,“Johnhasinitiatedanexperiment”.Itoffersanextensiblevocabularyof:

● verbs5,whichdescribe theactionperformedduring the learningexperience, e.g. “answered”,“asked”,“interacted”.

● activities6,whichdescribesomethingwithwhichanActorinteracts,e.g.a“course”,“question”,“simulation”.

1http://courses.edsa‐project.eu2http://videolectures.net3https://moodle.org4https://moodle.org/plugins/logstore_xapi5 http://adlnet.gov/expapi/verbs6 http://adlnet.gov/expapi/activities



ThefollowingcodesnippetshowsanexamplexAPIstatementinJSONdeclaringthatauserhasenrolledontoacourse(keyeventactioninbold):

"actor":{ "name":"Testuser_name", "account":{ "homePage":"http:\/\/www.example.com\/user_url", "name":"1" } }, "context":{ "platform":"Moodle", "language":"en", "extensions":{ "http:\/\/www.example.com\/context_ext_key":{ "test_context_ext_key":"test_context_ext_value" }, "http:\/\/lrs.learninglocker.net\/define\/extensions\/info":{ "https:\/\/moodle.org\/":"1.0.0", "https:\/\/github.com\/LearningLocker\/Moodle‐Log‐Expander":"1.0.0", "https:\/\/github.com\/LearningLocker\/Moodle‐xAPI‐Translator":"1.0.0", "https:\/\/github.com\/LearningLocker\/xAPI‐Recipe‐Emitter":"1.0.0" } }, "contextActivities":{ "grouping":[ { "id":"http:\/\/www.example.com\/app_url", "definition":{ "type":"http:\/\/id.tincanapi.com\/activitytype\/site", "name":{ "en":"Testapp_name" }, "description":{ "en":"Testapp_description" }, "extensions":{ "http:\/\/www.example.com\/app_ext_key":{ "test_app_ext_key":"test_app_ext_value" } } }a } ], "category":[ { "id":"http:\/\/www.example.com\/source_url", "definition":{ "type":"http:\/\/id.tincanapi.com\/activitytype\/source", "name":{ "en":"Testsource_name" }, "description":{


"en":"Testsource_description" } } } ] }, "instructor":{ "name":"Testinstructor_name", "account":{ "homePage":"http:\/\/www.example.com\/instructor_url", "name":"1" } } }, "timestamp":"2015‐01‐01T01:00Z", "verb":{ "id":"http:\/\/www.tincanapi.co.uk\/verbs\/enrolled_onto_learning_plan", "display":{ "en":"enrolledonto" } }, "object":{ "id":"http:\/\/www.example.com\/course_url", "definition":{ "type":"http:\/\/lrs.learninglocker.net\/define\/type\/moodle\/course", "name":{ "en":"Testcourse_name" }, "description":{ "en":"Testcourse_description" }, "extensions":{ "http:\/\/www.example.com\/course_ext_key":{ "test_course_ext_key":"test_course_ext_value" } } } }

ThexAPI statements collected from theEDSAonline coursesportal are stored inan instanceof theLearning Locker7 software, the reference open source LearningRecord Store (LRS). It offers a datarepository designed to store learning activity statements generated by xAPI compliant learningactivities.Amongitsfeaturesistheabilitytovisualisedataviadashboards,scalability,functionalityforgeneratingcustomreports,itsRESTfulAPI,aswellasthefunctionalityforexportingdataasCSVandJSON.Figure1showsascreenshotofthedashboardfortheEDSALearningLocker8.

7 http://learninglocker.net8http://analytics.edsa‐project.eu



Figure3.2:TheEDSALearningLockerdashboard.

2.1.1 DataPublicationPlan

The Learning Analytics dataset from the EDSA online courses portal has been published9 under aCreativeCommonsAttribution4.0licenseandwillbeupdatedquarterly.ThedatasetconsistsofxAPIstatementsdescribingtheinteractionofuserswiththeEDSAonlinecoursesportal.Itisanonymisedtopreventidentificationofusers,butwithdistinctIDsthatallowthetrackingofindividualuserpaths.

2.2 OtherDatainusebytheEDSAConsortium

2.2.1 JSIVideoLecturesData

VideoLectures.NET2 isa freeandopenaccesseducationalvideo lecturesrepository. It containsover20,000lecturesbyscholarsandscientistsatresearcheventsincludingconferences,summerschools,workshopsandsciencepromotionalevents.

Lecturesontheportalfallintovariouscategories,includingDataScience,asestablishedwithintheEDSAproject.

9https://alexmikro.github.io/learning‐analytics‐dataset‐from‐the‐edsa‐online‐courses‐portal


ForthefirstroundofLearningAnalyticstasksonVideoLectures,weuseasetofrawlogfilesextractedfromtheVideoLectures.NETportalcontainingeventsbetweenSep2012andDec2015.The log filesprocessedincludethefollowinginformation:

● IDofentry● timestamp● SessionID● logentrytype● lecture● otherinformation,includingeventtype,IPaddress,location(ifcaptured).

Figure1presentsasnapshotofaVideoLecturesrawlogfile.

Figure3.2:SnapshotofaVideoLecturesrawlogfile.

Processingtherawlogsextractskeyeventtypessuchas:

● view‐auseraccessingalecturewebpage,● download‐theuserdownloadingapresentation,e.g.,avideo,fromthelecturewebpage,● search‐theuserperformingasearchusingtheportal.

Wealsocollectedlogfilescapturingthebehaviourofselecteduserswatchingselectedlectures‐referredtoasrangeslogfiles.

Rangeslogfiles(seeFigure3)presenttheactionsoftheuserwhilewatchingvideos,suchasmovingforwardorbackwardsontheplayer,skippingsomevideosection,etc.

# id,time,session,log,lecture,ref_type,ref_id,ref_name,"data" 101220275,2015-01-01 00:00:01.622361,101220276,,seehealth2010_golob_bpas,vl.lecture,11547,seehealth2010_golob_bpas, 101220276,2015-01-01 00:00:01.637593,101220276,session,,vl.log,101220276,,"ip=188.165.15.126 key=8bxdr82qbswggputjnocc52034t4isop" 101220277,2015-01-01 00:00:05.724054,101220278,-,mit7012f04_introduction_biology,vl.lecture,6305,mit7012f04_introduction_biology, 101220278,2015-01-01 00:00:05.743161,101220278,session,,vl.log,101220278,,"lc=AU ip=123.125.71.90 key=em0n1ml4mbydy8v55y40z4ieh9ge5od6" 101220279,2015-01-01 00:00:06.423510,101138550,-,wsdm08_yang_aksbch,vl.lecture,4309,wsdm08_yang_aksbch, 101220280,2015-01-01 00:00:09.773501,101220281,-,icml08_larochelle_cud,vl.lecture,5281,icml08_larochelle_cud, 101220281,2015-01-01 00:00:09.798654,101220281,session,,vl.log,101220281,,"ip=95.82.45.167 key=fgf5m0dccxbnhma26aayjo92u6pt7tvc"



Figure1.3:SnapshotofaVideoLecturesrangeslogfile.


Summaries of the VideoLectures data are available for download10, with a Creative CommonsAttribution‐Noncommercial‐NoDerivativeWorks3.0license,inlinealsowithrequirementsforreuseinEDSA.

2.2.3 TU/eMOO

ThedatasetsweanalyseherewereobtainedfromCourserafortheMOOC‘‘ProcessMining:DataSciencein Action’’11 for 43,218 registered students. These datasets are centred around the studentsparticipatingintheMOOCandthestreamofclickeventstheygeneratedonthecoursewebpages.Thedatasetstructurecomprises:

● clickstreamdata,● studentdata,● coursestructuredata.

10https://github.com/innanoval/edsa‐videolectures‐statistics‐dataset‐1/tree/gh‐pages/data11https://www.coursera.org/learn/process‐mining

{"id": 101220316, "time": "2015-01-01T00:01:07Z", "session": "fxkr0jtofwbsm57wo8nibnta1mvjwahm", "username": null, "ranges": [0, 25654, 859046, 862966, 1180104, 1189517, 1102009, 2307009], "video": {"id": 11018, "part": 1, "lecture": {"id": 10825, "type": "vl", "slug": "yalephys200f06_shankar_lec23"}}} {"id": 101220399, "time": "2015-01-01T00:03:45Z", "session": "zvcyr4hteql2ha777wirs2ikjwsre2uz", "username": null, "ranges": [0, 33697], "video": {"id": 4842, "part": 7, "lecture": {"id": 4336, "type": "vtt", "slug": "mlss08au_lucey_linv"}}} {"id": 101220411, "time": "2015-01-01T00:04:08Z", "session": "nmg1ycikq82ennvw10vce9iv2kn7u1zf", "username": null, "ranges": [351977, 360658, 3605976, 3657013, 3911988, 3917335, 3920258, 3922223], "video": {"id": 5335, "part": 1, "lecture": {"id": 5262, "type": "vit", "slug": "icml08_collins_spp"}}} {"id": 101220423, "time": "2015-01-01T00:04:26Z", "session": "liwah46wrs9jzxhwoc84d17jd89a27a5", "username": null, "ranges": [1637170, 2290000], "video":{"id": 4730, "part": 1, "lecture": {"id": 4645, "type": "vl", "slug": "mit801f99_lewin_lec01"}}} {"id": 101220498, "time": "2015-01-01T00:06:02Z", "session": "tgszt77hwshb2zbs4gaohioohp363vxg", "username": null, "ranges": [1215575, 1391116, 1058963, 1281640], "video": {"id": 4749, "part": 1, "lecture": {"id": 4664, "type": "vl", "slug": "mit801f99_lewin_lec20"}}} {"id": 101220625, "time": "2015-01-01T00:09:35Z", "session":


Clickstreamdata:duringacourse,studentsvisitthecoursewebsiteto,amongothers,watchlecturevideosandanswerquizzes.Asstudentsclickthroughthewebsitetolookupthesevideosandquizzes,they leavea trailofclickevents,collectivelycalledaclickstream.Eachevent inaclickstreamcanbeassociatedwith,e.g.,aparticularlectureoraparticularquizsubmission.Pagesvisitedbyastudentarerecordedasapageviewaction, inadditiontointeractionwithothercoursematerial,e.g.accessingalecturevideoisrecordedasavideoaction.

Studentdata:foreachstudentwehaveinformationontheexacttimeregisteredforthecourseandtheirfinalcoursegrade.Registrationrecordsalsoincludewhetherastudentregisteredforthespecial(paid)signature track, inorder toobtain a verified certificate. The coursegrade consistsof twoparts: thenormalgradeandadistinction. Inaddition,eachstudent isassignedanachievement levelbasedongradesobtained.Whereastudentdoesnotcompletethecourseexams,theachievementlevelisabsent.Whereastudentcompletestheexamsbutwithanormalgradebelowthepassmarkthisisrecordedasfailingthecourse.Studentswithnormalgradesaboveboththenormalpassmarkandthedistinctiongradeachievetheleveldistinction.

Course structure data: in Coursera MOOCs, lectures and quizzes are grouped into sections, eachtypicallyspanningaweek.Eachsection isvisible to thestudentsatapredetermined time(theopentime),inordertogivestructuretothecourse.Withinasection,lecturesandquizzesmayhavetheirownopentime,tofurtherguidestudentstofollowaparticularstudyrhythm.Quizzeshavedeadlines(theclose time) and may be attempted multiple times up to a given submission maximum before thisdeadline.


DuetorestrictionssetbyCourserathedatausedfortheanalysisofthiscoursecannotbemadeavailableforreuse.



3. DataAnalysis

AsT3.4progresseswewillcontinuetoexamineonlinelearningresourcesusingdifferentapproaches,including those reported in this deliverable, to generate multiple views on the data and thereforeincreasecoverageinouranalysis,toobtainamorecompletepictureofthepatternsandtrendswithinthedata.Tovalidatebothourapproachandourresultswe:

1. employmultipleanalyticaltechniquesonasingledataset–theEDSALearningLocker–to:a. identifydifferencesinpatternsduetodifferencesinperspectivetaken,b. validate,throughtriangulation,theresultsobtainedforeachapproachandthe

approachesthemselves,c. identifyadditionalrequirementsfordatacollectionandanalysisnecessarytoprovide

confidentrecommendationsoncourseandcurriculumdesign,andtomatchuser(skill)profilesandintereststolearningmaterial.

2. employthesameanalyticaltechniqueonindependentlycollecteddatasets–toidentifydifferencesinpatternsandtrendsdueto:

a. learnertype,b. coursetype,c. contentdeliverymechanism,d. commitmentoreffortrequiredofstudents.

Sections3.1,3.2and3.3reportexploratoryanalysisofthethreedatasetsdescribedinsections2.1,2.2.1and2.2.3respectively,usingoneormoreofstatisticalanalysis,visualanalysisandprocessmining.Thesectionconcludes(in3.4)byapplyingallthreeanalyticalapproachestothedataforonecourseintheEDSALMS,"FoundationsofBigData"20.Theaimofthisexerciseistogenerateoverviewsofdatacontent,toprovideaninitialpictureofstudentinteractionwiththeresourcesavailableineachdatasets,andidentify recurringandpotentiallyanomalouspatternsofbehaviour.This is tohelpusdiscoverhowstudentbehaviourmapstoidentificationofresourcesrelevanttotheirneedsandinterests,subsequentengagementwiththematerialandotherstudentsandinstructors,coursecompletionandperformance.Wesummariseherehowweemploythethreetechniques,withexamplesofrelevantworkinthefield.Wereportineachsub‐sectionadditionalinformationonwheredatastructureand/orcontentlimitthedepthofanalysisorthetechniqueinuse.

Statisticalanalysis:thisistoprovidesummariesofthedatastructureandcontent,andhelptoidentifyregionsofinterest(ROIs)andoptimaltechniquestoemployformoredetailedanalysis.OfthethreedatasetsanalysedinthisdeliverableonlytheVideoLecturesdatasetcontainssufficientdatatocarryoutstatistically significant analysis; section 3.1 therefore provides relatively more detailed statisticalanalysis,while sections 3.2 and3.3 report only basic summaries, focusingmoreon other analyticalapproaches.

Visualanalysis:thisistoemployvisualisationasameanstorevealpatternsandtrendswithinthedata,toremovecognitiveloadtomoreintuitiveperceptionandaidespeciallyin‐depthanalysis[Thomas&Cook2005].Wemakeuseofdifferentvisualisationtechniquestosupportordriveouranalysis;thisapproach is not unknown ‐ Conde et al., (2015) demonstrate the construction of a visual analyticsdashboardtosupporttheapplicationof learninganalyticstoinformeddecision‐makinginarealusecase. The study,which analyseddata collected over five years, demonstrates the power in utilisingmultiplevisualisation‐drivenanalyticstechniquestosupportanalysisaswellasreportingandthereuse


of findings to thedifferentparticipants,as inourcase, to studentsand instructors.Bakhariaetal.,(2016a),aspartofresearchtowarddevelopingaconceptualframeworkforLA,reportfindingsfromavisualanalyticsdashboarddevelopedtoaidenquiry‐basedevaluationoflearningdata.Theyaim,aswedo,tofeedtheresultsofLAintothedesignoflearningmaterial.

Processmining:thisistosupporttheconstructionofamodeldescribingthelearningprocessasdefinedbycoursedesignersand/orinstructors intheformalcoursestructures.This isto,amongothers,aidprocess discovery, conformance checking and enhancement of the models in use, using a set ofalgorithms,toolsandtechniquestocompareeventdataduringactualinteractionwithacourseagainstthis defined structure [van der Aalst 2011]. Mukala et al., (2015), for instance, demonstrate theapplication of process mining to a MOOC, to investigate student behaviour as recorded by eventscaptured during interaction with course material. The study provides insight into differences andsimilarityacrosscategoriesofstudentsthroughoutthecourseandhowthiscorrelatestocompletionandperformance.

AppendixAcontainssnapshotsofthedataextractedfromtheEDSALearningLocker(seesection2.1)usedinouranalysis.

3.1 LearningAnalyticsTaskemployingVideoLectures

InordertoanalyseuserbehaviourattheVideoLectures.NETportal,weemployasetofdataanalysistechniques,usingweb‐based, interactiveprototypeshostedontheVideoLecturesLearningAnalyticsDashboard12toprovidestatisticalanalysisandvisualexplorationfeatures.

3.1.1 StatisticalAnalysis

Thisanalysisisperformedfromfourperspectives:

● aggregatedperspectiveforalllectures● perspectiveofasinglelecture● aggregatedperspectiveofallviewers● perspectiveofsingleviewer.

Inaddition,wehavedevelopedasetofmetricsforviewersandlecturesthatprovideinsightintouserbehaviourontheVideoLectures.NETportal.

Lecturemetrics(seeFigure4.8):

● numberofviewers ● numberofviews

12http://learninganalytics.videolectures.net



● avg(average)movesforward ● avgmovesbackward ● avgtimespent● avgtimein%(consideringfullvideotimeas100%) ● stddev(standarddeviation),timespent ● stddev,timein% ● stddev,movesforward● stddev,movesbackward.

Viewermetrics(seeFigure4.9):

● numberofviews ● avgtimespent● avgtimein% ● stddev,timespent ● stddev,timein% ● avgmovesforward ● stddev,movesforward● avgmovesbackward ● stddev,movesbackward.

Theanalysispresentedhereconsidersasetof1,000mostpopularvideos(byviews)relevanttoEDSAtopicsfortheperiodSep2012toDec2015.Userinteractionresultedin:

● 7,055,427views● 3,020,090downloads● 1,045,860visitors● 866,912searcheswithintheportal.

TheVideoLecturesportalisindependentoftheEDSAproject,withdatacoveringamuchwiderrangeoftopics and fields. However, the broad range of topics coveredmeans that it also captures learningmaterialofrelevancetoEDSA;Figure4.1showsasearchcloudgeneratedfrompopulartermsusedtoretrieve resources during the collection period.We see that interest extends to skills core to DataScience as identified by research in EDSA, some of the more frequently used including “machinelearning”, which corresponds also to the acronymMLSS ‐ the “Machine Learning Summer School”,statistics,datamining,deeplearning,theprogramminglanguagepythonwhichsawhighfrequencyofoccurrenceinselectedregionsintheEUinthedemandanalysiscarriedoutinWP!(seeD1.4).Wealsoseeadditionalrelatedterms,includingtheproduct“hadoop”,goodknowledgeofwhichcontributestotheanalysisof“bigdata”,anotherkeyEDSAskill/term.


Figure2.1:SearchcloudshowinginterestinlearningmaterialavailablefromtheVideoLectures.NETportalfordatacollectedSep2012‐Dec2015‐includinganumberofkey

skillsinDataScience.

Condeetal.,(2015)intheirlearninganalyticsdashboardillustratetheuseofasemantictagcloudtomapevolutionoftopicsincoursediscussionfora.AsweprogressthroughT3.4wewillcarryoutsimilaranalysis,usingperiodicsnapshotsofinterestasexpressedthroughsearchforlearningmaterialtofollowhowthischangesspecificallyforDataScienceterms,asthefieldgrowsanddemandforrelevantskillsbecomesmoreclearlydefined.

3.1.2 VisualExploration&AnalysisThis section details the prototype interface developed for interactive analytics tasks on theVideoLecturesportal‐theinteractive,web‐basedVideoLecturesLearningAnalyticsDashboard10andtheVideoLecturesExplorerTool.

VideoLectures Explorer: is a tool for exploring the lectures published on VideoLectures.NET. Thedashboardenablestheusertosearchthroughlecturesandfindsimilaritiesbetweenthem,e.g.,tofindlecturesofaspecificcategoryorpresenterofinterest.

Database:thedatabaseusedforthevisualisationandbasicstatisticalanalysiscontainsdatafromalllecturesonVideoLectures.NET.Foreachlecturethedatabasecontainsthelecture'stitleanddescription,thenameofthepresenterandtheorganisationwherethelecturewasheld,thelanguageinwhichitwaspresented,itspublicationdate,videoduration,numberofviewsandthescientificcategoriesthelecturebelongsto.ThedatabasewasconstructedusingtheVideoLecturesAPI13.

13http://videolectures.net/site/api/docs



SearchOptions:theseareintwoparts:thebasicandtheadvancedsearch(seeFigure5).Basicsearchoptionsconsistofasingleinput,wheretheusercanenterthenamesofpresenters,organisationsorcategories.

Theadvancedsearchoptionsmaybeaccessedbyclickingonthebuttonnexttothesearchbutton.Itexpandstheadvancedoptionswhichcanbeusedforamorerefinedsearch.Thisallowsthesearchertospecifywhetherinputsearchtermsaretobematchedtocategories,presentersororganisation.Further,itenablessearchbynumberofviewsandthelanguagethelecturewaspresentedin.

Thesearch fieldsprovideautocomplete functionality toaid identificationofmatchingkeywordsandterms.

Figure3:SearchoptionsforVideoLecturesExplorer.Theredsquaremarkstheadvancedoptionsbutton.

3.1.3 TheLandscapeThemain visualisation is the landscape. The snapshots in Figures 4.3, 4.4 and 4.5 show similaritybetweenlecturesinacategory,byqueryingusingthreeskills‐“DataMining”,“Databases”and“Python”,respectively‐identifiedascoretodatascienceaspartofresearchinEDSA,andwhichdemandanalysisinEDSAfoundtobeinhighdemandforjobrolesinDataScience(seeD1.1andD1.4).Eachlectureisrepresentedbyapointwithsizemappingtonumberofviews.Similaritybetweenlecturesismappedtodistancebetweenpoints;moresimilarlecturesarebroughtclosertogether.Hoveringoverapointbringsupatooltipcontaininginformationaboutthelecture:itstitleanddescription,thenameofthepresenterandorganisationwherethelecturewasheld,thelanguageinwhichthelecturewaspresented,whichscientificcategoriesitbelongsto,itsduration,whenitwaspublishedandthenumberofviewssincethelastdatabaseupdate.Anydataattributeforwhichvaluesarenotavailableisomittedfromthetooltip.Landmarksshowareaspopulatedwithlecturesthatareofthesamecategory.Theusercanzoominandoutofthelandscapetoenablemoredetailedsearch.Whilezoomingthesizeofthepointsgetlargersoit’seasierfortheusertohoveroverthepointsandgetthelectureinformation.

InFigures4.3,4.4and4.5weseethatqueryingonthethreeskillshighlightsotherrelevant lecturescategorisedunder thoseused in thequeries ‐ “DataMining”, “Databases”and “Python”.Wealsoseeadditionalskillsco‐occurringwiththese,including“BigData”and“MachineLearning”otherskillswithhighdemandalsolistedinthecoresetofDataScienceskillsinD1.4.

Becausetheterm“DataScience” isquitenewithasonlyrecentlybeenincludedasacategoryintheVideoLectures.NETdatabase.Thedataiscurrentlybeingupdatedtotagrelevantlectureswiththisnewcategory,afterwhichqueryingonthecategorywillreturnrelevantresults.WethereforeshowthreequeriesforexistingcategoriesinComputerSciencethatfallalsounderDataScience.Figure4.3showsthelandscapegeneratedfor“datamining”,oneofthemostfrequentlyoccurringskillsinjobpostings


across theEU (see alsoFigure4,which shows “datamining” tobe among themorepopular searchterms). We see a number of hits,with closely related terms including “databases” and “knowledgeextraction”,ataskwhich“machinelearning”supports.Theresourceselectedisatutorialon“BigData”.

Figure4.3:Thelandscapeoflecturesretrievedusingthekeyword“DataMining”,showingothercloselyrelatedterms.Theoverlaymapstothelargepointshowninthelower,right‐hand

corner‐aresourcewitharelativelylargeamountofviewscomparestoallotherhits.

Figures 4.4 and 4.5 show the landscapes for “Databases” and “Python” respectively. The lecturehighlightedinFigure4.4ison“DataMining”inthehealthindustry;otherrelatedtopicsinthelandscapeinclude“machinelearning”.Figure4.5,whichshowsoneofthemostfrequentlyoccurringtermsunderprogrammingandscriptinglanguages,“python”inthedemandanalysis(D1.4)retrievestermsdirectlyrelated to programming and also applications and other skills that employ programming, such as“machinelearning”.



Figure4.4:Thelandscapeofdatabaselectures,createdusingthekeyword'Databases'.

Figure4.5:Thelandscapeofpythonlectures,createdusingthekeyword'Python'.


TheLandscapeviewhasasitsaimtomapthequeriedlecturedatasetintoatwo‐dimensionalvectorspacetoallowplottingonthecomputerscreen.Forthisweusedifferentmachinelearningalgorithms.First,byusingthebag‐of‐wordsmodel14,werepresentthequerieddatasetasvectors.Wethenconstructaterm‐documentmatrixusingthesevectors.UsingLatentSemanticIndexing15wereducetheamountofinformationnoisethatiscontainedinthedataset.Afterthatweusemultidimensionalscalingtomapthetheterm‐documentmatrix intoatwo‐dimensionalvectorspace.Thecoordinatesarethenscaledsuchthatthecoordinatevaluesarebetweenzeroandone,whichmakesthefinalvisualisationprocesseasier. After that we add the lecture detail to corresponding coordinates, which are then used tovisualisethelandscape.ThisprocedureiswrittenusingtheQMinerdataanalyticsplatform27.

Landmarksareconstructedbyfirstrandomlydistributinglandmarklocationsaroundthelandscape.Aradiusisthensetforeachlandmarklocationandlecturepointswithinthisradiusarecollected.Oneofthecategories that thebounded lecturepointsall fallwithin israndomlyselectedandthe landmarknamesettothecategory.

When the landscape is generated thedashboard also showsan additional informationwindow (seeFigure 4.6). This is in two parts: the first shows the query information, containing the names ofpresenters, organisations and categories, the minimum and maximum number of views and thelanguagetheuserusedtoquerythedata.Thesecondpartcontainsbasicstatisticsaboutthequerieddata:thenumberof lecturesinthequerieddata,thetotalnumberof lectureviewsandthescientificcategorieswith the number of their occurrences in the queried data. Clicking on a category in thedashboard automatically queries the data using the category name, and the informationwindow isupdatedalongwiththelandscapeview.

Figure4.6:Theadditionalinformationwindow.Createdusingthe'ComputerScience'categorywithminimumnumberofviewssetto10000.

14https://en.wikipedia.org/wiki/Bag‐of‐words_model15https://en.wikipedia.org/wiki/Latent_semantic_analysis



3.1.4 TheVideoLecturesLearningAnalyticsDashboard

Wedescribe functionality for theprototype interfacebuilt for theVideoLecturesLearningAnalyticsDashboard:

● alllecture/perlecture/allviewers/perviewerperspectives,● basicstatisticsondownloads,viewsandsearches,● trendsfordownloads,viewsandsearches,● metricsforvisitorsandlectures,● thesearchcloud;● interactivesearchforparticularlecture/visitorrelatedinformation,statisticsandtrends.

Figure4.7showsthetrendforviewsontheVideoLectures.NETportalfromlate2012toJan2016.Figure4.8showsthetrendfordownloadsattheportal,whileFigure4.9showsthegraphforusersearch.Allgraphsareinteractive–theusercandynamicallyrestrictthetimeperiod(byclickinganddraggingaselectiononthegraph)–theviewwillautomaticallyadjusttothenewquery.

Figure4.7:VideoLecturesViewTrends.


Figure4.8:VideoLecturesDownloadTrends.

Figure4.9:VideoLecturesSearchTrends.

Figures4.7,4.8and4.9showtrendsgeneratedbasedonVideoLectures.NETlogsbeforetheDataSciencecategorywascreatedonVideoLectures.NETportal,andthereforeshowoveralltrendsforinteraction



with resources available from theportal. As corresponding resources are taggedwewill be able toextracttrendsspecifictotheDataSciencecategory.

Figure4.10andFigure4.11presentsmetricstablesforlecturesandviewersrespectivelyattheportal,asdetailedinthediscussionofmetricsinsection4.1.1.Toallowcomparisonagainstaspecificmetric,tablecolumnsaresortable.

Figure4.10:VideoLecturesLectureMetrics.


Figure4.11:VideoLecturesViewersMetrics.

The interface enables search and focus on a lecture/viewer of interest; Figure 4.12 presents theinformation, statistics and trend for a selected lecture, on “Deep learning in Natural LanguageProcessing”.



Figure4.12:VideoLecturesInformationTrend:focusonlectureon‘DeepLearninginNaturalLanguageProcessing’.

Figure4.13,similarly,showstheinformation,statisticsandtrendforaselectedviewer.

Figure4.13:VideoLecturesViewerInformation/Trend.

Figure4.14showsamapofvisits.AhighnumberofvisitscomingtoVideoLectures.NETportaloriginateintheUSA,followedbyCanada,ChinaandAustralia.


Figure4.14:VideoLecturesVisitStatistics.

3.1.5 SummaryByadoptingstatisticalandvisualanalysistechniquesontheVideoLectureslogs,weareabletoexplorehowviewersingeneralbehaveontheVideoLectures.NETportal.BymappingIPaddressestolocationsweobtainsomedemographicinformation‐whichcountriesandorganisationsviewerscomefrom,howtheyview,searchanddownloadmaterialstoredonthesite.WehaveidentifiedthatmostviewerscometotheportalfromtheUSA,China,UK,India,Germany,Canada,Australia,SloveniaandFrance.Thetopfivegroupsofusersbyviewcomefrom,indescendingorder:

● UnitedStates/Google‐41,822● Germany‐35,519● Iran/RasanehAvabaridPrivateJointStockCompany‐23,033● UnitedStates/Yahoo!‐21,928and20,320fromtwodistinctIPs● RussianFederation/Yandexenterprisenetwork‐18,931.

Diver et al., (2015) perform a similar mapping of IP address to location, to retrieve also furtherdemographical information, on language, employment, internet access and typical bandwidth andspeeds, to compare how the distribution of engagement and achievement to this backgroundinformation. Kizilcec&Halawa (2015) also examine correlation between geographical location anddropoutandperformance.Ourexploratoryanalysishashighlightedtheneedtocollectadditionaluserdemographicinformation;asT3.4progresseswewillcarryoutadeeperinvestigationofusersbasedonthe current andmoredetailed informationweareable toobtain, to identifyhowuserbackgroundsimpactengagement,completionandperformance.

VideoLectures.NET seesmore access from outside the EU; as part of the contribution to the EDSAproject, we aim to to attract more visitors from the EU. We therefore now publish a regular



VideoLecturesnewsletteron theEDSAprojectwebsitewitha listof recommendedvideosbasedonpopularityovertheprevioustwotothreemonths.WearealsobuildingdirectlinksbetweentheEDSAdemandanalysisdashboard(seeD1.4)andtheVideoLectures.NETportal.ThisistofeedinformationondemandandskillsanalysisintotheidentificationoffurthercategoriesinDataScience,andthereforeimprovetherecommendationofrelevantlearningresourcestomatchqueriesinthedemandanalysisdashboardforlearningmaterialonatopic(skillorskillsetofinterest)orauser’sskillprofile.

Acrossallviewerswefindthatthevideosseeingthemostinterest‐themostpopular,are:

● bythenumberofviewers:○ MITOpenCourseWare(OCW)○ tutorialsonDeepLearning○ MLSS(theMachineLearningSummerSchool)

● bytimespentwatching:○ videosonDeepLearning

● bytimespentasapercentageofvideolength:○ KDD2014(theACMconferenceonKnowledgeDiscoveryandDataMining)○ the2013W3Cworkshopon”MakingtheMultilingualWebWork”○ ISWC2014(theInternationalSemanticWebConference).

Weareablealsotomapactivityovertimeforeachviewerandeachresource(video).

VideoLectures.NET,havingbeenbuiltpriortoandindependentofEDSA,providesanindependentviewon interest inavarietyof topics includingDataScience.Building thesedynamic tools for theportalallows us tomonitor access to learningmaterial relevant to EDSA.With the definition of this newcategoryinthedatabaseandasweobtainmoredataonthisrelativelynewfieldwecanstarttorestrictour analysisof thedata to resources categorisedasDataScience andalso those in other categoriestaggedwithskillsandtermswehaveidentifiedinEDSAasrelevanttoDataScience.Thiswillallowustoexploreinmoredepthwhatcontributestopopularityofspecificresourcesandcategoriesinthefield.WewillfeedanyinsightthusobtainedintotheoveralllearninganalyticstaskasT3.4progresses,andspecifically into further categorisation of online learning material, both in the VideoLectures.NETdatabaseandbylinkingalsotorelatedcoursesandselectedcoursematerialdeliveredbyEDSA.

3.2 LearningAnalyticsTaskemployingtheEDSALearningLockerThistaskemploystheeventdatacapturedintheLMSdescribedinsection2.1,coveringcourseactivitycapturedfrom17Nov2015andupto13Jun2016.Weexploreherestudentbehaviourforallcourses,andthenforselectedcoursesandstudentsintheEDSALMS.

Similar to the analysis carried out for the VideoLectures.NET data,we examine the data from fourperspectives,correspondingtothoseinsection3.1:

● aggregatedperspectiveforallcourses● perspectivefromasinglecourse● aggregatedperspectiveofallstudents● perspectiveofasinglestudent.


Westartwitha summaryofdatacontent insection3.2.1, reportingcounts forkeydataattributes ‐eventsrecords,numberofusersoverallandperselectedcourses.Theinformationonevents,studentsandcoursesthatrecordrelativelyhighorlowactivity(peaksandoutliers,respectively),aswellaswhatatleastinitiallyappearstobethegeneralpictureisusedtoguideinitialexploration.BeyondthiswedonotcarryoutfurtherstatisticalanalysisasthereisnotsufficientdatatodateoninteractionwithcourseshostedbyEDSAforthisapproachtoprovidefurtherinsightintostudentbehaviour.Importantly,also,more detailed statistical analysis here would not paint a representative picture of activity, as highattritionisseeninallcourses,occurring,withrateshighestsoonafterenrolmentontoacourse.Thisisnotunusualinonlineself‐study‐Diveretal.,(2015)andKizilcec&Halawa(2015),amongothers,reportsimilarobservations;bothstudiesdelvedeeperintostudentbehaviourtouncoverpotentialreasonsforthisandtheimpactofstudentbehaviouroverallinsuchcoursesonachievementandcompletion.

Theanalysisprocesshereisthereforedrivenbyvisualexploratorydiscovery,feedingtheoutputofthestatistical analysis into the construction of simple interactive visual analytics modules to explorecontentinmoredepthandthusrevealROIsthatinturnpointtooptimalmethodstouseforfurther,moredetailedanalysis.

3.2.1 BasicStatisticalAnalysisProcessingthedatato feed intothevisualanalysisprocess, the followingsummary informationwasobtained:

● totaleventdatarecordsforthetimeperiod:7891● no.ofcoursesaccessed:27● uniqueuserIDs:322‐notethatasingleusermaylogintothesystemusingdifferentIDs,from

socialmediaorusingtheIDobtainedbyregisteringwiththeEDSALMS.● IPaddressesinuse:477‐asforuserIDs,asingleusermayloginfromdifferentIPs,ormultiple

usersmayrecordthesameIPaddress,forexample,iftheyusemachinesinthesameinstitution.FurtherworkinT3.4willcross‐referenceIPaddresseswithuserIDs,to:

1. mergeuserIDs2. group users by institution/affiliation and track any trends thatmight pertain to this

attribute,suchashighercompletionratesby,say,professionals/practitionersinthefieldforaspecificcourseorcoursecategory.

● total activities captured: 7196, out of the eight activity types. Of these eight, one, started,recordednooccurrences.Notethatenrolledontoisrecordedonlyforsomecoursesastheactionisnotimplementedbyallsystemshostingonlinecourses,includingMoodle.Noristheactioncaptured for MOOCS as enrolment or the equivalent is captured on the host site.The breakdown by activity type, illustrated also in Figure 4.17, listed in reverse order offrequencyfollows:

○ "viewed” ‐5562○ "loggedinto" ‐731○ "enrolledonto"‐455○ "registeredto"‐336○ "loggedoutof" ‐80○ "answered" ‐17○ "completed" ‐15○ "started" ‐0



Table4.1showscountsforeventtypesforthetop5coursesbyactivitycountandenrolments.Theseare, from the bottom, "Process Mining"16, followed by "Big Data Analytics"17, which saw relativelyintenseactivityoverthesecondhalfoftheperiod.Third,“BigDataArchitecture"18,wastheonlyothercourse,withBigDataAnalytics",thatcapturedstudents'quizresults."EssentialsofDataAnalyticsandMachineLearning"19sawconsistentactivityover thecollectionperiod,recording thesecondhighestnumberofviewsfor105uniqueuserIDs.Finally,"FoundationsofBigData"20sawthehighestactivityoverall,with1634eventsgeneratedby117uniqueuserIDsfromFeb2016totheendofthecollectionperiodinmidJun2016.WeincludealsothenumberofuniqueuserIDsrecordingeventsforeachcoursewiththeenrolmentcount;thelatterexceedstheformerassomestudentsenrolledmorethanonceontothesamecourse.Furtherinvestigationisrequiredtodeterminewhythisoccurred.

Figure4.17providesanoverviewofallactivityovertimeforallcourses,includingthoselistedinTable4.1.(NotethattheentryatthetopinFigure4.17‐“EDSAOnlineCourses”‐isthedefaultlogin/logoutpointfortheEDSAOnlineCoursesportalandnotacourseinitself.)

Table4.1:Topfivecoursesbyactivityandenrolmentfromlaunchofportalto13Jun2016.

ActivityFoundationsofBigData

EssentialsofDataAnalytics

andML

BigDataArchitecture

BigDataAnalytics

ProcessMining

viewed 1,507 1,243 739 563 575

enrolled onto(uniqueuserIDs)

127(117) 113(105) 69(64) 76(59) 36(35)

answered(quiz) ‐ ‐ 9 8 ‐

completed(quiz) ‐ ‐ 9 6 ‐

Browsing the statistical summarieswe find that student activity counts vary significantly, betweenstudents on a single course and for the same student (ID) acrossmultiple courses. The summarieshoweverstillhelpusidentifyusersofpotential interest‐oneofthemostactiveIDs,social_user_163,recorded454events21overtwoandahalfmonths,themajorityofwhichwereviews.Theremainderwere10loginsandenrolmentontocoursesincludingthetwoofthemostpopular:‘BigDataAnalytics”and“BigDataArchitecture”.

16http://courses.edsa‐project.eu/course/view.php?id=1517http://courses.edsa‐project.eu/course/view.php?id=3318http://courses.edsa‐project.eu/course/view.php?id=2719http://courses.edsa‐project.eu/course/view.php?id=2520http://courses.edsa‐project.eu/course/view.php?id=2621Notethatnodesinthevisualisationsmayoverlapwherethereisintenseactivityoveraveryshorttimeperiod.Thesesnapshotsdonotdistinguishoverlappingusingadditionalvisualencoding.Forsocial_user_163,forinstance,intenseviewingactivityoccurredovershortburstsandisthereforenotobviousintheoverviewsnapshots;howeverzoomingintotheROIsintheview(section4.2.2.2)revealsthisdetail.


Figure4.15:EventactivityforauserIDrecordingmainlyviewingactivityafterenrolmentinasub‐setofthesixcoursesinteractedwith.

Anotheruser ID, social_user_304, recorded65 events in about anhouron a singleday, across eightcourses.Afterenrollingontosixofthesecoursestheuserreturnedtothesecond‐“BigDataAnalytics”.Almosthalfoftheeventsforthisuserwereonthiscoursealone:theypagedthroughthecoursecontentandmadetwoattemptsatansweringquizzes.Theuserendedtheirsessionbyviewingcontentinthecourse“BigDataArchitecture”.

Figure4.16:EventactivityforanIDrecordingintenseactivityoverroughlyonehouronasingledayacross8courses.Thisincludedattemptingaquizinthecoursethatsawthehighest

activity.



3.2.2 VisualExploratoryAnalysisEachactivityrecordistimestampedandwitharecordoftheIPaddressanddevice/browsertypeusedtoaccesstheresource22,allowingthemappingofauser’spaththroughthesystemoveralland foracourse or activity of interest. Figure 4.17 shows the activity pattern for all courses during the datacollectionperiod.Itshouldbenotedthatasinglepointasviewedonthislayoutmayrepresentmorethanoneinstanceofthesameactivityoccurringatthesametime;wecurrentlydonotprovideadditionalvisualcuestodistinguishthese.ZoomingintoanROIhoweverprovidestheextraspacetorevealdetailforclusters.

We looknextat the interactionofmultipleusersacrossallandselectedcourses, to identifyROIs toexamineinmoredetail,fromtheperspectiveofacourseand/orauser.

3.2.3 Course‐centricPerspectiveFigure4.17highlightswhichcoursesrecordedintenseactivityoverthewholeorpartoftheperiod.Registrationandlogginginandoutarenotshownhereforindividualcoursesasthesearecapturedforthe user entering into the EDSA LMS (entry “EDSAOnline Courses” at the top). Further analysis isrequiredtolinkthesetospecificcourses,byfollowingaselecteduser’spath.

Figure4.17:Overview‐PatternsseenforeventdataforallcoursesintheEDSALRM‐eachpointrepresentsatleastoneoccurrenceofthecorrespondingeventforacourseperdate.

Selectedeventsforthecourse‘BigDataArchitecture’arehighlighted,withdetailincludingtheusertriggeringitinthepopup.

Somecoursesseefairlyregularburstsofactivitythroughoutthecollectionperiod.Regardlessofdensityof activity this bursty pattern recurs, with periods of up to several dayswith no interaction.MoredetailedanalysisbeyondM18willidentifypotentialcausesofthesepatternsandthepotentialimpactofcoursestructure(includingstaggeredreleaseofmodules),dayoftheweekorholidayperiodssuchasChristmasandNewYearthatseedowntimeacrosswideregionsoftheworld.

22FurtherinvestigationofIPaddressesandaccessdevicewillbecarriedoutinthesecondhalfoftheprojecttoobtainmoreinformationonstudentdemographics(seealsosection6.1).


Vieweventsmaybefurtherbrokendowninto

● coursemoduleviews○ url,e.g.,pointerstootheronlinematerialsuchaswebinarsandcoursefeedbackpages○ (coursemodule)page,e.g.,pointerstosyllabi,extrainformationonthecourse,webinars○ (coursemodule)resource‐thecomponentsofacourse,suchaspagesaddressingcourse

topics● discussionfora

○ discussionsonaspecifiedtopic○ discussionforumviews

● feedbackpages● quizzes

○ views○ attempts.

Wefocusinthisanalysisonviewedactivityeventsindiscussionforaandtodowithquizzes.Thelatterdifferfromansweredandcompletedeventsexplicitlyrecordedforquizzesandwhichprovidedetailonquizquestionsandresults.Whereaviewedeventconcernsquizzes,however,thisdoesnotincludeanydetailbeyondthecoursenameandinsomecasesamoreinformativequizlabel.Wedistinguishbetweentheseandquizviewedeventsbyusingacrosswithabrokenborderinpinkandredforquizviewsandattempts respectively (a small pink circle is used for viewed and a red cross for answered eventsrespectively).(AppendixAcontainsasnapshotoftheresultsobtainedforaqueryfollowinganameduser,withexamplesofthelevelofdetailonquizinformationforthethreeeventtypes.)

Figure4.15illustratesthisadditionalencodingfortheuserwithIDsocial_user_163.

3.2.4 FocusonaSingleUserToallowamoredetailedinspectionoftheactivityofsocial_user_163(overviewinFigure4.15)followthepathofthisuser,oneofthemostactive,forthecollectionperiod.Welookinmoredetailatotherselectedusersinsection3.4.1,toexaminefurtherpatternsofbehaviour.

Wezoomfirstintothestartoftheperiodofactivity(20Mar)torevealdetailontheactivityhere.Figure4.18narrowsdowntothefirstclusterseen,overthefirstfourdays.Thisrevealsfiveclusterseachwithadistinctlogin/outaction,andseveralquizviewsandattemptsforthecourse"ProcessMining"inthefirstburst.Activityforothercoursesfallsafterthisperiod.

Figure4.18:zoomingintothefirstfourdaysofthecollectionperiodforoneofthemostactiveusers(seeFigure4.15)wherequiteintensiveactivitywasrecorded.Theredarrowpointstoa

quizviewandtheolivetoattemptsatansweringquizquestions.



Thereisactivityononlyonecourse‐“ProcessMining”;wethereforefilteroutallothercoursesaccessedby social_user_163 and zoom in further to the first twodays, and then the first fourhours (top andbottom respectively, Figure 4.19). This shows, after enrolment, quick views of a number of coursemodules,withtwoshortandtworelativelylongerpauses,whichmaybedetailedreadingbeforemovingontoanothermodule.Shortlyafterthefourthbreak,thisuserappearstobrowsethroughandattemptaseriesofquizquestions(thebrokenbordersonthecrossiconsindicatethesearecapturedasquizviewedeventsratherthanansweredevents).Arelativelylongpausefollowsviewsofquestionsinthequiz'DataMining',followedbyseveralfurtherattemptsatquestionsinthisquiz.Anotherpausefollowsafterviewingofaquizquestionbeforetheuserlogsoutofthesystem.

Figure4.19:Zoominginfurtherintothefirsttwodays,thenthefirsttwohoursofactivityforsocial_user_163torevealdetailonactivityduringthelatterperiod.

Afterthisburstofactivitytherearefourfurthershortviewsofcoursematerialoverthefollowingfourdays.There isnofurtheractivity forthenextthreeweeks,afterwhichsocial_user_163viewsfurthermaterialin'ProcessMining'andenrolsontothreeofthefourmostpopularcourses(seeoverviewinFigure4.15).

Nofurtheractivityisrecordedforanotherfiveweeks,afterwhichthisuserviewsadditionalmaterialin"Process Mining". After entering its discussion forum social_user_163 briefly views material in the"ProcessMiningMOOC:DatascienceinAction"butdoesnotengageanyfurtherwithit.Afterabreakofapproximatelyafortnighttheuserreturnstoviewadditionalmaterialin"ProcessMining",withthelastrecordonthe4thofJune(forthecollectionperiodending13Jun).

3.2.5 SummaryBycarryingoutsimplestatisticalanalysisduringdataprocessingwewereableto,withtheaidofthevisualoverviewofthecompletedataset(inFigure4.17),identifyinitialROIstoexploreinmoredetail.

The exploratory visual analysis showsone recurring trend: even for themost active andpersistentstudents:periodsof intenseactivity followedby longbreaks.Different factorsmayaccount for this,includingdefinedcoursestructureandthestaggeredreleaseofmaterialtypicallyusedtostructureself‐studyinonlinecourses.Additionalpotentialfactorsincludedifficultywithelementsinthecoursethatmayrequireadditionalstudy,aswellasother(external)factorsmoredifficulttoidentifyand/orcontrol,


suchasdemographicsandcommitmentsoutsidestudy.Somestudies,e.g.,Diveretal.,(2015),linkdelaysinaccessingcoursematerial(onceavailable)todropout.Kizilcec&Halawa(2015)examinetheimpactofgreaterdifficultyaccessinghelp fromotherstudentsandinstructors inonline learningon,amongothers,poorresultsinorincompleteassessment(quizzes,assignmentsandexams).Wetakeacloserlook (in section 3.4.1) at access to discussion fora to discoverwhat correlationmay exist betweeninteractionwiththeseforaandcontinuedengagementwithacourse.FurtherworkinT3.4,asweobtainalsoinformationoncoursecompletion,willinvolvefurtherinvestigationtoconfirmwhethertheseandotherfactorscontributetothepatternsobservedandwhatimpacttheyhaveonstudentperformance.

A second frequently occurring pattern is short periods viewing course material after enrolment,followedbyabandonmentof the course (Figure4.26highlights thispattern forone course, and theanalysiscarriedoutinsection3.3usingprocessminingreportssimilarfindings).Thismaybeattributedtotheneedtoviewcoursematerialtodeterminesuitabilityorconfirminterest‐somecoursesdonotallowbrowsing before enrolment. Further investigation of thematerial viewed in such instances isnecessarytoconfirmthatthisisthecaseand/orwhatotherfactorsinfluenceearlydropout.Thefindingsfromthismoredetailedinvestigationshouldfeedintoguidingstudentsinthefirstinstanceinselectingcoursesalignedtotheirneedsandinterests.

3.3 LearningAnalyticsTaskemployingProcessMining

Processminingprovidesasetofalgorithms,toolsandtechniquestoanalyseeventdata[vanderAalst2011]. Threemain perspectives offered by processmining include process discovery, conformancecheckingandprocessenhancement.Discoverytechniquesallowtheenactmentofprocessmodelsfromlog data. Conformance checking attempts to verify conformity to a predefined model and identifydeviations, if any. Enhancement provides for models to be improved based on results of processdiscoveryandconformancechecking.

Giventheonline‐basedformatandnatureofMOOCs,itispossibletotrackstudentactivitiesbyfollowingtheindividualclickstheymakeoncoursewebpages.Thedatageneratedcangiveusinsightintohowandwhenstudentsfollowlecturesandhowtheyprepareforexams.WeanalyseaMOOChostedontheCourseraplatform‐“ProcessMining:DataScienceinAction’”deliveredbyTUEindhoven.

3.3.1 Preprocessing:BuildinganEventLogWeareinterestedinanalysingstudentbehaviourbasedonthetrailsofclickeventstheygenerate.Beforewecoulduseprocessminingtoanalysethisbehaviour,wefirstneededtomaptheMOOCdatatoaneventlog.Therearetwothingsweneededtospecifyforthismapping:whatconstitutesanevent,andwhatmakesacase(i.e.,asequenceofevents).

Aswearefocusingonthebehaviourofstudents,weconsidereachstudentasanindividualcase.Theclickstreamastudentgeneratesisthebasisfortheevents inthiscase.Thishighlightstheseparationbetweencaseandevent.Fortheanalysisreportedherewewillprimarilyfocusonactioneventsoftypepageview.

WefilteredtheMOOCdatatoobtainaviewofthelecturewatchingbehaviourofstudents.Foreachcase,westoredthedataavailableaboutthestudent,includingtheircoursegradedata.Foreachclickstream



event,wecreatedaneventbelonging to the correspondingcase (basedon the studentuser id)andstoredthiswiththereferencelectureastheeventname.

Basedondifferentdataattributes,wecandetermineseveralstudentsgroups.First,wegroupstudentsthatfailedthecourseorsuccessfullyobtainedacertificate,withthelattersplitintoanormalcertificateor onewith distinction. The second grouping attribute iswhether or not a student enrolled in thesignaturetrack.

3.3.2 VisualisationofLearningBehaviourIn this section,weuse thedotted chart and aprocessminingdiscovery algorithm (FuzzyMiner) tovisualisetheeventdataanddiscovertheactuallearningprocess.TheaimistovisuallyprovideinsightsontheoverallMOOCandprofilestudentbehaviourthroughoutthedurationoftheMOOC.Weconsiderthree important dimensions in this analysis: the general lecture videos viewing habit, the quizsubmissionbehaviour.TheseinsightscanhelptounderstandhowstudentsstudyandwhatimpactsuchbehaviourshaveontheirinvolvementintheMOOC.

VisualisingViewingBehaviour:wemakeuseofthedottedchart(seeFigure4.20)tovisualisethepathfollowedbystudentswhileviewingvideos.Thisprovidesabroadrepresentationofstudentwatchingbehaviourthroughoutthecourse.

Figure4.20:DottedChartdepictinggeneralviewingbehaviourthroughoutthedurationoftheMOOC.

InFigure4.20thedottedchartdepictstheviewingbehaviourforallstudentswhoregisteredfortheMOOC,focusingonwhenandhowtheywatchvideos.Thex‐axisdepictsthetimeexpressedinweeks,whilethey‐axisrepresentsstudents.Sevendifferentcoloursrepresentdifferenteventsatagivenpointintimeascarriedbystudents.Thewhitedotsshowthetimingwhenstudentsviewedmiscellaneousvideos(twovideosoncoursebackgroundandintroductiontotools).AllvideosforWeek1aredepicted


withbluedots,greendotsrepresentvideosforWeek2,graydotsshowthedistributionforvideosinWeek3,yellowdotsshowlectureviewsforWeek4,Week5videosareseeninredandWeek6lecturevideosaredepictedbydarkgreendots.Onecannotethatthevideosfromweek2areneverwatchedinweek1;thisresultsfromtheorganisationofthecoursewebsite,wherestudentscannotseethesevideosinweek1(andsimilarlyforlaterweeks).

Looking at Figure 4.20 we observe that a significant number of students drop out throughout thedurationofthecourse.Also,manystopwatchingafterthefirstweekandabout50%ofstudentsdropoutbytheendofthesecondweekofthecourse.Further,notallstudentswatchthevideosinsequence:althoughallwatchWeek1beforewatchingWeek2,eventowardtheendofthecoursewestillseemanywatchingWeek1videos.Thisindicatesthatsomevideosarewatchedrepeatedlyandthatanumberofstudentsjointhecourselate.Thisrepresentsaninterestingpattern‐wherestudentsjoininglatemaybe looking at selected coursematerial, and/or studentswho joined at the start are reviewing somematerial.Thispatterncouldbeinvestigatedbydelvingmoredeeplyintoanalysisofstudentbehaviour.

Inordertogetdetailedinsightintothistrend,wealsogroupthestudentsintosub‐groupsbasedontheirprofiles.Wemakethisclassificationbasedontheirfinalperformance(distinction,normalandfail)andthe type of the certificate they sign up for (signature or non‐signature track). We consider onlydistinctionstudentsonthesignaturetrackasanillustrationinFigure4.21.

Figure4.21:DottedChartforDistinctionstudentsonSignatureTrack.

InFigure4.21,thesestudentsfollowasequentialpatternastheywatchthevideos.SomejoinalittlelateatWeek2orWeek3,butthegeneraltrendremainsthatmostofthemwatchvideossequentiallyastheyaremadeavailable.Thiscanbeseenbylookingatthedemarcationimposedbyrespectivelecturevideocolour.ThisisalsocapturedbytheprocessmodelsdepictedinFigure4.22.Successfulstudentstypicallyfollowvideossequentiallyorwithorderlyloops(therepetitionofasetofvideosintheirintendedorder)while unsuccessful students appear to be volatile and unpredictable in theirwatching pattern (thevisualisationofthisprocessisexemplifiedbythemodelshownontherightsideofFigure4.22).Welooknextattheprocessmodelsdepictingbothsuccessfulandunsuccessfulstudentlearningpaths.



ProcessDiscovery:entailslearningaprocessmodelfromtheeventlog.Onecanmakeuseofaneventlogasinputtoanumberofprocessminingalgorithmsinordertovisualiseandenacttherealbehaviour(sequentialsteps)ofstudents.Weusethefuzzymineralgorithm[Gunther&vanderAalst2007]tomineourdataset.Outofallstudentprocessmodelsweconsiderforillustrativepurposestwoextremes:thedistinctionstudentsonthesignaturetrackandfailingstudentsnotonthesignaturetrack.TheresultingmodelsaredisplayedinFigure4.22.

ThemodelsinFigure4.22indicatethatdistinctionstudentstendtohaveamorestructuredlearningprocess,withasinglepathwherepossibleloopsarehighlighted.Onthecontrary,failingstudentsfollowanunstructuredlearningprocessthatexemplifiesthevolatilityandunpredictabilityoftheirlearningpatterns.Althoughthefuzzymineronlyshowsthemostdominantbehaviour,Figure4.22stillshowsthattherearemanyalternativepathsthroughacourse.

Figure4.22:ProcessModelsforSignatureTrackDistinctionStudentswithfew“loopbacks”(left)vs.Non‐SignatureTrackFailingstudentswith“loopbacksanddeviations”(right).

3.3.3 QuantificationofLearningBehaviourWe perform conformance checking using the normative model in Figure 4.23 to analyse viewingbehaviourforallsub‐groupsofstudents.Byexploringthealignmentdetailsweareabletoextractdetailsabouttheoverallwatchstatus(momentatwhichavideoisseen,suchas“early”)andviewinghabits.Wemakeuseofthealignmentonthemodeltounderstandwhetheravideowaswatchedearly(earlierthanintheintendedorder),orlate(laterthanintheintendedorder)–orregularlywatchedasintended.Wealsousetemporalinformationtostudyviewinghabits(daily,weekly,inbatches,etc.)


Figure4.23:Normativemodel.

WatchStatus:Figure4.24givesaviewontheoverallvideostatusforbothsignatureandnon‐signaturetrackstudents.Thisindicatesthatsuccessfulstudentsappeartobemorecommittedtowatchingvideosthanunsuccessfulstudents.Wenotehoweverthatunsuccessfulstudentsalsofailtowatchsomevideosbecause they would have dropped out before some resources were made available. Further,demarcationisobservedbetweensignature‐trackandnon‐signaturetrackstudents‐theformershowmoresignsofcommitmenttowatchingvideosacrossthedifferentsub‐groupsincomparisontotheirnon‐signature track counterparts. Successful students with the signature track certificate watchedregularlymorethan80%ofthevideos,whilethosewhofailedwatched45%ofthevideos.Thenon‐signaturetrackstudentswhoweresuccessfulwatchedonaverage80%ofvideosregularly,whilethestudentsinthisgroupwhofailedwatchedonly15%ofthevideos.ThesetrendsareshowninFigure4.24forthesignaturetrack(top)andnon‐signaturetrack(bottom)students.



Figure4.24:Overallwatchstatusfortheentirestudentpopulation,SignatureTrack(bottom)andnon‐SignatureTrack(top).

ViewingHabit: The viewing habitmeasuredescribes the time commitment in the student learningbehaviour.Figure4.25indicatesthatforthemostpartsuccessfulstudentswatchvideosmoreinabatchanddonotwastealotoftimebetweenvideos.Unsuccessfulstudentsskipmorevideosthantheywatch.


Figure4.25:Overallrepresentationofviewinghabits,SignatureTrack(bottom)andNon‐SignatureTrack(top).

3.3.4 SummaryTakingourCourseraMOOCasacasestudy,weshowtheaddedvalueofprocessminingonMOOCdata.Ourresultsdemonstratethatthewaystudentswatchvideosaswellastheintervalbetweensuccessivelywatched videos are related to their performance. Results indicate that successful students follow asequentiallystructuredwatchingpatternwhileunsuccessfulstudentsarelesspredictableandwatchvideos in a less structured way. Moreover, student learning behaviour can be described from twodimensions:watchstatusandviewinghabit.Ingeneral,studentviewinghabitsaredeterminedbythetimebetweensuccessivevideoswhilethewatchstatusisdeterminedbyalignmenttothenormativemodel.

Amoredetaileddescriptionoftheresultscanbefoundin[Mukalaetal.,2015].TheimplicationsoftheseresultswillbestudiedinfurtherdetailasT3.4progresses.Itappearsthatmanystudentsprefernottofollowtheintendedorderofacourse.Thispresentsapossiblepathforimprovementofcoursedelivery:tryingtocatertotheneedsofstudentsforwhomthepathprescribedbytheinstructordoesnotseemrelevant.

3.4 FocusontheEDSAOnlineCourse“FoundationsofBigData”

AnalysisoftheeventdatacapturedtodateintheEDSALRMshowedactivitythatisattimessparseandatothersextremelydense,lookingatalldata,individualcoursesandindividuallearnerIDs.Onecoursegenerated enough events to allow analysis using all three approaches reported in this section:“FoundationsofBigData”.Wethereforetriangulatetheindependentanalysiscarriedoutontheeventdataforthiscourse,toprovideasinglefoundationonwhichtobegintobuildaframeworkforlearninganalyticsinEDSAandprovidepointers,andultimately,benchmarksthatwillcontributetothestateoftheartinthefield.

SampledatafromtheeventlogsforthiscoursecanbefoundinAppendixB.



3.4.1 Statistical&VisualAnalysisTheright‐handsideofFigure4.26showsthetrendseenforthetwoactivitiesrecordedforthecourse‐enrolmentontoacourseandviews(coursemodulepages,discussionforaandfeedbackforms).Fromtoptobottom,usersareorderedbydateenrolled,followedbytotalactivitycount.Thetwosnapshotsontheleftshow,respectively,eventdatathe20mostandleastactiveusers(bytotalactivitycount).Aninteresting observation is that users who persist beyond their initial session very often accessdiscussionforasoonafterenrolment(onaveragethehighestoccurrenceofthisactionacrossallusers),andagainfurtherintothecourse.

Thefirsteventrecordedforthecoursewasaview(bysocial_user_5)on16Dec2015,afterwhichnomoreeventswererecordedtill02Feb2016(verticaldelineator).Incidentally,thispointfallsshortlyafterformalpublicisingoftheEDSAonlinecoursesportalattheendofJan2016.Furtherinvestigationoftheinteractionpathofsocial_user_5andotherswhoaccessedtheportalpriortothisisnecessarytoidentifyanydifferencesinbehaviourbeforeandafterthispointforothercourses.

Beyond02Feb2016enrolmentandviewsgraduallyincreasedthroughtotheendofthedatacaptureperiodon13Jun2016.

Figure4.26:Overviewoftheeventactivityforthe117uniqueuserIDsforthecourse“FoundationsofBigData”,focusingalsoonthe20mostandleastactivelearnersrespectively.

There is no data onquizzes for this course;we dohowever see some interactionwith the course’sdiscussionfora(thearrowsinFigure4.27pointtoasub‐setofthenodesforvieweventsassociatedwithdiscussionfora),veryoftencoincidingwithenrolmentontothecourse.Theonlylabelleddiscussionforthiscoursewas“Greetings",withdiscussioncontentdescribedas“AMoodlediscussion”.Furtherforumactivityfellunderthetopic“Coursediscussionforum"‐describedsimplyas"Amodule"discussion.A"Generaldiscussionforum"fortheEDSAportal(ratherthanaspecificcourse)asksstudentstojointhediscussionforumtoobtainmoreinformationonDataScienceandEDSAcourses.Aseconddiscussionlabelled "EDSAcourses"anddescribedas "AMoodlediscussion"also fallsunder thegeneral courseportal.


TheLMSdoesnotprovidefurtherinformationondiscussioncontent.Weaimtoinvestigatefurtherthekindsofdiscussionsinthe“Coursediscussionforum",aswellasotherspecificdiscussionsstarted,todeterminewhatadditionalimpactboththesocialnatureoftheactivityandtheactualdiscussionshaveonengagementandperformance.

Figure4.27:FlatteningthelayoutontherightinFigure4.26toillustratetheburstyinteractionseenacrossallcourses.Thearrowspointtorecordsofinteractionwithdiscussionfora.

Figure 4.15 (section 3.2.1) shows the activity recorded for all courses which saw activity bysocial_user_163,themostactiveuseronthiscourse‐seeFigure4.28,whichzoomsintothesnapshotinthetop,left,Figure4.26.Figure4.29focusesonactivitybysocial_user_163andsocial_user_20,thefirsttoenrolontothecourse(“FoundationsofBigData”).

Figure4.28:Activitytrendsforthetop20mostactivelearners,orderedfirstbyenrolmentdate(top)thenbytotalactivitycount(bottom),highlightingthetopuserineachcase.

Weseeagainactivitybyuser inshortbursts;someusersreturntothecourseafterabreak,butthemajorityappeartodropoutaftertheinitialinteractionperiod.(SeealsoTable4.2,whichshowsthetotalactivitycountforthe30mostactiveusersforthiscourse,withveryquickdropoffininteraction.)

Lookingagainatsocial_user_163wesee(inFigure4.29)intenseactivityimmediatelyafterenrolment,followedbytwoshorterspellsofactivityhalfwaythroughthe36hoursthisuserspentonthecourse.Social_user_20,thefirsttoenrolontothecourse,recordsactivityover2days,butforonly22(more)views,mosttowardtheendofthattime.



Figure4.29:Focusonsocial_user_163andsocial_user_20‐seealsoFigure4.28.

Mostuseractivityoccursinshortburstsoverashorttotaltimespan‐minutestohourstoafewdays,asseenforeventheactiveusersinFigure4.29.ThethreeusersinFigure4.30breakthistrend,spanning,forthelongest,threemonths,andwithoneandahalfmonthseachfortheothertwo.

Figure4.30:Focusonsocial_user_94,social_user_131andsocial_user_137,whoseactivityspannedlongerperiodsthanaverage.

Wemodifysortordertolargestnumberofeventsforthiscourseonly‐theresultsareshowninTable4.2 and Figure 4.31 for the top 30 IDs. Activity drops away quickly from from 108 events forsocial_user_151 to under 20 for the last eight. Social_user_163 drops to third position, aftersocial_user_94. These users, especially social_user_151 engage with the course discussion fora, inadditiontoviewingcoursematerial(seeFigures4.32and4.33).Asisthenormengagementwiththecourseisclusteredintoaseriesofshort,relativelyintenseactivityeventsforuserssuchasthese,whopersistbeyondtheinitialenrolmentandmoduleresourceviews.

Table4.2:Fromthemostactive,thetop30mostactiveusersonthecourse"FoundationsofBigData‐activitycountdropsfrom108tounder20forthelasteight.

UserID Totaleventsfor"FoundationsofBigData"

UserID Totaleventsfor"FoundationsofBigData"

social_user_151 108 social_user_91 26



social_user_163 61

social_user_84 24









oodle_user_5409 32 social_user_1 17






Figure4.31:Activitytrendsforthetop30mostactivelearnersinthiscourseonly,orderedbytotalnumberofeventsperuserIDforthiscourse.ThechartatthetopshowstheexponentialdropinactivityforthedatainTable4.2,whilethebottomdetailsactivitytypefortheseusers.

Weseeonecluster inthecentreofFigure4.31fortheactivityofsocial_user_151 ‐zoomingintothisregionatthetop,Figure4.32,showsactivityinthreeburstsoveralmosttwodays.Thefirsttwoclusters(middle)showinitialenrolmentfollowedbyaviewevent.Afterabreakofalmostninehourstheusertriggersanotherviewevent.Afteranotherpauseofaboutadayweseemostoftheactivityofthisuser(bottom),includinginteractionwiththecourse’sdiscussionforaandenrolment(forasecondtime)ontothesamecourse,morethanhalfwaythroughthissession.

Figure4.32:Zoomingintothefirst(middle)andlast(bottom)ofthreeclustersforsocial_user_151,themostactiveuserforthecourse.

Whilewithathirdfewereventsrecordedthanthemostactiveuserforthiscourse,social_user_94breaksthetrendbyengagingwiththecourseovertwomonths.Zoominginfromtheoverviewforthetop30users(inFigure4.31)toactivityforonlysocial_user_94,Figure4.33showsthenorm,burstyinteraction.Weexaminethefirstfewdaysofinteraction(toptwo,withincreasingzoomintothestart),thefourth


andthefinalclusterofevents.Inadditiontocoursemoduleresourcesweseefairlyregularinteractionwiththediscussionfora.

Figure4.33:Zoomingintotheclusterstorevealdetailforthesecondmostactiveuser,social_user_94,forthecourse"FoundationsofBigData".

Inadditiontothemethodsusedaboveforvisualanalytics,byanalogytotheanalysiscarriedoutusingtheVideoLecturesLearningAnalyticsdashboard (section3.1) and to aid linking of theoutcomes ofanalysiswecreatedatoolforanalysingtheEDSALearningLockerlogs.

Figure4.34presentsa visualisationof activityover time for the courseof interest, theEDSAonlinecourse“FoundationsofBigData”.

Figure4.34:Activityovertimeforthecourse“FoundationsofBigData”.



Figure4.35showsmetricsforanumberofEDSAonlinecourses,including“FoundationsofBigData”.ThegraphinFigure4.34andthemetricshereconfirmthatthisisthemostpopularcourse(discountingthat for theentry to theportal labelledas “EDSAOnlineCourses”),withover1700actionsand161viewerstodate,andonaverage11actionsperviewer.

Figure5.35:CoursesMetricsforanumberofcoursesavailablefromtheEDSACoursesportal,includingourfocus‐“FoundationsofBigData”.

3.4.2 ProcessMiningFinally,weapplyprocessminingtothesamecourse.Thedatastudiedisextractedfromasetofpagevisitsfromthecoursesportal1ontheEDSAprojectwebsite,eachvisitcomprisingtheURLofthepagevisited,accompaniedbytheIDoftheuserandthetimestampofthevisit.

Aneventhereisdefinedasapagevisit.Aresourceisoneofapdffile,avideooraquiz.Outoftheevents,wediscardvisitstothehomepageoracourselistpage,keepingexclusivelythevisitstoresources.Wealsodiscardeventsfromtheusers“admin”,“demo”or“guest”.Finally,eliminatingthe36userswhovisitedonlyoneresource,121usersremain.Thesespentamediantimeof25minutesbetweentheirfirstandlastpagevisitedonthewholewebsite.Thislowvaluereflectsthefactthatmanyusersjustbrowsethroughafewoftheresourcesbeforeendingtheirvisittothecoursesectionofthewebsite;theupperlimitforthetimespanonthiscoursetothispointisthreemonths.

“FoundationsofBigData”wasfoundtobethemostpopularcourse‐bynumberofevents,enrolmentandaccessofresourcesacrossthecourse.Allresourcesinthiscoursewereaccessedatleast10times.We seemuch lower coverage and overall access in all other courses in theEDSAportal,where themajorityofresourceswereaccessedlessthansixtimespercourse.“FoundationsofBigData”istheonlycoursethatcurrentlyprovidessufficientdatatoobtainmeaningfulresultsusingprocessmining.Toaidhumaninterpretationofourresultswematcheacheventtoitscorresponding(human‐readable)name,e.g.,aURLtothetitleofacoursemodule.


3.4.3 NormativemodelForuserswhovisitedatleasttworesourcesinthecourse,27%arerepeatvisitstothepreviousresourceaccessed.82%ofeventchangesaretothenextexpectedevent,asprescribedbycourseorder/structure.Wealsoobservethatusersdropoutof thecourseatanytime,except in themiddleof latechapters(whiletheycouldifsotheywanted).

Figure4.36showsthemodelbuiltmanuallytomatchthepatternsobserved,whichfollowsthepathprescribedbythecoursedesignerorinstructor.Themodelislinear,withtheeventsorderedasexpectedforthecourseonthesecondline.Theleftmostcirclestartstheprocess,andthelonecircleatthebottom(fourthline)terminatestheprocess.Therectanglesonthefirstlineeachrepresentoneresourceinthecourse,anyofwhoseeventsmayberepeated.Thegreyrectanglesonthethirdlineallowtodropoutthecourseatanytime,leadingtotheendoftheprocessonthefourthline.

Figure4.36:Modelillustratingpatternsobservedinthecourse“FoundationsofBigData”.

3.4.4 AnalysisUseractionsmatchwelltheexpectedmodel.Inthedetailedview(Figure4.37),thegreenandpinklinesatthebottomofeachstatenoderepresenttheproportionofeventstakingplaceasexpected(green)vs.eventsnottakingplaceasexpected(pink).Wherethereisnogreen/pinklinealleventstookplaceasexpected(seetheeventsonthefirstlinethatrepresentrepetitions).Anotherindicationofdifferencebetweenthedataandthemodelisthesizeoftheyellowcircles:whentheyarelarger,moreirrelevanteventstakeplace.Thepinklinerepresentseventsthatareinthemodelbutnotinthedata,whiletheyellowcirclesrepresenteventsthatareinthedatabutnotinthemodel.

The first event, the course syllabus (first rectangle on the second line) shows an exception to ourprediction:themajorityofusersskipthesyllabus.Weseethereforeapredominantlypinklineatthebottomofthisevent.Asecondexceptionisthelargeyellow(third)circleonthesecondline:manyeventsfollowingviewingofthefirstelementofthecurriculumarefollowedbyanarbitraryfurtheritem,whichmayrepresentthewishofuserstohaveanindicationofwherethiscourseisgoingbeforegoingonorgivingup.

Therestofthenormativemodelisfollowedasexpected.Standardmetricsforthequalityofthemodelreportprecisionas0.7,andthefitnessas0.9.



Figure4.37:Detailofthemodelforthecourse“FoundationsofBigData”,showing,largely,conformancetocoursestructureasprescribed,withtwoexceptionstotheruleatthestartof

theprocess.

Wediscussinsection4theresultsobtainedforthiscourseandfortheotherdataononlinecoursesanalysedinthissection,andpointtofurtherrequirementsforthedetailedanalysisrequiredtoobtaindeeperunderstandingofstudentbehaviour,potentialcausesoftheseandhowwecanfeedourfindingsintoimprovingthestudentexperience.

.


4. Discussion

4.1 InitialOverviewsofStudentOnlineLearningBehaviour

Investigating student behaviour for the data collected for three independent sets of online coursematerialbeforehoningintoasinglefocus‐themostpopularcourse(byeventcountandenrolment)availablefromtheEDSAOnlineCoursesportal,usingthethreedifferentapproachesandperspectives,provides broader insight into user actions. The first two analysis approaches provide overviews ofstudent behaviour from the perspective of the student accessing the material. The third approachinvestigates behaviour from the perspective of the course as prescribed in the curriculum, beforeoverlayingactualbehaviourontopofthis.

ThedifferentperspectivesthusobtainedinthispreliminaryanalysisrevealdifferentROIs,eachofwhichtriggermoredetailedinvestigationthatshouldhelptounderstandbetteruserbehaviourandpotentialreasonsforactionstaken‐whyacourseisinitiallyaccessed,whyauserislikelytodropoutandwhen.

TheCourseraMOOCandtheEDSAOnlineCoursesportalrequirestudentstocommittoanumberofstudysessionstocompleteacourse.Thesecoursesallrequireviewingand/orreadingoflectureandsupplementary learningmaterial, some additionallywith requirements for completing assignments,quizzesandinsomecases,examstobothcompleteandsuccessfullypassthecourse.TheVideoLecturesportalontheotherhandallowsuserstoselectandviewsingle,independentlectures,tutorialsandotherlearningmaterialacrossabroadrangeorwithinaspecifiedcategory(ies).

Ouranalysisstartswiththegenerationofbroadoverviewsofinteractionwiththematerial,lookingattopicsthatattract interest,whatwithinthebroadertopicor individualcourseisaccessed,andwhatotheractionsuserscarryoutduringtheirstudysessions.Informationonuserdemographicsislimited,duemainlytoprivacyissues;furtherworkwillseekwaystocapture,withoutviolatingethicsorprivacy,anonymous or collective information on user backgrounds, in order to feed this into identifyingpotential (external) triggers for student actions that negatively impact their ability to successfullycompleteacourse.

Fortheself‐studycourseswefind,overall,enrolmentoccurringthroughtothefinalstagesofcourses,butwithhighattritionsoonafterindividualstudents’initialinteractionwithacourse‐atrendseeninonlineself‐study[Diveretal.,2015;Mukalaetal.,2015].Dropoutcontinuesascoursesprogressbothforearlyandlaterentrants;however,wherestudentspersistthroughtothelaterstagesofacoursetheyaremorelikelytocompleteit.Interactionwithcoursematerialgenerallyoccursindistinctbatches,withperiodsofnoorverylittleactivityseenthroughoutallcourses,eventhemostpopular.Thismaybeduetostaggeredreleaseofcourseinformation;wewillinvestigatethisfurthertoidentifyothercontributingfactorsandanyimpactthismayhaveoncompletionandsuccess.Commontoallthreesetsoflearningresourcesandonlineself‐study,however,istheverylowbartoandcostofenrolment,especiallywhencomparedtof‐fcourses;relevantstudiesshowthatthismayalsocontributetothethehighdropoutrate,especiallywhereothercompetingfactorssuchastimeandadditionalcommitmentsofstudentsoverrideeveninterestandadesireforself‐improvement[Diveretal.,2015;Kevanetal.,2016].Thisis



reflectedalsointhecomparisonofinteraction,performanceandcompletionofthe(paying)signaturetrackvs.(free)non‐signaturetrackstudentsinsection3.3.

Wediscussinmoredetailinsection4.2ourfindingsforasinglecourse‐thatattractingthemostinterestin the EDSA Online Courses portal. Triangulating the results obtained from the different analysisapproachesbothrevealsinsightthatasingleperspectivealonemaynot,andhelpsustoidentifyfurthertechniquesthatwillaugmentouranalysis.

4.2 TriangulationofResultsforEDSACourse“FoundationsofBigData”

Ourfindingsindicatethatusersfirstenrolontoacoursetoaccessitscontent,andtheninitiallybrowsecoursematerial,sometimesasprescribedbythecoursestructure,orbydippinginandoutofanumberof coursemodules, seemingly to get a feel for the course. Further analysis of individual user pathsconfirmed the initial statistical analysis that indicated variation in user activity. After the initialexplorationofcoursematerialweseebehavioursplit intomaintwocohorts‐dropoutorcontinuedinteractionwithcoursematerial(albeitwithfurtherdropoutdowntheline)‐seeFigures4.20and4.26.Onaverage,fairlyshorttotalinteractiontimeisseenperuserforacourse,evenforthemostpopular.Thishoweverincreaseswhenconsideringonlyuserswhopersistbeyondtheinitialinteractionperiod,fromtimespansofafewminutestotwohoursormoreinasingledayorsession.Overall,userstypicallyaccesscoursematerialusingbatchmodeandsequentially‐aspertheprescribedorderforthecourse.Again,where students continue to access coursematerial beyond the initial interaction session, theburstyorbatchaccessmodepersistsoverseveraldays,andasanexception,touptoafewmonthsasseenfortheuserswithIDssocial_user_94,social_user_131andsocial_user_137(seeFigure4.31).

Beingonline,self‐studycourses, interactionwithotherstudents isnaturallyrestricted.Weseesomeinteractionwithdiscussionforaatdifferentpointsintimeforallcourses;studentswhoaccessedthediscussionforafelltendedtobethoserecordingrelativelyhighertotalactivitycountandinteractiontime on the course. This reinforces potential for inter‐student and student‐instructor discussion tomotivate retention. Kizilcec&Halawa (2015) found similar patterns of activity. Diver et al., (2015)further found in their largescale studyMOOCsa linkbetweendropout rateand lower frequencyofaccesstodiscussionfora,notingalsothepotentialforadditionalbenefitinthesocialactivity.Furtheranalysis in T3.4will examinewhat impact of this interaction,with its added social element, has onretention, final coursegrades.Wewill also,wherewe are able to access the contentofdiscussions,investigatehowthismaybeusedtodeterminewherestudentsencounterdifficultywithcoursematerialandwhy(seealso[Bakhariaetal.,2016a;Condeetal.,2015]).

Whilethemostpopularcourse,noquizinformationisreportedforthecourse“FoundationsofBigData”.Wethereforedonothaveinformationonstudentgrades,andrelyonlyonaccesstocoursematerialasanindicationofprogressandcompletion.AsT3.4progressesandmorestudentinteractioniscapturedwewillrevisitothercourseswithandwithoutstudentgradinginformationtomaphowbehaviourasseen in individualandacrossallcourses impactsperformanceandretention.ROIs identified ineachperspectivewillbefurtherinvestigatedusing,wherepossible,allthreeanalysisapproaches,toincrease


coverageinouranalysisandprovidemoredetailtoansweralsoadditionalquestionsraisedduringourinitialanalysis.

4.3 ContributiontoFace‐FaceAndBlendedCoursesinEDSA

LearninganalyticsinWP3looksat

1. face‐face(f‐f)andblendedcourses,whereattendanceisrestrictedbystudentlocation,2. online courses, which remove restrictions due to physical access/location, but with the

attendantdisadvantageofrestricted,remoteaccesstoinstructorsandfellowstudents.Thisdeliverablefocusesonanalysisofeventdatacollectedfromthelatter,whileD3.2focusesontheformer.While in‐person courses benefit from direct contactwith teaching staff and other studentsonline,self‐studycourseshaveatbestremote,oftenasynchronousaccesstootherparticipants.Wearethereforereliantoneventinteractionlogsandformal,onlinefeedbackformstocollectinformationonstudentsthatmaybeusedtoaidtheinterpretationoftheirbehaviour.Conversely,theeventlogsprovideobjective records of student interactionwith coursematerial, and consequently,maps of relativelydetailedstudentbehaviourthatisavailableonlythroughexperienceandintuitioninf‐finstruction,andeventhenonlytoalimitedextent,outsideformalassignmentsandexams.F‐Fandblendedcoursesontheotherhandtendtohavefairlydetailedinformationonstudentdemographics,informationthatismoredifficult to collect in online courses, especiallywhere students are not registered for a (paid)certificate of attainment or other qualification. By combining information collected about studentbackgrounds, interactionandbehaviour inthesetwocontextsweenvisagemorecomprehensive, in‐depthanalysisofstudentbehaviourandthefactorsthatinfluencethis.

We acknowledge that differences in design and delivery of online and f‐f courses preventstraightforwardmappingof interactionandbehaviour.This is also the casewithinonlineand f‐f orblendedcoursesondifferenttopicsandwithdesignedfordifferentcontexts,duealsototeachingstyleandrelevantlearningfacilitiesavailable.Diveretal.,(2015)acknowledgethisintheircomparisonoftwoMOOCS.Nguyen(2015)comparesonline learning to f‐f courses, to identifywhat contributes toeffectiveonlinelearning,inordertocontinuetoimproveonthisformofstudy.TheremainderoftheprojectwillseeclosersynergybetweenanalysisinthetwopartsofWP3asoutlinedintheintroduction(section1),toensureweachievetheaimsofLearningAnalyticsandthoseoftheprojectasawhole.

4.4 SynergywithEDSADemandAnalysisTask

Akeygoaloftheanalysisofdatasciencejob(andskill)demand(WP1)istodeterminewheretrainingofnewDataScientistsandretrainingofpractitionersinthefieldorsimilarfieldsisneededtocloseskillgapsidentified.Thedemandanalysistaskprovidesinformationondemandalongthreekeycriteria:

1. overtime2. acrosslocation(andcorrespondingspokenorworkinglanguage)3. byskillorskillset.

Wealsoconsider the impactofa fourthcriterion‐domainor industrysector,oncategorisationandrankingorrelevanceofskillsseenascoretoDataScience.

Informationgainedinthedemandanalysistaskonthecategorisationandrankingofskillsistofeedinto(individual) course and programme design, to allow students to create learning plans combining



courses on skills core to their target role type(s) and domain or sectorwith those that teach skillstypicallyrequiredalongwiththeformer.Thedemandanalysistaskaimsalsotobuildinsupporttoguideend users in constructing “skill profiles”. This will be based also on information collected frompractitioners inthefieldabouttheirperspectivesonskillscoretotheirrolesanddomain,aswellascapabilitywithintheexistingworkforceandcapacityofthejobmarkettoabsorbtheseskills.

Thelearninganalyticstask,ontheotherhand,aimstoderiveinformationoneffectivenessofcoursecontentanddeliveryandthe impactofstudentdemographics(individualandgroup)andadditionalfactorssuchasmotivationoncoursecompletionandgrades.Bydemonstrating(relative)importanceofselectedskillstoaroleorjobtypeandoptimalpathstofollowtothese,analysisresultsfromthedemandanalysistaskshouldcomplementfindingsinlearninganalytics,toallowmoreinformeddecisionstobemadeincourseandcurriculadesign.

Informationobtainedoninterestandengagementwithspecificcoursesandcategoriesofcourseswillalsobefedintorecommendationoflearningmaterial,bothcompletecoursesthatrequireagooddegreeofcommitment,withtheopportunitytoobtainverifiablequalifications,andstandaloneandbite‐sizelearningresources‐lectures,tutorialsandpresentations,amongothers,thatrequirelowereffortbutareablealsotosupplementmorestructuredlearninginrelatedcourses.

4.5 InitialProposalfora‘LearningAnalyticsFramework’forEDSA

TheoverallaimofLearningAnalyticsintheEDSAprojectistofollowevidence‐basedbestpractice:

● forcoursedesign(contentanddelivery),● to structure and tailor student feedback to guide selection of courses, and following this,

encourage retention and provide the resources required to ensure student completion andsuccess(boththeindividualandthecohort),

● toprovideformalmeasuresandbenchmarksdescribingstudentbehaviour,performanceandretentionforreusewithinandoutwiththeproject,basedon:

○ (inadditionto)coursematerial:○ dataonstudentconsumptionofmaterial(passiveengagement),○ studentco‐constructionofdata‐throughinteractionindiscussionfora,completionof

assignments,open‐endedquizzesandexamresponses(activeengagement).

Weaim touse this information andour analysis to obtain reliable, data‐driven insight into studentbehaviour and identify critical and more general factors that impact success [see, among others,Bakhariaetal.,2016a;Diveretal.,2015;Kevanetal.,2016;Nguyen2015;Wellsetal.,2016].Basedonthepreliminaryanalysisdetailed in thisdeliverablewehave identifiedcore informationrequired toachievethisaim,anddifferencesingranularityaswellasgapsinthecurrentdatacollectionorreportingfunctionalityinthecoredatastoreswefeedfromforouranalysis.Tosucceedherewerelyondetailed,comprehensiveinformationon

● studentdemographics(anonymisedandcollatedwherenecessarytoprotectprivacy)tosupportdefinitionofandclusteringbasedonstudenttypes.Demographicaldatamustinclude:

○ ageandgender,○ geographicallocation(home,schooland/orwork),○ highestlevelofeducationalattainment,○ for in‐work students ‐ occupation along with domain of specialisation and current

industrysector.


● detailedobjectiveandsubjective informationonstudentbehaviourcollectedacrossdifferentcoursetypesanddeliverymechanisms(self‐studyorguided/structuredonline,f‐fandblendedcourses),includingbutnotlimitedto:

○ access to course material with indication of type, time spent on each, sub‐levels ofinteractionwhereapplicable,

○ relationships between course material, indicating also where required (core/) orsupplementary.

We thereforepropose the constructionof a formal framework for LearningAnalytics inEDSA,withpotentialforreusebeyondtheprojectremit,asfollows:

1. thedefinitionofcoursetypes;ouranalysisrevealedcleardistinctionbetweenthethreedatasetsweinvestigated‐EDSA‐specificcourses,MOOCsandVideoLectures.Differentcriteriamaybeusedfordefiningthese;furtherdiscussionwithintheprojectandfurtherstudyoftheliteratureshouldaddtothoseseenhere‐standalonelearningmaterialaslectures,tutorials,workshop,papers; complete courses broken down into defined sub‐topics with or without explicitassessmentandoptionsforcertification.Thisistoallowustomoreeffectivelymanagestudentexpectationofcoursecontent,commitmentandcertificationtype,amongothers.

2. thedefinitionofastructureforeachcoreorprescribedevent,actionorcasecapturedduringstudentinteractionwithacourse,

a. tosupportthederivationofmetadatabasedonthisstructureforthedatacollected,b. andguideeffective(re)useofanalysisresults.

3. cleardistinctionbetweenresourcesinacoursebasedon:a. formatanddeliverymechanism,e.g.,videovs.textpagevs.downloadablematerial(e.g.,

imagesorprintablePDFsofacompletemodule),b. content type, e.g., course module vs. course description (e.g., syllabi) vs. discussion

forum vs. supplementary information (e.g., external webinars, related datasets andexperiments),

c. availability of material and requirements for completing assignments, quizzes andexams.

AsT3.4progresseswewillrefinetheEDSALAframework,withanaimtoprovideastructuredworkflowforEDSAandoutwiththeproject.Weenvisagetheframeworkwillalsosupportthe identificationofoptimal analysis and results presentation techniques for different target user types ‐ students,instructors and other interested parties such as course design teams and policy‐makers in highereducation,on‐the‐jobskillstrainingandtherecruitmentmarket.



5. Conclusions

ThisdeliverablereportsinitialworkinT3.4,onthe“designanddeploymentoflearninganalytics”withinEDSA, to aggregate, harmonise and analyse event data generated through student interactionwithonline courses created for the EDSA project and by project partners and third parties, to trainindividualstodevelopcapabilityrequiredtoworkinassortedjobsinDataScienceacrosstheEU.Theaimistomonitorstudentbehaviourinordertoobtaininsightintothestudentexperienceandtheimpactof student behaviour, demographics and interests on course completion and level of success orattainment.FindingsfromT3.4aretofeedintothedesignofnewcoursesandcurriculaforDataScienceprogrammes,toenabletailoringofcontentanddeliverymechanismstothestudentcontext.Further,information obtained about demographic andother external factors that lead to non‐completion orfailureorotherwisenegativelyimpactthestudentexperiencewillfeedintothedevelopmentofbothstudentandinstructor‐specificfeedbackandsupportmechanisms.

Initial,exploratoryanalysisinT3.4servedthreemainpurposes:

1. toobtainanunderstandingofthestructureandcontentofthekeydatastoreswemustrelyonforlearninganalyticsinWP3,andthereforecollectadditionalrequirementsfordatacaptureandreuse,toensureweobtaindatathatwillenableustoanswerthequestionsposedinT3.4.

2. tobuilddataoverviewsthathelpedustoobtainapictureofstudentbehaviour,andthatwillcontinuetofeedintoidentifyingadditionaltoolsandtechniquestoemployformoredetailedanalysisofinterestingpatternsandtrendsrevealedinthebiggerpicture.

3. toprovideearlyindicatorsofthelinksfromthistasktootherworkinEDSA,first,analysisalsoof studentbehaviour in the f‐f andblended courses alsodelivered inEDSA.Tomeetoverallprojectgoalswemustalsocross‐fertilisefindingsinT3.4withthoseindemandandskillanalysistasksinWP1,toguideselectionofcoursestomeetstudents’ interestsandskilldevelopmentrequirementsfortargetjobroles,andtoimprovecoursecompletionandsuccess,andultimately,successinfillingjobrolesinDataScience.

5.1 NextSteps

BasedonourfindingstodateandlessonslearntinourinitialanalysisweconcludethedeliverablebypresentingourplansforcompletingtheworkinT3.4duringthesecondhalfoftheproject.

The visualisation‐driven dashboards and other visual representations employed in process mining(described in the analysis in section3)paint a pictureof interest and engagement across the threecoursetypesweexplored.TheVideoLecturesdashboardsspecificallyalsoindicatewhereanalysis inT3.4alignswithEDSAworkon(datascienceskillandjob)demandanalysis.

ToensurewebuildonandcontributetothestateoftheartinLAasT3.4progresseswemustexpandontheexploratoryanalysisthatlooksatuserinteractionatthecourseandsystemlevel,toinvestigateinmoredetailsub‐levelsinthemoregeneralviewingbehaviourreportedinthisdeliverable.Weaimtofeed the findings obtained to date intomore in‐depth analysis employing other analytical and datamining techniques, and considering also relevant research on distance learning, learning at scale,technology‐enhanced learning and educational datamining (EDM). Thismore detailed analysiswillexaminethedynamicsindedicatedcoursesandotherlearningresourcesinDataScience.Wewillalso


cluster and merge users based on additional demographic and contextual data, e.g., source IP,informationdescribingstudentscollectedatregistrationandfromcoursefeedbackforms,toidentifypatterns influencedby studentbackground and context, andhowbehaviourdiffersbetween coursetypesandlearningresourcesasdefinedinsection4.5.

TheLearningAnalyticscommunityrecognisestheimportanceofaccesstoopendataandresourcesinadvancingresearchinthefield,aswellasforpractical,successfuldeliveryofonlinecourses.Practicalresearch and applicationof LA rely on real use cases and real or synthetic data (typically real dataanonymisedtoallowreusewithoutviolatingprivacyorethicsasintheEDSALMS).VideoLectures.NETprovidesanindependent,openresourcecontaininglearningresourcesthatwehavealreadystartedtoinvestigate. To ensure our analysis approach and outcomes are reusable and replicable we mustexaminealsootherrelevantusecasesandresources.WewillexaminepilotsofLearningAnalyticstoolsembeddedintoorfeedingdirectlyfromLMSsuchasOUAnalyse,whichiscarryingoutliveanalysisofdatafromtheVirtualLearningEnvironment(VLE)oftheOpenUniversityintheUKtoprovideguidanceonthestructuringoffeedbacktodistancelearners,toaidretentionandfeedintotheuniversity’saimtoimprovethestudentexperiencethroughits“StudentsFirst:StrategyforGrowth”.AnotherexampleisEdinburghUniversity’sLARCproject23(theLearningAnalyticsReportCard),whichdirectlyinvolvesstudentsinthecollectionandanalysisofdataontheirlearningactivityandprogress.

WP3inEDSAalsoconsiderstheanalysisofstudentinteractionandbehaviourinface‐faceandblendedcourses.Thispreliminaryreportconsidersonlyonlinedata;asTask3.4progresseswewilllookalsoattheoutcomesofanalysisandlessonslearntincoursesthatseephysicalinteractionbetweenstudentsand their instructors, to identifywheresimilarbehaviour isobserved,andwheresuccessusingonemethodofdeliverymayfedintoovercominglimitationsintheother.

As part of the contribution of the EDSA project to research and practical application of LearningAnalyticsweareexploring,inconjunctionwiththedefinitionoftheframeworkdiscussedinsection4.5,theconstructionofsemi‐automatedworkflowsforlearninganalytics.Toworktowardthisgoalwewillemploythetechniquespresentedinthisdeliverable‐statistics,visualanalyticsandprocessmining‐withthevisualisationofinputdataandanalysisresults,andalsoothertechniqueswefindtoaugmentfurtherdetailedanalysis.Wewillbuildontheinsightgainedthroughtheexploratoryanalysiscarriedouttothispoint,tocontinuetoexploreoptionstowardboth(automated)dataminingandanalysistohandledataonascalemuchlargerthanthatemployedhere,andforhuman‐drivenexploration,analysisandmanagementof“own”data.

Wewillalsoexplorethedesignandconstructionofintuitiveuserinterfacestoguidethepresentationand reuseof findings fordifferentaudiences.Forexample, an instructorwoulddownloaddataonacoursetheyaredeliveringalongwithstudentinteractionanddemographicdatafromtheirplatformofchoice(e.g.,theEDSAportal,FutureLearnorCoursera),andfeedthisintothetoolsweprovidethemwith to trigger, transparently, semi‐automated scripts and analysis modules that would generate

23http://www.de.ed.ac.uk/project/learning‐analytics‐report‐card



reportsonstudentprogress.Theywouldthenbeprovidedwithintuitive,low‐efforttoolsforcreatingcustomreportstoanswerqueriestypicalinonlineeducationdelivery,andtoguidetheminobtaininganswerstoquestionsoutsidethosetypicallyposedwithinthisscope.Correspondingsupportwouldbeprovidedtostudentsandpolicyanddecision‐makerstofeedthefindingsfromlearninganalyticsintostudy,theirregularworkandintooccasional,butrelatedtasks.

Finally,Inadditiontoexpandingourdatacollectionprocesstoincludeadditionalmetadataasdetailedinsection4.5,weaimtofeedintoouranalysis findings fromresearchon learninganalytics inotherrelatedprojects(asdiscussedabove),andlinktoopendatasets(suchastheLAKDataset&Challenge24;seealsoDietzeetal.,(2015))collectedaspartofsuchresearch.Theultimateaimistoprovide,basedonthe more in‐depth analysis and wider coverage this broader scope will enable, evidence‐basedguidelinesforfurtherresearchandforpracticaluseoftheoutcomesofLearningAnalyticsinEDSA,toaugmentcoursedeliveryandlearning,andimprovethestudentexperience.LinkingtoexistingdatasetswillalsosupportanimportantgoalinEDSAandEU‐fundedresearch,toprovideasLinkedOpenDatadatageneratedandtheoutcomesofresearch.

24http://meco.l3s.uni‐hannover.de:9080/wp2


6. References

[van der Aalst 2011] van der Aalst, Wil. (2011). Process Mining: Discovery, Conformance andEnhancementofBusinessProcesses,Springer

[Bakhariaetal.,2016a]Bakharia,Aneesha,Corrin,Linda,deBarba,Paula,Kennedy,Gregor,Gašević,Dragan, Mulder, Raoul, Williams, David, Dawson, Shane and Lockyer, Lori. (2016). A conceptualframeworklinkinglearningdesignwithlearninganalytics. InProc.,SixthInternationalConferenceonLearningAnalytics&Knowledge(LAK'16),pp.329‐338.

[Bakhariaetal.,2016b]Bakharia,Aneesha,Kitto,Kirsty,Pardo,Abelardo,Gašević,DraganandDawson,Shane. (2016). Recipe for success: lessons learnt from using xAPI within the connected learninganalyticstoolkit.InProc.,SixthInternationalConferenceonLearningAnalytics&Knowledge(LAK'16),pp.,378‐382.

[Conde et al., 2015] M. Á Conde, F. J. García‐Peñalvo, D. A. Gómez‐Aguilar and R. Therón. (2015),ExploringSoftwareEngineeringSubjectsbyUsingVisualLearningAnalyticsTechniques,IEEERevistaIberoamericanadeTecnologiasdelAprendizaje,10(4),pp.242‐252.

[Dietze et al., 2015]Dietze, S., Taibi,D., d’Aquin,M. (preprint ‐ 2015), Facilitating Scientometrics inLearningAnalyticsandEducationalDataMining–theLAKDataset,SemanticWebJournal.

[Diveretal.,2015]Diver,P.,Martinez,I.(2015).MOOCsasamassiveresearchlaboratory:opportunitiesandchallenges,JournalofDistanceEducation,36(1),pp.5‐25.

[Gunther & van der Aalst 2007] Gunther, Christian and van der Aalst,Wil. (2007). FuzzyMining ‐AdaptiveProcessSimplificationBasedonMulti‐perspectiveMetrics,Springer.

[Kevanetal.,2016]Kevan,JonathanM.,Menchaca,MichaelP.andHoffman,EllenS.(2016).DesigningMOOCsforsuccess:astudentmotivation‐orientedframework.InProceedingsoftheSixthInternationalConferenceonLearningAnalytics&Knowledge(LAK'16),pp.274‐278.

[Kizilcec&Halawa2015]Kizilcec,R.F.andHalawa,S.(2015).AttritionandAchievementGapsinOnlineLearning.Proc.,2ndACMConferenceonLearning@Scale(L@S'15),pp.57‐66.

[Kuzilek et al., 2015] Kuzilek, J., Hlosta, M., Herrmannova, D., Zdrahal, Z. andWolff, A., (2015). OUAnalyse:AnalysingAt‐RiskStudentsatTheOpenUniversity,LearningAnalyticsReview,LAK15‐1.

[Mukala et al., 2015]Mukala,Patrick,Buijs, JoosC.A.M.,Leemans,Maikel, vanderAalst,WilM.P.(2015).LearningAnalyticsonCourseraEventData:AProcessMiningApproach.Proc.,5thInternationalSymposiumonData‐drivenProcessDiscoveryandAnalysis(SIMPDA2015),pp.18‐32.

[Nguyen 2015] Nguyen, T. (2015). The Effectiveness of Online Learning:Beyond No SignificantDifferenceandFutureHorizons,JournalofOnlineLearningandTeaching,1(2),pp.309‐319.

[Thomas&Cook2005]Thomas, J.J. andCook,K.A. (2005). Illuminating thePath:TheResearchandDevelopmentAgendaforVisualAnalytics,IEEECSPress

[Wells et al., 2016] Wells, Marc, Wollenschlaeger, Alex, Lefevre, David, Magoulas, George D. andPoulovassilis, Alexandra. (2016). Analysing engagement in an online management programme andimplications for course design. In Proc., Sixth International Conference on Learning Analytics &Knowledge(LAK'16),pp.236‐240.



7. Appendices

7.1 AppendixA.‐ListofQueries

EventsforonlinecoursesdeliveredbyEDSAorthroughitswebsitearecapturedintheEDSALearningLocker(seesection2.1).FortheanalysisreportedinthisdeliverableonEDSAcourses,thefollowing(key)querieswererun,withsampleresultsasbelow.Inadditiontoqueriesprovidingcompletedumpsfor a specified user(s), a course, or all users/courses we use aggregate queries to flatten the datastructureandeasetheextractionofdataspecifictotargetedanalysis,basedonthefindingsfromthestartoftheanalysisprocessusedtoidentifyROIstoexamineinmoredetail.Welistbelowkeyqueries,withsnapshotsoftheresultsobtained.

7.1.1 FollowaNamedUserdb.statements.aggregate([ {"$match":{"statement.verb.id":{$exists:true}, "statement.actor.name" : { $in: [ "social_user_163", "social_user_20", "social_user_94", "social_user_131", "social_user_137","moodle_user_3270","social_user_202","social_user_189","social_user_304","social_user_95"]} } },{ "$group":{"_id":{studentId:"$statement.actor.name",activity:"$statement.verb.id",courseId:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.courseid",courseName:"$statement.context.contextActivities.grouping.definition.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_course.fullname",courseUrl:"$statement.context.contextActivities.grouping.definition.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_course.url",courseUrlAccessed:"$statement.context.contextActivities.grouping.id",activityLocationId:"$statement.context.contextActivities.parent.id",activityDefinitionType:"$statement.context.contextActivities.parent.definition.type",activityDefinitionName:"$statement.context.contextActivities.parent.definition.name.en",quizResult:"$statement.result.score.raw",quizSuccess:"$statement.result.success",quizCompletion:"$statement.result.completion",activityTriggerId:"$statement.object.id",activityTriggerDefinitionType:"$statement.object.definition.type",activityTriggerDefinitionName:"$statement.object.definition.name.en",activityTriggerUrl:"$statement.object.extensions.url",activityTriggerExternalUrl:"$statement.object.definition.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_module.externalurl",activityContextEventname:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.eventname",activityContextComponent:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.component",activityContextAction:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.action",activityContextObjectTable:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.objecttable",


activityContextObjectid:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.objectid",activityContextOtherId:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.other",activityContextTimecreated:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.timecreated",activityContextOriginatingDevice:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.origin",activityContextObject:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.object",ip:"$statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.ip",timestamp:"$timestamp",updatedAt:"$updated_at" } } }])

7.1.2 ResultsSnapshot{ "_id":{ "studentId":"social_user_304", "activity":"http://id.tincanapi.com/verb/viewed", "courseId":"15", "courseName":[ "EDSAOnlineCourses" ], "courseUrl":[ "http://courses.edsa‐project.eu" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu" ], "activityTriggerId":"http://courses.edsa‐project.eu/course/view.php?id=15", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/course", "activityTriggerDefinitionName":"ProcessMining", "activityContextEventname":"\\core\\event\\course_viewed", "activityContextComponent":"core", "activityContextAction":"viewed", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1467709645), "activityContextOriginatingDevice":"web", "activityContextObject":"course", "ip":"131.155.69.119", "timestamp":ISODate("2016‐07‐05T09:07:25Z"), "updatedAt":ISODate("2016‐07‐05T09:07:26.504Z") }},{ "_id":{ "studentId":"social_user_304", "activity":"http://id.tincanapi.com/verb/viewed", "courseId":"33", "courseName":[ "EDSAOnlineCourses", "BigDataAnalytics" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=33" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=33" ], "activityTriggerId":"http://courses.edsa‐project.eu/mod/quiz/view.php?id=340", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/quiz",



"activityTriggerDefinitionName":"Quiz", "activityContextEventname":"\\mod_quiz\\event\\course_module_viewed", "activityContextComponent":"mod_quiz", "activityContextAction":"viewed", "activityContextObjectTable":"quiz", "activityContextObjectid":"7", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1465667786), "activityContextOriginatingDevice":"web", "activityContextObject":"course_module", "ip":"145.120.15.201", "timestamp":ISODate("2016‐06‐11T17:56:26Z"), "updatedAt":ISODate("2016‐06‐11T17:56:27.310Z") }},{ "_id":{ "studentId":"social_user_304", "activity":"http://id.tincanapi.com/verb/viewed", "courseId":"27", "courseName":[ "EDSAOnlineCourses", "BigDataArchitecture" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=27" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=27" ], "activityTriggerId":"http://courses.edsa‐project.eu/mod/forum/view.php?id=288", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/forum", "activityTriggerDefinitionName":"Coursediscussionforum", "activityContextEventname":"\\mod_forum\\event\\course_module_viewed", "activityContextComponent":"mod_forum", "activityContextAction":"viewed", "activityContextObjectTable":"forum", "activityContextObjectid":"19", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1465666548), "activityContextOriginatingDevice":"web", "activityContextObject":"course_module", "ip":"145.120.15.201", "timestamp":ISODate("2016‐06‐11T17:35:48Z"), "updatedAt":ISODate("2016‐06‐11T17:35:48.935Z") }},{ "_id":{ "studentId":"social_user_304", "activity":"http://adlnet.gov/expapi/verbs/answered", "courseId":"33", "courseName":[ "EDSAOnlineCourses", "BigDataAnalytics" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=33" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=33", "http://courses.edsa‐project.eu/mod/quiz/attempt.php?attempt=30" ], "activityLocationId":[ "http://courses.edsa‐project.eu/mod/quiz/view.php?id=340" ], "activityDefinitionType":[ "http://lrs.learninglocker.net/define/type/moodle/quiz" ], "activityDefinitionName":[


"Quiz" ], "quizSuccess":false, "quizCompletion":false, "activityTriggerId":"http://courses.edsa‐project.eu/mod/question/question.php?id=96", "activityTriggerDefinitionType":"http://adlnet.gov/expapi/activities/cmi.interaction", "activityTriggerDefinitionName":"10", "activityContextEventname":"\\mod_quiz\\event\\attempt_reviewed", "activityContextComponent":"mod_quiz", "activityContextAction":"reviewed", "activityContextObjectTable":"quiz_attempts", "activityContextObjectid":"30", "activityContextOtherId":"a:1:{s:6:\"quizid\";s:1:\"7\";}", "activityContextTimecreated":NumberLong(1465667775), "activityContextOriginatingDevice":"web", "activityContextObject":"attempt", "ip":"145.120.15.201", "timestamp":ISODate("2016‐06‐11T17:56:14Z"), "updatedAt":ISODate("2016‐06‐11T17:56:15.862Z") }},{ "_id":{ "studentId":"social_user_202", "activity":"http://adlnet.gov/expapi/verbs/answered", "courseId":"27", "courseName":[ "EDSAOnlineCourses", "BigDataArchitecture" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=27" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=27", "http://courses.edsa‐project.eu/mod/quiz/attempt.php?attempt=18" ], "activityLocationId":[ "http://courses.edsa‐project.eu/mod/quiz/view.php?id=325" ], "activityDefinitionType":[ "http://lrs.learninglocker.net/define/type/moodle/quiz" ], "activityDefinitionName":[ "Quiz" ], "quizResult":NumberLong(1), "quizSuccess":true, "quizCompletion":true, "activityTriggerId":"http://courses.edsa‐project.eu/mod/question/question.php?id=82", "activityTriggerDefinitionType":"http://adlnet.gov/expapi/activities/cmi.interaction", "activityTriggerDefinitionName":"6", "activityContextEventname":"\\mod_quiz\\event\\attempt_reviewed", "activityContextComponent":"mod_quiz", "activityContextAction":"reviewed", "activityContextObjectTable":"quiz_attempts", "activityContextObjectid":"18", "activityContextOtherId":"a:1:{s:6:\"quizid\";s:1:\"6\";}", "activityContextTimecreated":NumberLong(1461099312), "activityContextOriginatingDevice":"web", "activityContextObject":"attempt", "ip":"77.11.16.208", "timestamp":ISODate("2016‐04‐19T20:55:11Z"), "updatedAt":ISODate("2016‐04‐19T20:55:12.841Z") }},{ "_id":{ "studentId":"social_user_202", "activity":"http://adlnet.gov/expapi/verbs/completed", "courseId":"27", "courseName":[ "EDSAOnlineCourses",



"BigDataArchitecture" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=27" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=27", "http://courses.edsa‐project.eu/mod/quiz/attempt.php?attempt=18" ], "quizResult":9.5, "quizCompletion":true, "activityTriggerId":"http://courses.edsa‐project.eu/mod/quiz/view.php?id=325", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/quiz", "activityTriggerDefinitionName":"Quiz", "activityContextEventname":"\\mod_quiz\\event\\attempt_reviewed", "activityContextComponent":"mod_quiz", "activityContextAction":"reviewed", "activityContextObjectTable":"quiz_attempts", "activityContextObjectid":"18", "activityContextOtherId":"a:1:{s:6:\"quizid\";s:1:\"6\";}", "activityContextTimecreated":NumberLong(1461099312), "activityContextOriginatingDevice":"web", "activityContextObject":"attempt", "ip":"77.11.16.208", "timestamp":ISODate("2016‐04‐19T20:55:12Z"), "updatedAt":ISODate("2016‐04‐19T20:55:12.841Z") }},{ "_id":{ "studentId":"social_user_163", "activity":"http://adlnet.gov/expapi/verbs/registered", "courseName":[ "EDSAOnlineCourses" ], "courseUrl":[ "http://courses.edsa‐project.eu" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu" ], "activityTriggerId":"http://courses.edsa‐project.eu", "activityTriggerDefinitionType":"http://id.tincanapi.com/activitytype/site", "activityTriggerDefinitionName":"EDSAOnlineCourses", "activityContextEventname":"\\core\\event\\user_created", "activityContextComponent":"core", "activityContextAction":"created", "activityContextObjectTable":"user", "activityContextObjectid":NumberLong(166), "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1458491893), "activityContextOriginatingDevice":"web", "activityContextObject":"user", "ip":"185.18.231.50", "timestamp":ISODate("2016‐03‐20T16:38:13Z"), "updatedAt":ISODate("2016‐03‐20T16:38:13.865Z") }},{ "_id":{ "studentId":"social_user_163", "activity":"http://id.tincanapi.com/verb/viewed", "courseId":"26", "courseName":[ "EDSAOnlineCourses" ], "courseUrl":[ "http://courses.edsa‐project.eu" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu" ],


"activityTriggerId":"http://courses.edsa‐project.eu/course/view.php?id=26", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/course", "activityTriggerDefinitionName":"FoundationsofBigData", "activityContextEventname":"\\core\\event\\course_viewed", "activityContextComponent":"core", "activityContextAction":"viewed", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1460454440), "activityContextOriginatingDevice":"web", "activityContextObject":"course", "ip":"192.168.101.23", "timestamp":ISODate("2016‐04‐12T09:47:20Z"), "updatedAt":ISODate("2016‐04‐12T09:47:20.606Z") }},{ "_id":{ "studentId":"social_user_94", "activity":"https://brindlewaye.com/xAPITerms/verbs/loggedout/", "courseName":[ "EDSAOnlineCourses" ], "courseUrl":[ "http://courses.edsa‐project.eu" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu" ], "activityTriggerId":"http://courses.edsa‐project.eu", "activityTriggerDefinitionType":"http://id.tincanapi.com/activitytype/site", "activityTriggerDefinitionName":"EDSAOnlineCourses", "activityContextEventname":"\\core\\event\\user_loggedout", "activityContextComponent":"core", "activityContextAction":"loggedout", "activityContextObjectTable":"user", "activityContextObjectid":"97", "activityContextOtherId":"a:1:{s:9:\"sessionid\";s:26:\"rfh9sd79au2crus7bv6ovt4us0\";}", "activityContextTimecreated":NumberLong(1457530739), "activityContextOriginatingDevice":"web", "activityContextObject":"user", "ip":"195.96.242.52", "timestamp":ISODate("2016‐03‐09T13:38:59Z"), "updatedAt":ISODate("2016‐03‐09T13:38:59.337Z") }},{ "_id":{ "studentId":"social_user_137", "activity":"https://brindlewaye.com/xAPITerms/verbs/loggedin/", "courseName":[ "EDSAOnlineCourses" ], "courseUrl":[ "http://courses.edsa‐project.eu" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu" ], "activityTriggerId":"http://courses.edsa‐project.eu", "activityTriggerDefinitionType":"http://id.tincanapi.com/activitytype/site", "activityTriggerDefinitionName":"EDSAOnlineCourses", "activityContextEventname":"\\core\\event\\user_loggedin", "activityContextComponent":"core", "activityContextAction":"loggedin", "activityContextObjectTable":"user", "activityContextObjectid":"140", "activityContextOtherId":"a:1:{s:8:\"username\";s:15:\"social_user_137\";}", "activityContextTimecreated":NumberLong(1457096110), "activityContextOriginatingDevice":"web", "activityContextObject":"user", "ip":"88.98.46.53", "timestamp":ISODate("2016‐03‐04T12:55:10Z"), "updatedAt":ISODate("2016‐03‐04T12:55:10.415Z") }}



7.1.3 CaptureUserActivities

Thesameusermayloginusingmultipleidentitiesormultipleusersmayloginfromasinglelocation.Thesequeries,bysortingonIPaddress,capturebothcases,andweusefurtheranalysistodistinguishbetweenthetwo.

Queriesasin9.1maybeusedtoextractthedatarequired.

7.1.4 GroupbyEventwithinaCourse

Thissetofqueriesfollowsactivitywithincourses,tomapthepathoftheinitialgroupregisteredforacoursethroughtocompletion,inordertocaptureinteractionwithlessonsandcoursework,andfall‐offasthecourseprogresses.

Queriesasin9.1butwiththefollowingmatchcriteriafor,e.g.,thecoursewithID26‐"FoundationsofBigData"aswellaseventstotheloginpromptfortheEDSAOnlinecoursesportal‐whichcaptureslogin,logoutandregistrationeventsforallothercourse.Thisqueryalsofiltersouteventsloggedbytheusersadmin,guestanddemo.

{"$match":{"statement.verb.id":{$exists:true}, "statement.context.extensions.http://lrs&46;learninglocker&46;net/define/extensions/moodle_logstore_standard_log.courseid" :{$in:["1","26"]}, "statement.actor.name" : { $nin: [ "admin", "guest","demo"]} } }


7.2 AppendixB.‐SampleSnapshotsofEventDataforCourse"FoundationsofBigData"{ "_id":{ "studentId":"social_user_48", "activity":"http://id.tincanapi.com/verb/viewed", "courseId":"26", "courseName":[ "EDSAOnlineCourses", "FoundationsofBigData" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "activityTriggerId":"http://courses.edsa‐project.eu/mod/forum/view.php?id=287", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/forum", "activityTriggerDefinitionName":"Coursediscussionforum", "activityContextEventname":"\\mod_forum\\event\\course_module_viewed", "activityContextComponent":"mod_forum", "activityContextAction":"viewed", "activityContextObjectTable":"forum", "activityContextObjectid":"18", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1467900935), "activityContextOriginatingDevice":"web", "activityContextObject":"course_module", "ip":"88.0.206.95", "timestamp":ISODate("2016‐07‐07T14:15:35Z"), "updatedAt":ISODate("2016‐07‐07T14:15:36.082Z") }},{ "_id":{ "studentId":"moodle_user_3804", "activity":"http://id.tincanapi.com/verb/viewed", "courseId":"26", "courseName":[ "EDSAOnlineCourses", "FoundationsofBigData" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "activityTriggerId":"http://courses.edsa‐project.eu/mod/page/view.php?id=253", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/page", "activityTriggerDefinitionName":"Learningobjectives&syllabus", "activityContextEventname":"\\mod_page\\event\\course_module_viewed", "activityContextComponent":"mod_page", "activityContextAction":"viewed", "activityContextObjectTable":"page", "activityContextObjectid":"22", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1466171578), "activityContextOriginatingDevice":"web", "activityContextObject":"course_module", "ip":"131.211.81.80", "timestamp":ISODate("2016‐06‐17T13:52:58Z"), "updatedAt":ISODate("2016‐06‐17T13:52:58.356Z") }},{ "_id":{ "studentId":"social_user_304",



"activity":"http://id.tincanapi.com/verb/viewed", "courseId":"26", "courseName":[ "EDSAOnlineCourses", "FoundationsofBigData" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "activityTriggerId":"http://courses.edsa‐project.eu/mod/forum/view.php?id=287", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/forum", "activityTriggerDefinitionName":"Coursediscussionforum", "activityContextEventname":"\\mod_forum\\event\\course_module_viewed", "activityContextComponent":"mod_forum", "activityContextAction":"viewed", "activityContextObjectTable":"forum", "activityContextObjectid":"18", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1465666540), "activityContextOriginatingDevice":"web", "activityContextObject":"course_module", "ip":"145.120.15.201", "timestamp":ISODate("2016‐06‐11T17:35:40Z"), "updatedAt":ISODate("2016‐06‐11T17:35:41.125Z") }},{ "_id":{ "studentId":"social_user_296", "activity":"http://id.tincanapi.com/verb/viewed", "courseId":"26", "courseName":[ "EDSAOnlineCourses", "FoundationsofBigData" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26", "http://courses.edsa‐project.eu/mod/forum/view.php?id=287" ], "activityTriggerId":"http://courses.edsa‐project.eu/mod/forum/discuss.php?d=5", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/forum_discussions", "activityTriggerDefinitionName":"Greetings", "activityContextEventname":"\\mod_forum\\event\\discussion_viewed", "activityContextComponent":"mod_forum", "activityContextAction":"viewed", "activityContextObjectTable":"forum_discussions", "activityContextObjectid":"5", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1465302695), "activityContextOriginatingDevice":"web", "activityContextObject":"discussion", "ip":"82.150.248.28", "timestamp":ISODate("2016‐06‐07T12:31:35Z"), "updatedAt":ISODate("2016‐06‐07T12:31:36.264Z") }},{ "_id":{ "studentId":"moodle_user_9856", "activity":"http://id.tincanapi.com/verb/viewed", "courseId":"26", "courseName":[ "EDSAOnlineCourses", "FoundationsofBigData"


], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "activityTriggerId":"http://courses.edsa‐project.eu/mod/url/view.php?id=315", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/url", "activityTriggerDefinitionName":"Yourfeedback", "activityTriggerExternalUrl":"http://courses.edsa‐project.eu/mod/feedback/view.php?id=285&courseid=26", "activityContextEventname":"\\mod_url\\event\\course_module_viewed", "activityContextComponent":"mod_url", "activityContextAction":"viewed", "activityContextObjectTable":"url", "activityContextObjectid":"76", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1464765251), "activityContextOriginatingDevice":"web", "activityContextObject":"course_module", "ip":"94.224.94.171", "timestamp":ISODate("2016‐06‐01T07:14:11Z"), "updatedAt":ISODate("2016‐06‐01T07:14:12.091Z") }},{ "_id":{ "studentId":"social_user_55", "activity":"http://id.tincanapi.com/verb/viewed", "courseId":"26", "courseName":[ "EDSAOnlineCourses", "FoundationsofBigData" ], "courseUrl":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu", "http://courses.edsa‐project.eu/course/view.php?id=26" ], "activityTriggerId":"http://courses.edsa‐project.eu/mod/url/view.php?id=315", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/url", "activityTriggerDefinitionName":"Yourfeedback", "activityTriggerExternalUrl":"http://courses.edsa‐project.eu/mod/feedback/view.php?id=285&courseid=26", "activityContextEventname":"\\mod_url\\event\\course_module_viewed", "activityContextComponent":"mod_url", "activityContextAction":"viewed", "activityContextObjectTable":"url", "activityContextObjectid":"76", "activityContextOtherId":"N;", "activityContextTimecreated":NumberLong(1459247700), "activityContextOriginatingDevice":"web", "activityContextObject":"course_module", "ip":"86.174.237.52", "timestamp":ISODate("2016‐03‐29T10:35:00Z"), "updatedAt":ISODate("2016‐03‐29T10:35:00.388Z") }},{ "_id":{ "studentId":"social_user_30", "activity":"http://www.tincanapi.co.uk/verbs/enrolled_onto_learning_plan", "courseId":"26", "courseName":[ "EDSAOnlineCourses" ], "courseUrl":[ "http://courses.edsa‐project.eu" ], "courseUrlAccessed":[ "http://courses.edsa‐project.eu"



], "activityTriggerId":"http://courses.edsa‐project.eu/course/view.php?id=26", "activityTriggerDefinitionType":"http://lrs.learninglocker.net/define/type/moodle/course", "activityTriggerDefinitionName":"FoundationsofBigData", "activityContextEventname":"\\core\\event\\user_enrolment_created", "activityContextComponent":"core", "activityContextAction":"created", "activityContextObjectTable":"user_enrolments", "activityContextObjectid":NumberLong(54), "activityContextOtherId":"a:1:{s:5:\"enrol\";s:4:\"self\";}", "activityContextTimecreated":NumberLong(1454701946), "activityContextOriginatingDevice":"web", "activityContextObject":"user_enrolment", "ip":"196.3.88.52", "timestamp":ISODate("2016‐02‐05T19:52:26Z"), "updatedAt":ISODate("2016‐02‐05T19:52:27.107Z") }}

7.3 AppendixC.‐ThirdPartyAPIsandAnalysisToolsUsed

D3.js25isemployedinanumberofvisualisationsdescribedinthisdeliverable,includingtheLandscapefortheVideoLectures.NETportal.Thesnapshotsshowninsection3.2aretakenfromavisualanalyticsprototypebuiltusingD3.js.ThetoolProM26wasusedfortheprocessminingtasks.

7.3.1 VideoLecturesLearningAnalyticsDashboard

Qminer27‐adataanalyticsplatformforprocessinglarge‐scale,real‐timestreamscontainingstructuredandunstructureddata.Theplatformfordatastoring,processinganddashboardserversidesupport.

Node.js28‐aJavaScriptruntimebuiltonChrome'sV8JavaScriptengine.Node.jsusesanevent‐driven,non‐blockingI/Omodelthatmakesitlightweightandefficient.Node.jsisusedforservercreationforlearninganalyticstoolsforVideolectures.NET.

Highcharts29‐graphsforgraphicalimplementationandsupportfordynamicinteraction.

DataTables30 ‐aplug‐in for the jQuery Javascript library. It isahighly flexible tool,baseduponthefoundationsofprogressiveenhancement,andwilladdadvancedinte

25https://d3js.org26http://promtools.org27http://qminer.ijs.si28https://nodejs.org/en29http://www.highcharts.com/about30https://datatables.net

d3.3 content and - european data science...

Documents