learning analytics platform, towards an open scalable ... · learning analytics platform, towards...

Learning Analytics Platform, towards an open scalablestreaming solution for education

Nicholas LewkowMcGraw-Hill Education

281 Summer St.Boston, MA

[email protected]

Neil ZimmermanMcGraw-Hill Education


[email protected]

Mark RiedeselMcGraw-Hill Education


[email protected]

Alfred EssaMcGraw-Hill Education


[email protected]

ABSTRACTNext generation digital learning environments require deliv-ering just-in-time feedback to learners and those who sup-port them. Unlike traditional business intelligence environ-ments, streaming data requires resilient infrastructure thatcan move data at scale from heterogeneous data sources, pro-cess the data quickly for use across several data pipelines,and serve the data to a variety of applications. As a solu-tion to this problem, we have designed and deployed intoproduction the Learning Analytics Platform (LAP), whichcan ingest data from different education systems using stan-dardized IMS Caliper events. The education events are trig-gered by student and instructor activity within Caliper in-strumented learning systems. Once sent to the LAP, eventsare transformed and stored in a data store where they canbe used for student, educator, and administrator visualiza-tions as well as education driven analytics research. TwoMcGraw-Hill Education platforms, Connect, used for highereducation, and Engrade, for K-12, are currently instrumentedto send the LAP event data which in turn feeds visualiza-tions for educational insight. Future plans for the LAP in-clude collection of education event data from a wide varietyof proprietary and open source education platforms, compu-tational engines for predictive analytics, and an open APIfor third-party analytics using LAP data.

KeywordsLearning Analytics, Event Processing, Heterogeneous Data,Streaming Data, Parallel Architecture

1. INTRODUCTIONIt is the goal of next generation digital learning systems touse big data and analytics to advance learning outcomes.These next generation systems should be able to providejust-in-time feedback to students and educators with an aimto increase the efficiency and effectiveness of digital educa-tion. Further, with the increasing instrumentation of all dig-ital media, digital learning environments should be instru-mented in a way that allows important education data to becollected in a standardized fashion for both real-time andafter-the-fact (batched) data analysis. This task requireslarge scale processing of streaming data utilizing massivelyparallel architectures which may ingest, store, and analyzedata in real-time.

Our solution to this problem is the Learning Analytics Plat-form (LAP). The LAP is designed to ingest educational datafrom present and future education platforms in the form ofstandardized events using the IMS Caliper spec [2]. Twoexisting education platforms, Connect for higher educationand Engrade for K-12, have already been instrumented tocreate and ship Caliper education events to the LAP. Onceingested by the LAP, the data is transformed and stored forbuilding of visualizations used for educational reports [4].These ‘insights’ include several real-time statistics for stu-dents and educators including time-spent, outcomes, sub-mission times near due dates, attendance, and class perfor-mance comparisons. Additionally, messages indicating neg-ative trends, such as repeatedly starting assignments nearor after the due date, are also presented to the user.

To meet the requirement of providing just-in-time feedbackto students and educators, an analytics platform must alsobe have a parallel architecture which may effectively ingeststreaming data. There are several proposed requirementsfor a streaming data architecture, including the ability tohandle data imperfections, generate predictable outcomes,and to guarantee data safety and availability [5]. Addition-ally, the architecture should be automatically scalable andfault tolerant for both software and hardware failures, par-ticularly, it must not lose any event data under any circum-

Proceedings of the 8th International Conference on Educational Data Mining 460

stance. Our architecture has the additional requirementsof ingesting data from heterogeneous sources and perform-ing data transformations using different data pipelines. TheLAP fulfills all of the above requirements in its current ver-sion with further refinements and additions planned for thenear future.

Details about the LAP are discussed in the following sectionsincluding information about the standardized data formatwhich was used, IMS Caliper events, as well as specifics onthe LAP architecture and performance. Information regard-ing future versions of the LAP is also discussed followed byconcluding remarks.

2. STANDARDIZED CALIPER EVENTSWith the continual adoption of digital education systems aglobal standard for educational event data, which is gener-ated from a large diversity of heterogeneous systems, hasbecome increasingly sought after. While there has been ad-vancement in this area by the Tin Can API [1], the IMSGlobal Learning Consortium has proposed a schema-drivensolution to this problem with their Caliper event spec [2].JSON-LD (linked data) is used for the Caliper events as away to link specific, normalized fields within a set of events[3]. Using the Caliper events, data from heterogeneous learn-ing systems can be created, transmitted, and collected foranalysis in a global and standardized fashion.

Caliper events strive to create a generalized framework thatcan be utilized by all types of learning events ranging from astudent using an interactive education tool, such as a learn-ing game, to an educator recording attendance in their class.The Caliper events are based on the data triple of “Actor”- “Verb” - “Object”. As an example, the event for a stu-dent submitting an assessment (homework, quiz, test, etc.)would have the Actor be the student, the Verb be the sub-mission of an assessment, and the Object would containinformation on the assessment being submitted, potentiallyhow long it took to complete.

In order to utilize the Caliper event spec, learning platformsmust be instrumented to create events when actions occur byeither students or educators. Currently the Connect and En-grade systems are instrumented to to create Caliper eventswhen actions occur, and to send the events to the LAP.Instrumentation is unique to the system in question andgreatly depends on how that system’s data is stored. In thecase of Connect, Caliper events are created through a seriesof database triggers when actions are taken by students. Thedatabase triggers are automated to create the Caliper eventsusing tables from their system databases when new infor-mation is passed to the system from a user. Future plansinclude instrumentation of several new external systems, al-lowing for increasingly rich data in the LAP for analysis andvisualization.

3. ARCHITECTUREIncreasing instrumentation of sensors and digital media re-quires streaming analytics architectures to analyze data inreal-time. Attention was paid to the development and designof the LAP architecture to ensure that it met all require-ments of a parallel streaming system containing both a datastore and an analytics engine. Key features include auto-

scaling fundamental architecture components with varyingload, fault tolerance for both hardware and software fail-ures, and the ability to process data and have it availablefor output API access immediately.

In the simplest of descriptions, the LAP is designed to:

1. Receive learning events from external applications throughan ingestion API.

2. Send raw events directly to long-term storage.

3. Validate each event for expected fields and types.

4. Process the events, which requires application depen-dent data transformations.

5. Store those events in the data store according to ap-plication dependent schema.

6. Query the data store and perform transformations andaggregations as needed for the output API.

Figure 1 displays a high level view of the LAP architecturewith learning events from three separate external applica-tions coming into the LAP. Once the events are received bythe LAP they are transformed according to an applicationspecific schema and stored in the data store. When a userrequests an insight visualization, the LAP output API iscalled by the external application. This triggers the resultsand analytics service to query the data store, aggregate andtransform the data as needed, and pass it to the insight ser-vice where the data is used to build the appropriate insightvisualization which is then passed to the user.

The ingestion API and collection service are configured toreceive IMS Caliper events sent from external applicationsthrough sensors which have to be implemented in those ex-ternal applications. To maximize performance over high-overhead HTTP, several events are sent simultaneously inan event container. The number of events sent in a singlecontainer can range from one to tens of thousands. Oncea container of events is received it is immediately sent toa long term storage system for backup purposes. The con-tainer is then opened and the events within are validated,transformed into an application specific schema, and sent tothe data store.

The data store utilizes the non-relational database Mon-goDB, which allows for great flexibility in data model schema.Events from external applications are not guaranteed tocome into the LAP in chronological order so the data modelschemas were developed to create a deterministic data storestate from events coming in arbitrary order. The data modelemployed for the current two applications consists of a twoschema model; one document type holds the student leveldata and the other holds the class level data. Building ofthe insight visualizations then requires the results servicequerying N student documents for a class with N students,and 1 document per class for a total of N + 1 documents.

The insight service is built external to the LAP to providethe user with the requested analytical visualization. Oncethe data store is queried and the appropriated documents are


Figure 1: High level view of the LAP architecture. Learning events are passed into the LAP through theingestion API and visualizations ’insights’ are produced through the insight service calling the LAP’s outputAPI. Internally, the LAP consists of a collection service, a data store, long-term storage, and a results service,which also performs analytics. Several instances of the collection and results services run in parallel.

returned, they are aggregated, transformed, and returnedto the output API to provide the client with data to buildthe visualization. This process is very lightweight and fast,allowing for quick feedback and response to the user whohas just finished an assessment.

The technology used for the LAP was chosen to be lightweight,reliable, and adaptable for future system iterations. Thecollection, results, and insight services were all built usingNode.js which allows for asynchronous communication runin parallel across several service instances. The analyticalvisualizations were built using JavaScript with AngularJSand D3.js frameworks which provide fast, responsive, andinteractive interfaces that are customizable. The data storewas built using MongoDB with a three member replica set,allowing for highly available data and low latency accessfor both read and write operations. Amazon Web Services(AWS) were used to host the LAP utilizing load balancingtechnologies routed to EC2 instances. An AWS S3 bucketwas used for long-term storage of the raw event containers.Details pertaining to the performance of the LAP as well asfuture plans for the system follow.

4. PERFORMANCEPerformance is very important to the success of the LAP. Itis imperative that the LAP be able to ingest data from sev-eral external systems during their peak times simultaneouslyin a manner which does not delay the real-time analysis onthe output of the system.

In testing the performance of the LAP, two main pointswithin the system were identified as the potential bottle-necks. The first of these points is the ingestion of eventsfrom external applications into the LAP while the second isthe querying, aggregation, and analysis done by the resultsservice prior to returning data to the output API.

For the current two external systems sending data to theLAP, our anticipated peak load is around 0.1 MB/sec. While

this load is not particularly large, future plans include in-gesting events from many more external systems, so the de-sired performance should easily be at least ten times largerat around 1 MB/sec. To test the performance from eventreception to data store insertion, an automated script wasdeveloped which creates 10 threads with each sending a se-ries of containers with varying numbers of events to the LAPrunning three instances of the collection service. The pro-cessing time, from data being sent from an external appli-cation to be inserted into the data store, was then mea-sured as a function of number of events per container. Fig-ure 2 displays the results of this test with collection ratesas MB/sec and the number of events per container rangingfrom 1 to 1000. The results shown in Figure 2 are informa-tive for a few reasons. First, it is clear that high collectionrates into the system requires more than one event to besent per container. In particular, the LAP can process 0.1MB/sec or more if the events are sent with at least 4 percontainer. To reach our desired bandwidth of ten times ourcurrent peak, 1 MB/sec, requires sending about 75 eventsper container or more with three collection instances. Thesecond realization from Figure 2 is that the collection rateof the system somewhat flattens out between 500 and 1000events per container, making it less efficient to process theselarger containers. This effect ultimately has to be weighedagainst the network bandwidths in sending events from ex-ternal applications and has not yet been tested. It shouldalso be mentioned again that these performance tests weredone with three instances of the collection service. Increasedrates could always be achieved by increasing the number ofcollection service instances, but these tests were done to de-termine collection rates for a static number of instances.

The second potential bottleneck in the LAP is the queryingof the results service and the load on the output API. Cur-rently the LAP is in a trial mode with tens of thousands ofusers. For this relatively low number of users the load onthe output API is not a major concern and detailed perfor-mance testing has not yet been done. The implementation


100 101 102 103

Events per Container

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8LA

P Colle

ction Rate [MB/sec]

Figure 2: The collection rate of Caliper events, in[MB/sec], processed by the LAP is shown as a func-tion of number of events per container. 10 parallelthreads were used to send data to the LAP duringthis test, simulating the load from several externalsystems sending events simultaneously. The systemtested had 3 instances of the collection service op-erating in parallel.

of auto-scaling within AWS, however, should easily handlelarge spikes in front end usage during peak hours.

5. FUTURE VERSIONSThe current LAP is the first iteration of a production sys-tem. It supports two external education systems, can sup-port more than ten times the anticipated peak load, andhas a modest analytics layer within its architecture. Thenext version of the LAP is currently being developed withthe goal of supporting many more external systems with to-tals of tens of millions of users. In addition to increaseduser load, the future plans for the LAP include a substan-tial computational layer, opening up the possibility for richeranalytics, as well as an open API for third party analyticsto be done using LAP data.

The initial success of the current version of the LAP hasled to plans for instrumentation of several new educationalsystems so their data may be ingested by the LAP. To beable to handle the increased numbers of users with the LAP,several changes and additions are needed. The most drasticof these changes is switching to a completely AWS system,fully utilizing Amazon cloud technologies [6]. Moving theLAP to a full AWS stack will allow for massive scalabilityand the ability to store data, perform analytics, and givesupport to millions of students, educators, and researchers.Further, future versions of the LAP will also have the abilityto perform more advanced analytics including predictive an-alytics, machine learning algorithms, and large distributedcalculations and aggregations. Incorporating a distributedcalculation layer into the LAP will allow for a richer set ofanalytics to be performed and thus give the ability for deeperinsight into large educational data sets.

Implementation of an open API for data consumption and

analytics by third parties is also planned for the LAP. Oneof the intended features of the LAP is the lack of PII dataheld within the data store. The de-identified of the LAPdata allows for third parties to ingest and do analysis on ourdata without concern for privacy. Creating an open API forthe LAP will help push the fields of learning analytics andeducational science by allowing researchers greater access tostudent and educator data.

6. CONCLUSIONSWe have built a platform able to ingest, store, and ana-lyze data from external learning applications in a scalablefashion. Two existing applications, Connect and Engradehave been instrumented to create and send standardizedCaliper learning events to the LAP. Once received, the learn-ing events are transformed within customized data pipelinesand stored within a fast data store, implemented using non-relational MongoDB. Analysis is done on the stored learningevents, creating visualizations of educational insight for stu-dents and educators. The architecture of the LAP allowsfor just-in-time feedback with the insight visualizations toits users. The current version of the LAP is able to processand store education events ten times faster than is requiredfor peak usage by the current two applications interfacedwith the LAP. Future versions are planed for the LAP andwill include a complete backend stack which is hosted byAWS and able to auto-scale across the entire platform. Ad-ditionally, an advanced computational layer and open APIare planed for future versions of the LAP. It is our visionthat present and future iterations of the LAP may providethe analysis, high quality educational data, and predictiveanalytics to drive the next generation of education.

7. ACKNOWLEDGMENTSThis paper is based on work supported by the McGraw-Hill Education Digital Platform Group (MHE DPG). Wewould like to extend our appreciation for all the help andthe informational support provided by the Analytics team atDGP and the MHE CDO Stephen Laster. Despite providedsupport, any opinions, findings, conclusions or recommenda-tions expressed in this paper are those of the authors and donot necessarily reflect positions or policies of the company.

8. REFERENCES[1] Experience API Version 1.0.0. Technical report,

Advanced Distributed Learning, 2013.

[2] Learning Measurement for Analytics. Technical report,IMS GLOBAL Learning Consortium, 2013.

[3] JSON for Linking Data, 2015. http://json-ld.org/.

[4] J. Feild. Improving student performance using nudgeanalytics. In Educational Data Mining, 2015.

[5] M. Stonebraker, U. Cetintemel, and S. Zdonik. The 8Requirements of Real-Time Stream Processing.SIGMOD Record, 34(4), 2005.

[6] E. Swensson. Big Data Analytics Options on AWS.2014.


learning analytics platform, towards an open scalable ... · learning analytics platform, towards...

Documents