simplifying learning analytics via the caliper analytics framework (288361849)
TRANSCRIPT
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 1/39
Rob AbelChief Executive
IMS Global Learning [email protected] , @LearningImpact
Simplifying Learning
Analytics via theCaliper AnalyticsFramework
Vince KellenSenior Vice Provost for Analytics and Technologies
University of [email protected] , @vkellen
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 2/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Agenda1. Brief Overview of Learning Analytics
2. Progress of IMS Caliper Analytics3. Predictions4. Challenges and Steps to the Future
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 3/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Agenda1. Brief Overview of Learning Analytics
2. Progress of IMS Caliper Analytics3. Predictions for Next Year Progress4. Challenges and Steps to the Future
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 4/39
Source: CETIS: Adam Cooper; A brief history of analytics
http://publications.cetis.ac.uk/wp-content/uploads/2012/12/Analytics-Brief-History-Vol-1-No9.pdf
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 5/39
Source: EDUCAUSEwww.cdn.educause.edu/visuals/shared/er/ERM1151/ERM1151_table.jpg
PersonalizedLearningandStudentSuccess
FacultyandInstitutionalPerformance
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 6/39
Learning Analytics : Does student appear to be participating and/orunderstanding a key concept?
Academic Analytics : Is the performance of a cohort of students athistorical norms?
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 7/39
Source: EDUCAUSEwww.cdn.educause.edu/visuals/shared/er/ERM1151/ERM1151_table.jpg
IMSFocus
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 8/39
“Analytics” is one of the hottest buzzwords ineducation. For ed-tech companies, it is also aselling point. By using vendors’ suites andsolutions, colleges will gain access to dataabout why some students succeed and whereand when others stumble -- or so the pitchgoes.
But there’s a catch, said Linda Feng, senior
product manager for analytics and SISintegration at Instructure . “The whole premiseis that all that data has to be in their world,”she said.
“All these systems have their own languages
that are written for their own needs,” saidJohn Whitmer, director for platform analyticsand research at Blackboard. “In order to talk acommon language, we have to agree to acommon vocabulary .”
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 9/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
The Future of the “LMS” aka The EdTech Ecosystem
Easier/BetterFunctionality
Easier/BetterInsight
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 10/39
© 2015 IMS Global Learning Consortium, Inc. All Rights ReservedPage 10
Advancing
TechnologyUsage
EducationalProcess + ResultsImprovement
Teacher/Student/InstitutionalProductivity Gain
Learning ImpactEdTech MaturityModel
E d u c a
t i o n a
l V a
l u e
Evolving Expectations
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 11/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Agenda1. Brief Overview of Learning Analytics
2. Progress of IMS Caliper Analytics3. Predictions for Next Year Progress4. Challenges and Steps to the Future
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 12/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Caliper Facilitated by IMS Global
12
• World’
s primary K-20 educational technology standards memberconsortium: 20 year history, 330+ member organizations (institutions& suppliers), accelerating growth & adoption
• 25+ IMS standards enable IT/academic technology flexibility,
efficiency and agile innovation in a time of rapid change• IMS works closely with our community on adoption of new
technologies and building awareness through Learning ImpactLeadership Institute conference and awards program (May)
imsglobal.org @LearningImpact
0
50
100
150
200
250
300
350IMS Global Learning Consortium Growth
IMS MemberOrganizationsEnd of Year
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 13/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
EDU
OPERATIONSPLATFORMS
MOBILE EDU APPS
PUBLISHER /COURSEWARE SOLUTIONS
LEARNINGPLATFORMS
S
EDU TOOLS /SERVICES
S
S
S
S
InstructorsStudents
Adaptive Personalize
CurriculumLearningResearch
S = Sensor
EP = Sensor EndPoint
IMS Caliper EnabledServices
● Caliper Sensor Endpoint(s)● Compliant Data Mgmnt Services
DataManagement
Services
EPData
Stores
Admins
Caliper Learning Analytics Ecosystem
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 14/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Why Caliper?Standardization : encourage a common, extensible yet structuredapproach to describing, collecting and transporting learner interactiondata for later consumption by researchers, educators, platforms, appsand services.
Innovation: provide a extensible data model, controlled vocabularies andan API that enables new uses of learner interaction data.
Interoperability : promote data exchange, sharing, mashups between
systems, institutions and people: learning analytics ecosystem
Stewardship : evolve an EDU-optimized technical specification under theauspices of IMS Global and its member institutions and organizations.
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 15/39
“Caliper opens up access to meaningful data”
“Caliper is built around the IMS Learning Sensor API to define basic eventsand standardize metrics across different learning environments, and alsouses IMS LTI standards to integrate standardized measurement with toolsinteroperability. Put together, Caliper is less about analytics, and moreabout the data standardization that will eventually drive learning analyticsefforts .”
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 16/39
Caliper v1.0 Certified Suppliers!
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 17/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Caliper event
action
actor
object
target
generated
edApp
group
membership
startedAtTime
endedAtTime
duration
Learning Context
Activity Context
Time (ISO 8601)
Base statement
fedSession
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 18/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
IMS Caliper: Click Stream Data forLearning Analytics
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 19/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Caliper fundamentalsInformation model : standardized set of learning activityprofiles & controlled vocabularies that extend set offoundational metrics.
API: governs the interactions between Caliper, apps,platforms, services, event store(s).Sensor : code library designed to simplify instrumentation ofplatforms/apps/services. Multiple bindings: Java, JS, Ruby,Python, PHP, .Net.
Conformance: test suite ensuring implementationcompliance.Dev community: docs, presentations, bootcamps, referenceimplementations.
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 20/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Scope: Caliper
describe
collecttransport store
analyzevisualizereportsharepredictinterveneadapt
instrument persist use
feedback
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 21/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Scope: Caliper 1.0
describe
collecttransport store
analyzevisualizereportsharepredictinterveneadapt
instrument persist use
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 22/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Caliper code repos (Github)
sensorscaliper-javacaliper-jscaliper-ruby
caliper-pythoncaliper-php (includesexample)caliper-net
supportcaliper-contextscaliper-conformancecaliper-common-fixtures
future?caliper-model/profilescaliper-docs
sample codecaliper-java-examplecaliper-net-example
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 23/39
Unizin is a Major Caliper Initiative
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 24/39
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 25/39
“There’s a danger that the vendor communitycan say, ‘This is our data, not yours, and youhave to pay us to learn how you teach
students,’” Kellen said. “I disagree with that.The academy needs to have the line of sight tothe student in digital form.”
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 26/39
Caliper Institutional Leadership Council(Caliper Real-time Analytics Messaging*)
*Caliper RAM = Processing thousands of events per second and responding withintervention messages within one second
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 27/39
Real-TimePersonalized Messaging
Tai lo r e d C o m m u n i c a t io n sf o r S t u d en t S u c c es s
Vince Kellen, Ph.D., Senior Vice Provost, Analytics and TechnologiesPatsy Carruthers, Director of Academic TechnologyChris Rice, Enterprise Architect, Academic Technologies
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 28/39
Caliper RAM Purpose:Integrate data needed to personalized learning and analyze
student success analytics in one real-time architecture• Real-time personalized interactions
• Tailor and target reminder services, avoid over messaging, enable timing of message deliverybased on user temporal proclivities, mix and match messages across learning, support andprogression areas
• Enrich the university’s ability to analyze and research both student success and student learning • Deliver targeted, on- demand peer tutoring based on the student’s profile • Target micro-surveys and assessments to capture additional information needed to improve
personalization• Give students ‘academic health’ recommendations on how they can improve in study,
engagement, support, etc.• Potentially let students opt their parents, friends in to this information so the student’s network
can help
• Allow for open personalized learning
• How content gets matched to students is psychologically complex• Several theories of how humans learn give many insights• Students differ in the following abilities and attributes: visual-object, visual-spatial, reasoning,
cognitive reflection, need for sensation, need for cognition, various verbal abilities, confidence,persistence, prospective memory, etc.
• We need an open architecture to promote rapid experimentation, testing and sharing of whatworks and what doesn’t
• We need real- time messaging so that learning interventions can ‘fit’ within the ‘window’ of short -term memory University of Kentucky
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 29/39
Confusion Detector I A student performs poorly on homework questions
aligned to a Basic Chemistry Learning Outcome,and misses 3 out of 5 in-class audience-responsequestions in class the following day.
Real-time analytics detects that the student has performedsimilarly poorly on questions and homework assignments alignedwith that learning outcome.
The real-time analytics service lets the student know that theyappear to be having trouble with the Learning Outcome inquestion (link to syllabus in Canvas/LMS).
Student is informed that a peer tutor is available and will becalling shortly.
Student receives supplementary information and links toadditional exam prep questions related to the Learning Outcome.
Peer tutor discusses the Learning Outcome and relatedconcepts with the student. Follow-up meetings can be arranged.Real-time analytics service places an entry into the students
advising log and notifies the faculty.If the student underperforms on questions aligned to that
learning outcome in the online examination software, repeatsteps 2-6 immediately following the exam, including a reminder
that (if exam is cumulative) that they will see this material againon the final exam.
Real-timeevent stream
“Uber Tutor” Person-to-person
Confused student
Real-time analytics Peer tutor
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 30/39
Confusion Detector IIIn the middle of working on an onlinehomework assignment, a General Chemistrystudent searches for [key chemistry concept]in the online course content, replays theassociated segment in the video lecture andhas gotten 2 out of 3 in-class audienceresponse questions incorrect in the priorlab/lecture related to this topic.
1.Through NLP content analysis, real-time analytics detectthe confusion regardless of the type of content andnotifies an Uber (peer) Tutor.
2.The real-time analytics service lets the student knowthat a peer tutor is available and will be calling
3.The peer-tutor and the student discuss the [keychemistry concept] and arrange to meet later to explain
4.Homework system displays follow-up questionstriggered by student check-in at the Uber Tutor meeting.
5.The real-time analytics service places a entry in thestudents advising log and notifies the faculty
Real-timeevent stream
“Uber Tutor” Person-to-person
Confused student
Real-time analytics Peer tutor
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 31/39
Early Alert
Several freshmen Chemistry students arestruggling with the material in the first few days.Each one scores low or fails the first formativeassessment.
1. Real-time analytics detects an LMS quiz or in-classaudience response system score below an instructor-
defined value and alerts each student and theinstructor of those grades.
2. The real-time analytics service sends an alert to thestudent and instructor of early difficulty withintroductory material.
3. Real-time analytics message to instructor triggersmessage from instructor to student featuring remedialmaterial and referral to The Study.
4. The real-time analytics service places a entry in thestudents advising log for follow-up
Real-timeevent stream
Mobile feedmessage
re or failure on early assessment
Real-time analytics LMS Mobile Feed
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 32/39
Class engagementSome students in a large-enrollment course
have delayed logging in to access readingsand video, placing them in danger of losingmomentum as the course begins. Introductoryposts on the discussion board are very brief.
1. The real-time analytics service detects low
recency and frequency of clickstream activitycombined with few or short, disengaged discussionresponses
2. The real-time analytics service sends apersonalized message in the learning systemstudent home page and messages the student
mobile feed3. Student accesses instructor’s recommendationsand content links contained in message andconsiders how to improve his engagement
4. The real-time analytics service places a entry inthe students advising log and notifies the faculty,
with an optional entry in the student academic alertsystem
Real-timeevent stream
Remindersand resources
Confused student
eal-time analytics
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 33/39
Abrupt class disengagement behavior
A student who has been engaged in class
experiences a family issue that causes a loss ofpersonal or financial support, and as a resultmust turn attention to those problems.
1. Through content analysis, real-time analytics detecta sudden drop-off of clickstream in the LMS, audience
response systems, use of self-service tools and classattendance
2. Real-time analytics notifies faculty and sendsmessages of concern and offer of assistance ispushed to student learning system homepage,student mobile feed and, optionally, to the student
mobile phone
3. The student finds multiple messages of concern andcontacts instructor or adviser to discuss
4. The real-time analytics service places a entry in thestudents advising log and sends an alert to the
student academic alert system
Real-timeevent stream
Messageof concern
Worried student
Real-time analytics Mobile message
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 34/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Agenda1. Brief Overview of Learning Analytics
2. Progress of IMS Caliper Analytics3. Predictions for Next Year Progress4. Challenges and Steps to the Future
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 35/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Next 12 Months• Caliper will be adopted by a majority of the global market-leading
learning platforms• Caliper will be used by several major publishers to collect millions of
events per week to analyze and improve quality of learning materials• Several universities will implement the first ever real-time analyticsmessaging at scale based on Caliper RAM
• Caliper profile to measure adoption/use of e-text and popular learningtools will be in production use by a first wave of leading U.S. universities
and their suppliers• Synergies between learning analytics and digital credentialing/badges will
emerge in the context of enabling student success
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 36/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Agenda1. Brief Overview of Learning Analytics
2. Progress of IMS Caliper Analytics3. Predictions for Next Year Progress4. Challenges and Steps to the Future
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 37/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
Learning Analytics Challenges
• Need to start with good data
• Need to have a clear goal
• Need organizational culture and capacity to enact action• To achieve any of the above in education, motivation for
improvement must be compelling
• “Good judgment” is still required – as much an art as a science
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 38/39
© 2015 IMS Global Learning Consortium, Inc. All Rights Reserved
How to Take Advantage of Caliper?
38
Collaborate with IMS communityInstitutional analytics leadership group
Supplier technical development and supportContact Rob [email protected]
Ask for Products IMS Caliper CertifiedRequire IMS Caliper conformance certification (play your
part in making sure the ecosystem is interoperable)imscert.org
8/20/2019 Simplifying Learning Analytics via the Caliper Analytics Framework (288361849)
http://slidepdf.com/reader/full/simplifying-learning-analytics-via-the-caliper-analytics-framework-288361849 39/39
Thank You!
[email protected] @LearningImpact
[email protected] , @vkellen