![Page 1: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/1.jpg)
The Learning Analytics Landscape Surveying Where we are &
Where we are Going
NERLA 2013 - Second North East Regional Learning Analytics Symposium
Post questions/thoughts using hash tag #NERLA13
Josh Baron Senior Academic Technology Officer
Marist College
![Page 2: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/2.jpg)
POP QUIZ
![Page 3: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/3.jpg)
36%
Reference: Integrated Postsecondary Education Data System (IPEDS)
![Page 4: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/4.jpg)
13% Four Graduation Rate for
Savannah State University (2010)
![Page 5: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/5.jpg)
![Page 6: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/6.jpg)
Presentation Outline
High-Level Overview of Learning Analytics (LA)
Open Academic Analytics Initiative (OAAI)
Peaks on the Learning Analytics Landscape
The Big Questions about Big Data
Group Discussion/Q&A
Post questions/thoughts using hash tag #NERLA13
![Page 7: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/7.jpg)
Analytics in Higher Ed from 50k Feet
Academic Analytics Learning Analytics A process for providing higher education institutions with the data necessary to
support operational and financial decision making*
The use of analytic techniques to help target instructional, curricular, and support resources to support the achievement of
specific learning goals*
Focused on the business of the institution Focused on the student and their learning behaviors
Management/executives are the primary audience
Learners and instructors are the primary audience
* - Analytics in Higher Education: Establishing a Common Language
![Page 8: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/8.jpg)
Different Routes on the LA Landscape
Target Audience Results from analytics
presented to…
Instructor – Decides what, if any, action to take with the learner.
Student – Interprets results & decides on
action to take
Analytics Approach Predictions are based on…
Learning Sciences – Theories and understandings of
student behavior
“Big Data”– Analysis of large data sets of student behavior
Institution
Program
Course
![Page 9: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/9.jpg)
The Open Academic Learning Analytics Initiative (OAAI)
![Page 10: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/10.jpg)
Open Academic Analytics Initiative
EDUCAUSE Next Generation Learning Challenges (NGLC)
Funded by Bill and Melinda Gates Foundations
$250,000 over a 15 month period Goal: Leverage Big Data concepts to create
an open-source academic early alert system.
![Page 11: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/11.jpg)
Student Attitude Data (SATs, current GPA, etc.)
Student Demographic Data (Age, gender, etc.)
Sakai Event Log Data
Sakai Gradebook Data
Predictive Model Scoring
Identifies students “at risk” to not complete
course
SIS
Data
LM
S Da
ta
OAAI Early Alert System Overview
Intervention Deployed “Awareness” or Online
Academic Support Environment (OASE)
Developed model using historical data
“Creating an Open Ecosystem for Learning Analytics”
![Page 12: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/12.jpg)
ONLINE ACADEMIC SUPPORT ENVIRONMENT (OASE)
• OER Content • Self-Assessments • Learning Skills -
Flat World Knowledge
• Learning Support Facilitation & Mentoring
![Page 13: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/13.jpg)
OAAI GOALS AND MILESTONES • Build “open ecosystem” for Learning analytics
• Sakai Collaboration and Learning Environment • Secure data capture process for extracting LMS data
• Pentaho Business Intelligence Suite • Open-source data mining, integration, analysis and reporting tools
• OAAI Predictive Model released under open license • Predictive Modeling Markup Language (PMML)
• Researching learning analytics scaling factors • How “portable” are predictive models?
• What intervention strategies are most effective?
![Page 14: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/14.jpg)
How important is “openness” to the Learning Analytics field? Use hash tag #NERLA13
![Page 15: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/15.jpg)
OAAI RESEARCH METHODOLOGY
• Compared Marist model to Purdue University • Share some common academic characteristics
• Conducting real world pilots • Pilots run in Spring and Fall 2012
• Introductory-level F2F courses
• ~35 courses at community colleges
• ~30 courses at HBCUs
• Conducting student surveys and focus groups
![Page 16: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/16.jpg)
INITIAL FINDINGS - PORTABILITY
• Marist found similar predictive elements and correlations as Purdue
• Student aptitude is strongest predictor
• Cumulative GPA
• LMS elements are secondary predictors
![Page 17: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/17.jpg)
INITIAL FINDINGS - PORTABILITY
Table 3: Prediction Analysis from Spring Pilots
College% of Semester
Completed# of
students Accuracy Recall Specificity Precision
25% 504 66.96% 70.76% 64.64% 55.00%50% 504 71.52% 78.22% 67.56% 59.41%75% 504 77.75% 72.53% 80.94% 69.84%
25% 502 59.13% 69.23% 56.31% 30.73%50% 601 70.92% 66.14% 72.51% 44.44%75% 649 74.77% 74.42% 74.88% 47.76%
25% 195 70.50% 86.27% 61.36% 56.41%50% 195 79.86% 72.55% 84.09% 72.55%75% 195 79.14% 72.55% 82.95% 71.15%
Savannah
Cerritos
Redwoods
Model accuracy remained in the 60-80% range when Marist-derived model was deployed at community colleges and HBCUs.
![Page 18: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/18.jpg)
INITIAL FINDINGS – INTERVENTIONS
• Found statistically significant differences between treatment and control groups • ~7% increase in course grades
• More students “mastered content” (C or better)
• Similar trend among low income students
• Students were more likely to withdraw • Similar to Purdue findings, not necessarily negative
![Page 19: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/19.jpg)
INSTRUCTOR FEEDBACK
"Not only did this project directly assist my students by guiding students to resources to help them succeed, but as an instructor, it changed my pedagogy; I became more vigilant about reaching out to individual students and providing them with outlets to master necessary skills. P.S. I have to say that this semester, I received the highest volume of unsolicited positive feedback from students, who reported that they felt I provided them exceptional individual attention!
![Page 20: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/20.jpg)
FUTURE RESEARCH INTERESTS
• Factors that impact on intervention effectiveness • Intervention Immunity – Students who do not respond
to first intervention tend to never respond
• Student Engagement – How can we increase the level of engagement between students and help resources?
• Can predictive models be customized for specific delivery methods and programs/subjects?
• Can Learning Analytics identify “at risk” students who would otherwise not be identified?
![Page 21: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/21.jpg)
What do you think should be the major focus of future LA research funding? Use hash tag #NERLA13
![Page 22: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/22.jpg)
• Purdue University’s Course Signals • College-wide learning analytics approach
• University of Michigan’s E2Coach • Course-specific learning analytics approach
• UMBC’s “Check My Activity” Tool • Student-centered learning analytics approach
• SNAPP - Social Networks Adapting Pedagogical Practice • Leveraging “big data” visualization
22
TOUR OF PEAKS ON THE LA LANDSCAPE
![Page 23: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/23.jpg)
• Built predictive model using data from… • LMS – Events (login, content, discuss.) & gradebook • SIS – Aptitude (SAT/ACT, GPA) & demographic data
• Leverage model to create Early-alert system • Identify students at risk to not complete the course • Deploy intervention to increase chances of success
• Systems automates intervention process • Students get “traffic light” alert in LMS • Messages are posted to student that
suggest corrective action (practice tests)
23
PURDUE UNIVERSITY’S COURSE SIGNALS
![Page 24: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/24.jpg)
• Impact on course grades and retention • Students in courses using Course Signals…
• scored up to 26% more A or B grades • up to 12% fewer C's; up to 17% fewer D's and F‘s
• Ellucian product that integrates w/Blackboard
24
PURDUE UNIVERSITY’S COURSE SIGNALS
![Page 25: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/25.jpg)
• Focused specifically on introductory Physics
25
UNIVERSITY OF MICHIGAN’S E2COACH
“…to say to each what we would say if we could sit down with them for a personal chat.”
![Page 26: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/26.jpg)
26
EXAMPLE MTS MESSAGE
![Page 27: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/27.jpg)
• UMBC found that students earning D/F’s use Bb 39% less then higher grade achievers • Not suggesting cause and effect • Goal is to model higher achiever
behavior • Provides data directly to student
• Compare LMS use to class averages • Can also compare averages usage
data to grade outcomes • Feedback has been positive
27
UMBC’S CHECK MY ACTIVITY TOOL
![Page 28: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/28.jpg)
• SNAPP (Social Networks Adapting Pedagogical Practice) – UBC/Wollongong • Visualize networks
of interaction resulting from discussion forum posts and replies
28
SOCIAL NETWORK ANALYSIS & VISUALIZATION
![Page 29: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/29.jpg)
THE BIG QUESTIONS…
![Page 30: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/30.jpg)
• “The obligation of knowing” – John Campbell • If we have the data and tools to improve student
success, are we obligated to use them? • Consider This > If a student has a 13% chance of passing a
course, should they be dropped? 3%?
• Who owns the data, the student? Institution? • Should students be allowed to “opt out”?
• Consider This > Is it fair to the other students if by opting out the predictive model’s power drops?
• What do we reveal to students? Instructors? • Consider This > If we tell a student in week three they have a
9% chance of passing, what will they do? • Will instructors begin to “profile” students?
30
LEARNING ANALYTICS ETHICS
![Page 31: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/31.jpg)
What are some of the big ethical questions surrounding LA? Use hash tag #NERLA13
![Page 32: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/32.jpg)
Big Questions about Big Data Could Learning Analytics end up driving the
wrong learning outcomes? What do you see as the biggest ethical
issues surrounding Learning Analytics? How important is “openness” in LA? Open
standards? Open licensing? If you were Bill and Melinda Gates what
would your funding priorities be around LA? Where is LA in the “hype cycle”? Where will
it end up?
![Page 33: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/33.jpg)
LEARNING ANALYTICS AND KNOWLEDGE CONFERENCE (LAK) THE INTERNATIONAL JOURNAL OF THE SOCIETY FOR LEARNING ANALYTICS RESEARCH SOLAR FLARES – REGIONAL CONFERENCES SOLAR STORMS – DISTRIBUTED RESEARCH LAB MOOCS ON LEARNING ANALYTICS http://www.solaresearch.org/
33
![Page 34: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/34.jpg)
More Questions?
Josh Baron Senior Academic Technology Officer
Marist College [email protected]
@JoshBaron
![Page 35: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/35.jpg)
OASE DESIGN FRAMEWORK
• Guiding design principals that allow for localization
• Will be releasing under a CC license
• Follows online course design concepts
• Learner – Content Interactions
• Learner – Facilitator Interactions
• Learner – Mentor Interactions
• Leverages Open Educational Resources (OER)
![Page 36: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/36.jpg)
DESIGN FRAME #1 LEARNER-CONTENT INTERACTIONS
• Self-Assessment Instruments • Assist students in identify areas of weakness related to
subject matter and learning skill • OER Content for Remediation
• Focus on core subjects (math, writing) • Organized to prevent information overload
• “Top rated math resources”
• OER Content for Improving Learning Skills • Focus on skills and strategies for learner success
• Time management, test taking strategies, etc.
![Page 37: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/37.jpg)
DESIGN FRAME #2 LEARNER - FACILITATOR INTERACTIONS
• Academic Learning Specialist role would • Connecting learners to people and services
• Promoting services and special events
• Moderates discussions on pertinent topics • Example: “Your first semester at college”
• Guest motivational speakers • Occasional webinars with upperclassman, alumni, etc.
• Allows learners to hear from those who “made it”
![Page 38: The Learning Analytics Landscape - NERCOMP › uploadFiles › 3898000001E8.filename.Learning... · College-wide learning analytics approach • University of Michigan’s E. 2. Coach](https://reader036.vdocuments.net/reader036/viewer/2022081402/5f0c4b967e708231d434b0c0/html5/thumbnails/38.jpg)
DESIGN FRAME #3 LEARNER - MENTOR INTERACTIONS
• Online interactions facilitated by student “mentor” • Facilitates weekly “student perspective” discussions
• Example: “Your first semester of college – the real story”
• Online “student lounge” for informal interactions • Let others know about study groups, etc.
• Help build a sense of community
• Blogs for students to reflect on experiences • Could be public, private or private to a group