learning analytics bett2013
TRANSCRIPT
Learning Analytics: Unlocking student data for 21st century learning?
Simon Buckingham Shum Knowledge Media Institute The Open University UK simon.buckinghamshum.net
BETT 2013, London — LearnLive HigherEd
@sbskmi #LearningAnalytics
2
70-strong lab prototyping next generation learning / sensemaking / social web media
linked data / semantic web services
learning objective:
walk out with
better questions than you can ask right now
3
Why are seeing this?
4
Why are seeing this?
5
Why are seeing this?
6
edX: “this is big data, giving us the chance to ask big questions about learning”
7 https://www.edx.org/about
A recent analytics product review…
8
A recent analytics product review…
“Some have tried to argue that this technology doesn't work out cost effectively when compared to conventional tests... but this misses a huge point. More often than not, we test after the event and discover the problem — but this is too late..”
9
Aquarium Analytics!
10
11
How is your aquatic ecosystem?
“This means that the keeper can be notified before water conditions directly harm the fish—an assured outcome of predictive software that lets you know if it looks like the pH is due to drop, or the temperature is on its way up.
This way, it’s a real fish saver, as opposed to a forensic examiner, post-wipeout.”
(From a review of Seneye, in a hobbyist magazine) 12
How is your learning ecosystem?
This means that the teacher can be notified before learning conditions directly harm the students — an assured outcome of predictive software that lets you know if it looks like engagement is due to drop, or distraction is on its way up.
This way, it’s a real student saver, as opposed to a forensic examiner, post-wipeout.
13
14
but you still need to know what good looks like…
and what to do when it drops…
15
16
fish
learners?
Purdue University Signals: real time traffic-lights for students based on predictive model
17
Purdue University Signals: real time traffic-lights for students based on predictive model
18
Predicted 66%-80% of struggling students who needed help
MODEL: • ACT or SAT score • Overall grade-point average • CMS usage composite • CMS assessment composite • CMS assignment composite • CMS calendar composite
Campbell et al (2007). Academic Analytics: A New Tool for a New Era, EDUCAUSE Review, vol. 42, no. 4 (July/August 2007): 40–57. http://bit.ly/lmxG2x
Purdue University Signals: real time traffic-lights for students based on predictive model
19
“Results thus far show that students who have engaged with
Course Signals have higher average grades and seek out help
resources at a higher rate than other students.”
Pistilli, M. D., Arnold, K. and Bethune, M., Signals: Using Academic Analytics to Promote Student Success. EDUCAUSE Review Online, July/Aug., (2012). http://www.educause.edu/ero/article/signals-using-academic-analytics-promote-student-success
Enabling staff to monitor courses and student academic success predictions
View profiles showing predictions of academic success in relation to success factors and cohort
Chris Ballard, Tribal Labs / @chrisaballard / www.triballabs.net
Predictive model relates predictions to student success factors to help staff identify interventions
Understand patterns of student activity and engagement with
university services
Chris Ballard, Tribal Labs / @chrisaballard / www.triballabs.net
predictive models are exciting
but there are many other
kinds of analytics
22
Analytics in your VLE: Blackboard: feedback to students
23
http://www.blackboard.com/Platforms/Analytics/Products/Blackboard-Analytics-for-Learn.aspx
24
https://grockit.com/research
Adaptive platforms generate fine-grained analytics on curriculum mastery
a data-centric culture doesn’t have to involve advanced technology
25
Emerging interest in learning analytics Professor Mark Stubbs | [email protected]
• Why? Make better decisions Example: Choosing a new VLE:
• Seek to correlate variables with final success/failure • Triangulate with extensive survey and focus groups • Result: Critical Success Factors inform
requirements for new VLE
MMU exploring
since 2010 …
Entry qualifications
VLE usage patterns
Exam results
Learner demographics
… planning
institution-wide
support for 2013
analytics for lifelong, lifewide learning?
27
Why do dispositions matter?
28
“Knowledge of methods alone will not suffice: there must be the desire, the will, to employ them. This desire is an affair of personal disposition.”
John Dewey
Dewey, J. How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process. Heath and Co, Boston, 1933
Validated as loading onto 7 dimensions of “Learning Power”
Changing & Learning
Meaning Making
Critical Curiosity
Creativity
Learning Relationships
Strategic Awareness
Resilience
Being Stuck & Static
Data Accumulation
Passivity
Being Rule Bound
Isolation & Dependence
Being Robotic
Fragility & Dependence
Univ. Bristol and Vital Partnerships provides practitioner resources and tools to support their application in schools, HEIs and the workplace 29
ELLI: Effective Lifelong Learning Inventory Web questionnaire 72 items (children and adult versions: used in schools, universities and workplace)
30
Analytics for lifelong/lifewide learning dispositions: ELLI
Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, Vancouver). Eprint: http://oro.open.ac.uk/32823
32
ELLI generates cohort data for each dimension
EnquiryBlogger: Tuning Wordpress as an ELLI-based learning journal Piloting from Yr 5, to secondary, to Masters level
33
Standard Wordpress editor
http://learningemergence.net/tools/enquiryblogger
EnquiryBlogger: Tuning Wordpress as an ELLI-based learning journal Piloting from Yr 5, to secondary, to Masters level
34
Categories from ELLI
http://learningemergence.net/tools/enquiryblogger
Plugin visualizes blog categories, mirroring the ELLI spider. Direct
navigation to blog posts from here
EnquiryBlogger: Tuning Wordpress as an ELLI-based learning journal Piloting from Yr 5, to secondary, to Masters level
35
EnquiryBlogger dashboard – direct
navigation to learner’s blogs from the visual
analytic
LearningEmergence.net more on analytics for learning to learn, authentic enquiry, leadership and complex learning systems
37
unpacking deeper learning example:
online student discourse
analytics that go beyond “number of forum posts”
+ “trending topics” 38
Social Network Analysis (SNAPP)
39 Bakharia, A. and Dawson, S., SNAPP: a bird's-eye view of temporal participant interaction. In: Proceedings of the 1st International Conference on Learning Analytics and Knowledge (Banff, Alberta, Canada, 2011). ACM. pp.168-173
What’s going on in these discussion forums?
Social Network Analysis (SNAPP)
40 http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation
Social Network Analysis (SNAPP)
41 http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation
2 learners connect otherwise separate clusters
tutor only engaging with active students, ignoring disengaged ones on the edge
Social Learning Analytics about to appear in products…
42
http://www.desire2learn.com/products/analytics (this is from a beta demo)
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011), ACM: New York. pp.22-33 http://oro.open.ac.uk/25829
Discourse analytics: what intellectual contribution does this learner make?
Rebecca is playing the role of broker, connecting peers’ contributions in meaningful ways
Semantic Social Network Analytics: shows if users agree or disagree
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011), ACM: New York. pp.22-33 http://oro.open.ac.uk/25829
Discourse analytics on webinar textchat
Ferguson, R. and Buckingham Shum, S., Learning analytics to identify exploratory dialogue within synchronous text chat. In: 1st International Conference on Learning Analytics and Knowledge (Banff, Canada, 2011). ACM
Can we spot the quality learning conversations in a 2.5 hr webinar?
-60
-40
-20
0
20
40
60
80
9:2
8 9
:32
9:3
6
9:4
0
9:4
1
9:4
6
9:5
0
9:5
3
9:5
6
10:
00
10:
05
10:
07
10:
07
10:
09
10:
13
10:
17
10:
23
10:
27
10:
31
10:
35
10:
40
10:
45
10:
52
10:
55
11:
04
11:
08
11:
11
11:
17
11:
20
11:
24
11:
26
11:
28
11:
31
11:
32
11:
35
11:
36
11:
38
11:
39
11:
41
11:
44
11:
46
11:
48
11:
52
11:
54
12:
00
12:
03
12:
04
12:
05
Average Exploratory
Discourse analytics on webinar textchat
Sheffield, UK not as sunny as yesterday - still warm Greetings from Hong Kong Morning from Wiltshire, sunny here!
See you! bye for now! bye, and thank you Bye all for now
Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar…
-60
-40
-20
0
20
40
60
80
9:2
8 9
:32
9:3
6
9:4
0
9:4
1
9:4
6
9:5
0
9:5
3
9:5
6
10:
00
10:
05
10:
07
10:
07
10:
09
10:
13
10:
17
10:
23
10:
27
10:
31
10:
35
10:
40
10:
45
10:
52
10:
55
11:
04
11:
08
11:
11
11:
17
11:
20
11:
24
11:
26
11:
28
11:
31
11:
32
11:
35
11:
36
11:
38
11:
39
11:
41
11:
44
11:
46
11:
48
11:
52
11:
54
12:
00
12:
03
12:
04
12:
05
Average Exploratory
Discourse analytics on webinar textchat
Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar but if we zoom in on a peak…
Discourse analytics on webinar textchat
-100
-50
0
50
100
9:2
8 9
:40
9
:50
1
0:00
1
0:07
1
0:17
1
0:31
1
0:45
1
1:04
1
1:17
1
1:26
1
1:32
1
1:38
1
1:44
1
1:52
1
2:03
Averag
Classified as “exploratory
talk”
(more substantive for learning)
“non-exploratory”
Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar but if we zoom in on a peak…
“Rhetorical parsing” to identify constructions signifying scholarly writing
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
OPEN QUESTION: “… little is known …” “… role … has been elusive” “Current data is insufficient …”
CONTRASTING IDEAS: “… unorthodox view resolves …” “In contrast with previous hypotheses ...” “... inconsistent with past findings ...”
SURPRISE: “We have recently observed ... surprisingly” “We have identified ... unusual” “The recent discovery ... suggests intriguing roles”
“What are the key contributions of this text?
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work, 21, 4-5, (2012), 417-448. http://oro.open.ac.uk/31052
Human analyst Computational analyst
learning objective – how are we doing?
walk out with
better questions than you could ask 30mins ago
51
How will my org. evolve from a digital exoskeleton to a nervous system?
52 Ed Dumbill: http://strata.oreilly.com/2012/08/digital-nervous-system-big-data.html
The Wal-Martification of education?
53 http://chronicle.com/blogs/techtherapy/2012/05/02/episode-95-learning-analytics-could-lead-to-wal-martification-of-college http://lak12.wikispaces.com/Recordings
“What counts as data, how do you get it, and what does it
actually mean?”
“The basic question is not what can we measure? The basic question is
what does a good education look like?
Big questions.
“data narrowness” “instrumental learning”
“students with no curiosity”
Analytics provide maps = systematic ways of distorting reality in order to reduce complexity
Buckingham Shum, S. and Deakin Crick, R. (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling and Learning Analytics. Proc. 2nd Int. Conf. Learning Analytics & Knowledge. (29 Apr-2 May, 2012, Vancouver, BC). ACM: New York. Eprint: http://oro.open.ac.uk/32823
“A marker of the health of the learning analytics field will be the quality of debate around what the technology renders visible and leaves invisible.”
55
Will your staff know how to read and write analytics?
This will become a key literacy.
56
What if you engaged your learners in the co-design of the analytics which will track
them?
Think about the conversations you’d need to have…
57
Are you ready for your performance indicators
to be computed from analytics?
58
Our analytics are our pedagogy
They promote assessment regimes
— which drive (and strangle) educational innovation
Join the community…
59
SoLAResearch.org / @SoLAResearch
LAKconference.org / @LAKconf
Learning Analytics Policy Brief Exec Summary for UNESCO IITE
60 http://bit.ly/LearningAnalytics
Learning Analytics: Unlocking student data for 21st century learning?
Simon Buckingham Shum Knowledge Media Institute The Open University UK simon.buckinghamshum.net
BETT 2013, London — LearnLive HigherEd
@sbskmi #LearningAnalytics