Interactive Learning Technologies
Dr. Jan Schneider
Page 2
Page
Teaching Machines
3
1924, Sidney L. Pressey
Page
Teaching Machines 1950s
4
https://www.youtube.com/watch?v=jTH3ob1IRFo
Page
1981?
5
Page
Intelligent Tutoring System
6
ANDERSON, I. R., & REISER, B. I. (1985). THE LISP TUTOR.
Page
How do we learn?
7
Page
How do we learn?
8
Page
How do we learn?
9
Page
Sensors in Learning
• What are sensors?
• A device which detects or measures a physical property and records, indicates,
or otherwise responds to it. (Oxford Dictionaries, 2018)
• Examples of sensors:
10
Page
Sensors in Learning
• What type of sensors?
11
Page
Sensors in Learning
• Do specific sensors matter?
12
Page
Sensors in Learning
13Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015). Augmenting the senses: a review on sensor-based learning support. Sensors, 15(2), 4097-4133.
Page
Cognitive Domain
• Contextual information
14Ternier, S., Klemke, R., Kalz, M., Van Ulzen, P., & Specht, M. (2012). ARLearn: Augmented Reality Meets Augmented Virtuality. J.
UCS, 18(15), 2143-2164.
Page
Cognitive Domain
15Kanjo, E. (2010). Noisespy: A real-time mobile phone platform for urban noise monitoring and mapping. Mobile Networks and
Applications, 15(4), 562-574.
Sensor Data for analysis
Page
Affective Domain
16Carroll, E. A., Czerwinski, M., Roseway, A., Kapoor, A., Johns, P., Rowan, K., & Schraefel, M. C. (2013, September). Food and mood: Just-in-time support for emotional eating. In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine
Association Conference on (pp. 252-257). IEEE.
Page
Psychomotor Domain
17Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015). Augmenting the senses: a review on sensor-based learning support. Sensors, 15(2), 4097-4133.
Page
Limitations of Sensors
18
Login
Sensor data is difficult to interpret
1 source might not be enough to interpret some task
Page
Limitations of Sensors
Multimodality
• What instrument is being played in the picture?
19
Page
• What is the instrument being played in the sound clip?
20
Limitations of Sensors
Multimodality
Page
• What are the instruments being played in this sound clip?
21
Limitations of Sensors
Multimodality
Page
• Why do we need multimodality? (Example 1)
• https://www.youtube.com/watch?v=Kmft674XPC0
22
Limitations of Sensors
Multimodality
Page
Multimodality
23
Page
Multimodality
• Why do we need multimodality? (Example 3)
• Change of interpretation
• https://www.youtube.com/watch?v=rn9V0cN4NWs24
Page
Multimodality
• Why do we need multimodality? (Example 4)
25
Page
Multimodality for human computer
interaction (1979)
Bolt, R. A. (1980). “Put-that-there”: Voice and gesture at the graphics interface (Vol. 14, No. 3, pp. 262-270). ACM.ISO 690
https://www.youtube.com/watch?v=RyBEUyEtxQo
Page
They encode
messages
Todays multimodality for
computers
e.g. 3D visualisations in AR
They decode
sensor inputs
e.g modern AR
headsets like
Hololenes
Page
Examples of Interactive Multimodal Learning
technologies
28
Page 29
„During the course lessons, arousal was positively and highly correlated (r = .66)
with achievement as measured by the students' grades.“
Pijeira-díaz, H. J., Drachsler, H., Kirschner, P. A. & Järvelä, S. (2018). Profiling sympathetic arousal in a physics course: how active are students? Journal of Computer Assisted Learning, SI on Multimodal
Learning Analytics. pre-print:
Making Predictions
Page
Understanding Learning Process
30
The distance between learners' hands and faces is a strong predictor of students' artefact quality,
which can indicate the value of student collaboration.
Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of
Computer Assisted Learning, 34(4), 366-377.
Page 31
Mimicking Expert Performance
http://wekit.eu/
https://www.youtube.com/watch?v=RkGXdpSB_AM
Page
Presentation trainer
Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2016). Can you help me with my
pitch? Studying a tool for real-time automated feedback. IEEE Transactions on Learning
Technologies, 9(4), 318-327.
Page 33
Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015). Stand tall and raise your voice!
a study on the presentation trainer. In Design for teaching and learning in a networked world (pp.
311-324). Springer, Cham.
Presentation trainer
(Real-time Feedback)
Page 34
Dashboard interface
Is not useful!
Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015). Stand tall and raise your voice!
a study on the presentation trainer. In Design for teaching and learning in a networked world (pp.
311-324). Springer, Cham.
Presentation trainer
(Real-time Feedback)
PageSchneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2016). Can you help me with my
pitch? Studying a tool for real-time automated feedback. IEEE Transactions on Learning
Technologies, 9(4), 318-327.
Presentation trainer
(Real-time Feedback)
Page 36
0
0,2
0,4
0,6
0,8
1
1,2
1 2 3 4 5 6
Treatment Group
Control Group
Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2016). Can you help me with my
pitch? Studying a tool for real-time automated feedback. IEEE Transactions on Learning
Technologies, 9(4), 318-327.
Presentation trainer
(Real-time Feedback)
Page 37Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2016, September). Enhancing public
speaking skills-an evaluation of the Presentation Trainer in the wild. In European Conference on
Technology Enhanced Learning (pp. 263-276). Springer, Cham.
0,00
0,50
1,00
1,50
2,00
2,50
3,00
3,50
4,00
4,50
5,00
Qua
lity
Del
iver
y
Kno
wled
ge
Confid
ence
Clarit
y
Enter
tain
ing
Enthu
siasm
Post
ure
Voi
ce
Ges
ture
s
Paus
es
Pi tch 1
Pitch 2
Presentation trainer
(Real-time Feedback)
Page 38
Presentation trainer
(Real-time Feedback)
• 1 feedback instruction that
does not go away until the
learner corrects the mistake
works.
• For good and for bad, there
is no emotion between PT’s
feedback and the learner.
PageSchneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2017, June). Do You Know What Your
Nonverbal Behavior Communicates?–Studying a Self-reflection Module for the Presentation
Trainer. In International Conference on Immersive Learning (pp. 93-106). Springer, Cham.
Presentation trainer
(Self-reflection)
PageSchneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2017, June). Do You Know What Your
Nonverbal Behavior Communicates?–Studying a Self-reflection Module for the Presentation
Trainer. In International Conference on Immersive Learning (pp. 93-106). Springer, Cham.
Presentation trainer
(Self-reflection)
Does it work?
• Inconclusive evidence
• Learners identify problems but do not know what to do about them.
• It seems that learners change behavior for a few seconds and then go back to old habits.
Page
Schneider, J., Börner, D., van Rosmalen, P., & Specht, M. (2018). Do you Want to be a
Superhero? Boosting Emotional States with the Booth. Journal of Universal Computer
Science, 24(2), 85-107.
The Booth
Page
Schneider, J., Börner, D., van Rosmalen, P., & Specht, M. (2018). Do you Want to be a
Superhero? Boosting Emotional States with the Booth. Journal of Universal Computer
Science, 24(2), 85-107.
The Booth
Page
Schneider, J., Börner, D., van Rosmalen, P., & Specht, M. (2018). Do you Want to be a
Superhero? Boosting Emotional States with the Booth. Journal of Universal Computer
Science, 24(2), 85-107.
The Booth
Does it work?
Page
Schneider, J., Börner, D., van Rosmalen, P., & Specht, M. (2018). Do you Want to be a
Superhero? Boosting Emotional States with the Booth. Journal of Universal Computer
Science, 24(2), 85-107.
The Booth
Does it work?
Page
Limitations
Customizable MultimodalLearning Solution
Hardware costs
Software costs
Page
MMLA Examples
Schneider, J., Di Mitri, D., Limbu, B., & Drachsler, H. (2018, September). Multimodal learning
hub: A tool for capturing customizable multimodal learning experiences. In European Conference
on Technology Enhanced Learning (pp. 45-58). Springer, Cham.
Di Mitri, D., Schneider, J., Klemke, R., Specht, M., & Drachsler, H. (2019, March). Read Between
the Lines: An Annotation Tool for Multimodal Data for Learning. In Proceedings of the 9th
International Conference on Learning Analytics & Knowledge(pp. 51-60). ACM.
Page
MMLA Examples
Multimodal Tutor Builder Kit. In: Proceedings of International Conference on Learning Analytics
and Knowledge, Arizona, US, March 7–9, 2019 (LAK '19) .
Calligraphy tutor Virtual Reality Presentation Trainer
Page
Conclusions
48