intelligent systems and learning centred design
DESCRIPTION
What are intelligent systems for enhancing learning? How do we design them so as to enhance a certain learner's experience? This talk starts with pedagogy theories focusing on designing learning experiences according to learner characteristics. Then it boldly connects them to specific design methodologies, which aims at producing usable and pedagogically effective intelligent systems for delivering enhanced learning material and experiences. Then the real pulp starts: how we designed the intelligent system of the TERENCE FP7 EU project for designing adequate learning material, specifically, smart games for reasoning about stories and adequate to the needs of poor comprehenders. The talk concludes reflecting upon the speaker's change in research areas possibly due to several factors, such as personal reasons, interests and project-dependent requirements. The on-going work is also briefly illustrated, that is, how to design smart games not only for learners but also with learners, engaging and including all.TRANSCRIPT
LEARNER CENTRED DESIGN
INTELLIGENT SYSTEMSand
Rosella Gennarihttp://www.inf.unibz.it/~gennari
Distinguished Speakers Oxford Women in CS
Settingtechnology enhanced learning
ClimaxTERENCE case study
Resolutionreflections
Story outline
Settingtechnology enhanced learning
ClimaxTERENCE case study
Resolutionreflections
Story outline
TECHNOLOGY ENHANCED LEARNING
Technology Enhanced Learning (TEL) is the usage of technology for supporting a learning experience
Herby we take a narrow view: intelligent TEL =
Artificial Intelligence (AI) technology based products for supporting a learning experience
TEL 4 LEARNING EXPERIENCE
How can we design technological products that supports their users’
learning experience?
LET'S SEE EDUCATORS' VIEWPOINT...
Maria Montessori (1870-1952)
Paraphrasing her words, adequate tasks that come in a prepared environment, designed on top of the learner characteristics can effectively support the learner’s learning
was the first Italian woman physician and educator, best known for Montessori pedagogy
ousability of technology learning products
opedagogical effectiveness of technology learning products
TEL 4 LEARNING EXPERIENCE
Adequate tasks that come in a prepared environment designed on top of the learner characteristics can effectively support learning
HOW TO DESIGN USABLE AND PEDAGOGICALLY EFFECTIVE TEL
Based on UCD process diagram (© Tom Wellings)
requirement specification
designevaluation
plan
models + prototypes
intermediate product
final product
HOW TO DESIGN USABLE AND PEDAGOGICALLY EFFECTIVE TEL
USABILITY + P. EFFECTIVENESS
Settingtechnology enhanced learning
ClimaxTERENCE case study
Resolutionreflections
Story outline
TERENCE DESIGN
Based on UCD process diagram (© Tom Wellings)
requirement specification
designevaluation
plan
models + prototypes
intermediate products
final products
TERENCE was an FP7 TEL project blending user centred and evidence based design
USABILITY + P. EFFECTIVENESS
THE PROBLEM
‣ TERENCE developed an adaptive learning system (ALS) that, via a learner GUI, recommends poor comprehenders
- its learning material, i.e., books of stories and games
- its learning tasks, i.e., reading and playing
‣ so as to stimulate their reading comprehension
‣More than 10% of primary school children, older than 8, are diagnosed with deep text comprehension problems
‣ They are referred to as poor comprehenders
THE TERENCE WORLD
a d e q u a t e b o o k o f s t o r i e s
s i g n i n
a d e q u a t e s m a r t g a m e s
r e w a r d
TERENCE INTELLIGENT TEL PRODUCTS
ALS LayerGUI Layer
Learner
EducatorExpert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
UserManager
OpenRDF
StoryManager
OpenRDF
GameManager
OpenRDF
VisualisationManagerillustrations
NPL
Reasoner
AdaptiveEngine
Visualisation
������������
Reasoning Module
AnnotationModule
VisualisationModule
game generation
adaptation to learners
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
TERENCE DESIGN
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
DATA GATHERING METHODS
‣Data for designing the learning material and tasks were from
- contextual inquiries with
‣ IT & UK diagnosis
‣ as well as IT & USA evidence-based medicine therapy experts
- field studies with educators and primary school children
4510
~ 500
CHARACTERI
STICS
Persona Name: Carol
Age: 8
Classroom: year 4
RC levels: low reading levels
Rural/Urban: urban
Deaf/hearing: deaf
First Language: Italian Sign Language
Cochlear Implantation (if deaf): yes
Degree of hearing loss (if deaf): profound
Motor skills (if deaf): average
Summary of the class represented by this persona
A younger deaf girl who is very enthusiastic about using new technology (such as iPhone and IPad) and who adores her Nintendo DS. Her reading RC levels are very low, but she reads together with her parents to learn new words and spelling. She also likes to do many other things such as drawing, taking care of her pets and going to the park.
Quote “I really love Mario and Luigi. And I would love to have an iPhone and an IPad, like my dad.”
Personality Open
Role in classroom Active
Role out of the classroom
Active
Console/Technology Carol and her sister watch TV after school. They like Tom & Jerry, Ben 10, Hello Kitty and Mickey Mouse Clubhouse.
Carol sometimes uses the computer, but only to play minilab games. Her computer is in her bedroom, but her parents don’t allow her to use it all of the time. She can use it only one hour per day. Carol’s dad has an iPhone and an IPad, and Carol would really like to use those as well, but her dad tells her she is a bit too young. Carol likes watching him with his IPad and iPhone though.
Carol doesn’t use a mobile phone.
Carol plays games on the computer and on her Nintendo DS. She plays by herself. She likes the mini-clip games on the computer, and Mario Kart and brain training games on her DS.
She likes games with non-photorealistic human avatars, and prefers fantasy avatars to animal avatars.
Socio-Cultural Level of his/her own family
Medium
School performance Carol has sever reading problems. In her class she is below average in all activities but drawing, where she feels she can truly express her intimate feelings.
Homework After school, Carol does her homework together with her mum.
LIFE
STYLE
Outdoors Activities Carol often goes to the park with her mum.
Indoors Activities Games on the DS Carol reads sometimes. Her mum and dad help her reading in the evening. She likes some of the stories they read together, but mostly, she wants to read because she has to learn new words and spelling. Carol likes drawing and taking care of her pets. Her mum often plays with her.
Home activities Carol also likes to help her mum in the kitchen or in the garden.
Sport activities Carol practices no specific sport.
.
SMART GAME REQUIREMENTSWhat for Description
Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games.
Scheduling of reading and playing
1st silent reading; 2nd playing smart games; 3rd playing relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via visual representation and reasoning
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
SMART GAME REQUIREMENTSWhat for Description
Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games.
Scheduling of reading and playing
1st silent reading; 2nd playing smart games; 3rd playing relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via visual representation and reasoning
who is the actor of … ? what does (a main) character do?
when does … happen in relation to a central event?
why does the central event happen?
SMART GAME REQUIREMENTSWhat for Description
Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games.
Scheduling of reading and playing
1st silent reading; 2nd playing smart games; 3rd playing relaxing games
Constraints on actions Learners should get faster, hence a game has a maximal resolution time
Progress and feedback Monitor and give learners (1) visible idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via visual representation and reasoning
points for each smart
coins for all smartunlocked if read+play
visual feedback
Instructions Questions Motivational Interaction
Choices Choices for learner Fixed event
Solutions Choices that are either correct (c) or wrong (w)
Feedback Interaction Consistency Explanatory Solution
Smart points Proportional to the learner’s ability in the game level
Relaxing points
Constant
Avatar Happy/sad states
Time solution constant interaction constant
Rules States of the system, actions of the learner, constraints
What for Description
Difficulty levels Macro levels for learners: - entry: character games; - intermediate: time games; - top: causality games.
Scheduling of reading and playing
1st silent reading; 2nd playing smart games; 3rd playing relaxing games
Constraints on actions
Learners should get faster, hence a game has a maximal resolution time
Progress and feedback
Monitor and give learners (1) idea of progress, (2) explanatory feedback, (3) recall their attention and solicit them to give a resolution (in time)
Representation Production can be impaired hence promote resolution via visual representation and reasoning
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analyticalHMI experts or domain experts
heuristic evaluationformative, summativeexpert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS HOW
analyticalHMI experts or domain experts
heuristic evaluationformative, summativeexpert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
G1. interfaces follow general design guidelines
G2. interfaces support the user’s next step to achieve a task
G3. interfaces provide users with timely feedback
Instructions are not under focus and cannot be easily read
Game question and possible resolutions should be proximally close
Game question and possible resolutions should be proximally close
Evaluation of interfaces
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
problem: 256 stories,
each with ~12 games
Smart game design
how can we automatise the
development of smart games via AI (and, hopefully, be efficient)?
Smart game design
enriched annotations
story
annotations
ALS LayerGUI Layer
Learner
EducatorExpert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
UserManager
OpenRDF
StoryManager
OpenRDF
GameManager
OpenRDF
VisualisationManagerillustrations
NPL
Reasoner
AdaptiveEngine
Visualisation
������������
Reasoning Module
AnnotationModule
VisualisationModule
Semi-automated generation
enriched annotations
story
text
text
text
text
annotations
Semi-automated generationALS LayerGUI Layer
Learner
EducatorExpert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
UserManager
OpenRDF
StoryManager
OpenRDF
GameManager
OpenRDF
VisualisationManagerillustrations
NPL
Reasoner
AdaptiveEngine
Visualisation
������������
Reasoning Module
AnnotationModule
VisualisationModule
text
text
text
text
image
image image image
enriched annotations
story
annotations
Semi-automated generation
games
template visual
text
text
text
text
image
image image image
enriched annotations
story
annotations
Semi-automated generationALS LayerGUI Layer
Learner
EducatorExpert
Learner GUI
Expert GUI
Persistence Layer
OpenRDF
UserManager
OpenRDF
StoryManager
OpenRDF
GameManager
OpenRDF
VisualisationManagerillustrations
NPL
Reasoner
AdaptiveEngine
Visualisation
������������
Reasoning Module
AnnotationModule
VisualisationModule
text
story
text + visual
games
AUTOM. MANUAL AUTOM.
Semi-automated generation
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analyticalHMI experts or domain experts
heuristic evaluationformative, summativeexpert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS WHEN
analyticalHMI experts or domain experts
heuristic evaluationformative, summativeexpert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
EXAMPLE METHODS IN TERENCE
LARGE-SCALE STUDY DESIGN
Common design of the intervention with TERENCE:
‣ how: pretest/posttest design, with experimental and control groups
‣ hypothesis: TERENCE improves reading comprehension measured with standardized text comprehension tests
ControlExperimental
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs
-and requires (1) reading
A 3-PHASE INTERVENTION
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs
-and requires (1) reading (2) playing smart
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs
-and requires (1) reading (3) playing relaxing(2) playing smart
A 3-PHASE INTERVENTION
‣ Post-test (pedagogical only) for re-assessing text comprehension
‣ Pre-test for (1) assessing txt comprehension, (2) initialising TERENCE
‣ Stimulation phase for experimental group with usage sessions so that each -lasts < 45 minutes for attention needs
-and requires (1) reading (3) playing relaxing(2) playing smart
A 3-PHASE INTERVENTION
The experimental group is of 344 learners:
‣Avezzano: 270 learners:
- 7-9 years old: 118
- 9-11 years old: 152
‣ Pescina: 74 learners:
- 7-9 years old: 37
- 9-11 years old: 37
‣ They were tested (January-February), stimulated (March-May), and re-tested (may-June)
EXPERIMENTAL GROUP IN IT
Pre-post performances for text comprehension (dependent variable) were as follows:
‣ Pescina:
- pre: 14 poor comprehenders (20.59%)
- post: 6 poor comprehenders (8.82%)
‣ Avezzano:
- pre: 15 poor comprehenders (5.95%)
- post: 2 poor comprehenders (0.79%)
MAIN RESULTS IN IT
PrePescina Avezzano
5,95%
20,59%
‣ Wilcoxon signed-rank test supports that differences are statistically significant
- Pescina: z=-4.904, p<0.0001
- Avezzano: z=-2.266, p=0.0234
EXAMPLE METHODS IN TERENCE
APPROACH WITH WHOM EXAMPLE METHODS HOW
analyticalHMI experts or domain experts
heuristic evaluationformative, summativeexpert evaluation
cognitive walk-through
small-scale learnersobservations
formativethink aloud
large-scale learners field studies summative
EXPERT EVALUATION
Experts of pedagogy: 1 coordinator; 9 evaluators
Sophie'comes'down'the'steps
He had never been beaten before, since he only ever raced with kids who were smaller and slower than him. He wanted a rematch, so the two boys set off again. Ben was paddling as fast as he could, still he didn’t make it to the wall before Luke. It was completely unfair, he thought. Luke was so much faster. No sooner had they climbed out of the water, than he saw his sister coming down the steps. She was smiling at Ben and gave him a playful pat on the shoulder. She also gave Ben a friendly speech about winners and losers.
revise selection of solutions
revise selection of central event
How-to:
1. each pair of evaluators read a story, and edited its games
2. the coordinator revised their work
3. a pair of evaluators was blindly assigned revised games, and another the manually created games
Main edit tasks:
(1) creation of missing games (~recall)
(2) revision of games (~precision)
From D4.2 and D4.3 technical annex
overall assessment of generation
text
story
text + visual
games
revision of Automated Reasoning (AR) selection of central events and solutions
revision of Natural Language Processing (NLP) of text
text
text text text
Edit tasks in details
Requirements Prototypes Analytic + small ev. Int. prod. Analytic+large Fin. prod.
TERENCE DESIGN
From D4.2 and D4.3 technical annex
overall generation
text
story
text + visual
games
AR selection of central events and solutions
revision of NLP text
text
text text text
Analyses of evaluation results
AR selection of central events for games:
>Results: only in 15 out of 250 cases (6%), it was necessary to select a different central event than the automatically generated one
From D4.2 and D4.3 technical annex
>Implications for AR: none picked up
Automated part evaluation-based re-design
AR selection of plausible solutions:
>Results: out of 140 changes of selection of solutions, the majority was for wrong solutions
- generate a wrong solution from correct one by changing participants, e.g.,
<correct_sentence id="2">The man ran and fell on the ground.
</correct_sentence>
<wrong_sentence id="2wh1">Peter ran and fell on the ground.
</wrong_sentence>
>Implications for WP4: new heuristics for wrong plausible solutions in the last part of Y3,
From D4.2 and D4.3 technical annex
Automated part evaluation-based re-design
Overall generation: development times:
>Results for revision time:
- 12’6” per game instance:
↑ 12’8” for time games
↓ 10’6” for who games
>Results for creation time:
- avg. 23” per game instance
text
storytext + visual
games
From D4.2 and D4.3 technical annex
>Implications for AR: the semi-automated development process seems to be promising for optimising development times
Automated part evaluation-based re-design
Game over1st 2nd 3
Sep. 2011 December 2012 September 2013
Sophie'comes'down'the'steps
He had never been beaten before, since he only ever raced with kids who were smaller and slower than him. He wanted a rematch, so the two boys set off again. Ben was paddling as fast as he could, still he didn’t make it to the wall before Luke. It was completely unfair, he thought. Luke was so much faster. No sooner had they climbed out of the water, than he saw his sister coming down the steps. She was smiling at Ben and gave him a playful pat on the shoulder. She also gave Ben a friendly speech about winners and losers.
revise selection of solutions
revise selection of central event
Requirements+for Description
Dif$iculty*levels Macro*levels*for*learners:4*entry:*character*games;4*intermediate:*time*games;4*top:*causality*games.*
Scheduling*of*reading*and*playing
1st*silent*reading;* 2nd* playing* smart* games;*3rd*playing*relaxing*games
Constraints*on*actions
Learners* should* get* faster,* hence* a* game* has* a* maximal*resolution+time
Progress*and*feedback
Monitor* and* give* learners* (1)* idea* of* progress,* (2)*explanatory*feedback,*(3)*recall*their*attention*and*solicit+them*to*give*a*resolution*(in*time)
Representation Production*can*be* impaired* hence*promote* resolution*via*visual*representation+and+reasoning
Instruc(ons Ques%onsQues%ons Mo%va%onalMo%va%onalMo%va%onalMo%va%onal Interac%onInterac%on
Choices Choices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learnerChoices3for3learner 3Fixed3event3Fixed3event3Fixed3event
Solu(ons Choices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%onsChoices3or3their3combina%ons3that3are3correct/wrong3(c/w)3solu%ons
Feedback Interac%on Consistency3(c/w)Consistency3(c/w)Consistency3(c/w) ExplanatoryExplanatoryExplanatory Solu%on
Smart6points Propor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3levelPropor%onal3to3the3learner’s3ability3in3the3game3level
Relaxing6points ConstantConstantConstantConstantConstantConstantConstantConstant
Avatar Happy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3statesHappy/sad3states
Time solu%on3constantsolu%on3constantsolu%on3constant interac%on3constantinterac%on3constantinterac%on3constantinterac%on3constantinterac%on3constant
Rules States3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraintsStates3of3the3system,3ac'ons3of3the3learner,3constraints
data structures
NLP+ AR1 for stories
AR1 for txt games
frameworkAR2 + NLP1 for txt games
AR2 for stories AR3 + NLP2 for txt games
requirements
Settingtechnology enhanced learning
ClimaxTERENCE case study
Resolutionreflections
Story outline
Till 2007
Cate
gory
Axis
AR
HMI
TEL
Game
0 4 8 12 16
Response: work areas Amsterdam U. and CWI
FBK-irstFree U. of Bolzano
From 2007
Cate
gory
Axis
AR
HMI
TEL
Game
0 4 8 12 16
Response: work areas
Free U. of Bolzano
Possible explanation?
co-designgamification cooperative learningHOW
WHY engagement design together inclusion
childrendesigners teachersWHO
WHAT
GACOCO
treestree puzzle
Gamification of protocol (tasks, subtasks and types of feedback)
Gamification (competition for cooperation)MISSIONS
CHALLENGES
REWARDSwell
done!
Acknowledgments to
TERENCE colleagues and schools
Current colleagues and schools
DIARY FOR PRESENTTHE TERENCE BOOK
Settingtechnology enhanced learning
ClimaxTERENCE case study
Resolutionreflections
Story outline
?