learning analytics applied to serious games, invited talk at ecgbl 2013 porto, portugal
TRANSCRIPT
Learning Analytics Applied to Serious GamesEuropean Conference in Game-Based Learning (ECGBL 2013), Porto, Portugal
Baltasar Fernandez-Manjon, [email protected] , @BaltaFMe-UCM research group, www.e-ucm.es
2
About me and context
‣ CS Professor at Complutense U.• Director of e-UCM
‣ e-UCM research group about Learning technologies
• 15 researchers• Serious games
- Application to the medical domain• European projects
- GALA- SEGAN- CHERMUG
• www.e-ucm.es
3
Learning analytics and SG?The NMC Horizon
Report: 2013 Higher Education Edition
‣ new and emerging technologies on teaching, learning, and research
Time-to-Adoption Horizon: Two to Three Years
‣ Games and Gamification
‣ Learning Analyticshttp://www.nmc.org/publications/2013-horizon-report-higher-ed
4
Serious Games use?
‣ Serious Games have probed to be educationally effective in several domains
• Medicine, military, business, corporate training‣ But still is a low adoption of Serious Games
• Cost? ROI?‣ Serious Games considered usually as a
complementary content• Mainly used for motivational purposes• No actual impact on the final mark
‣ Difficult to include Serious Games in the learning curriculum
• Assessment of acquired learning?
5
Black box model‣ Games as
independent pieces of content
‣ No information about what is happening during the in-game play
‣ Or very simple
• Completed or not completed
• Time used
del Blanco et al (2013). Using e-Learning standards in educational video games. Computer Standards & Interfaces 36 (1) pp. 178–187
6
Learning Analytics
‣ Improving education based on data analysis
• Data driven
‣ Evidence-based education
‣ Related with …• Educational data mining
• Business intelligence
• Visual analytics
www.ed.gov/edblogs/technology/files/2012/03/edm-la-brief.pdf
7
Different uses
‣ Learning analytics• Data analysis that helps students improve
learning outcomes.‣ Academic/program analytics
• Data analysis that provides information of what is happening in a specific program and how to plug holes or otherwise adjust.
‣ Institutional analytics• Data analysis that helps make decisions about
how to improve at the institutional level.
Learning Impact Blog, Big data: Cool; Small data: Cooler http://www.imsglobal.org/blog/?p=258
8
When data is analyzed?
‣ Off-line• Analyzing data after use• Discovering patterns of use• Allows to improve the experience for future
‣ Real time• Analyzing data while the system is in use to
improve/adapt the current learning experience• Allows to also use it in actual presential classes
‣ Mixed approach
9
Extensive Data
‣ Large number of participants‣ Relatively limited number of variables‣ Usually very little demographic information‣ Relatively few observations for each user‣ Wide but shallow data set
Adapted from: Learning Analytics and Educational Data Mining WorkshopNew York University – CREATE Lab April 4–5, 2013
10
Intensive Data
‣ Relatively low number of participants‣ Large number of observations for each
variable‣ Large number of variables for each
participant, such as• User actions, In-Game Events• Survey responses; Extensive demographic
information• Video Observations• Biometric Data (HR, RESP, GSR, EEG, EKG)• Eye-tracking
‣ Narrow but deep data set
‣ Correlations among different data?Adapted from : Learning Analytics and Educational Data Mining WorkshopNew York University – CREATE Lab April 4–5, 2013
11
But there is a parallel world …
http://us.battle.net/wow/en/media/screenshots/races?keywords=&view#/goblins04
12
Game Analytics‣ Application of
analytics to game development and research
‣ Telemetry
• Data obtained over distance
• Mobile games, MMOG
‣ Game metrics
• Interpretable measures of data related to games
• Player behaviour
13
Game metrics
‣ User metrics
• Customer
• Player
‣ Performance metrics
• Technical infrastructure
‣ Process metrics
• Development of the game
‣ User metrics
• Generics metrics
• Genre specific metrics
• Game specific metrics
14
Game metrics
Springer, Game Analytics Maximizing the Value of Player Data, pag 22
15
Game requirements for LA‣ Most of games are black boxes.
• No access to what is going on during game play.
‣ We need access to game “guts”• Game state, game variables
‣ Or the game must communicate with the outside world
• Using some logging framework• Not applicable to COTS games (yet)
‣ Mozila Open Badges? http://openbadges.org/
SESSION VARIABLES
PLAYER GAME
VARIABLES
GAME VARIABLES
TRACES
GOALS
From game data to educational information
PLAYER VARIABLES
User Information
Educational System Information
SERIOUS GAME
Sessions
GAME PLAYER SESSIONS+ =
A player plays a game. Produces a game session.
One game session is a set of traces of ONE specific player in ONE specific game.
SESSIONS =
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
player starte
d phase
1
player scored
200in
phase 1
player clicked
in Help
button
VARIABLESGENERATO
R
SESSION
Analyzing game sessions
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
TRACE
VAR
VAR
VAR
VAR
VAR
VAR
VAR
VAR
VAR
VAR
GOAL RESULT
GOAL RESULT
GOAL RESULT
GOAL RESULT
GOAL RESULT
GOALSGENERAT
OR
VAR DEFINITI
ONVAR
DEFINITION
VAR DEFINITI
ONVAR
DEFINITION
VAR DEFINITI
ON
VAR DEFINITI
ON
GOALDEFINITI
ON
GOALDEFINITI
ONGOAL
DEFINITION
GOALDEFINITI
ON
Variable generator: User sessions
VAR DEFINITION VAR+ =
TRACE
TRACE
varName:‘beatgame‘, operation: { type:'trace_present‘,traceType: ‘logic', trace: { event:‘end_game’ }}
event: ‘end_game',timeStamp: 1380181610150
usersession.beatgame = true;+ =
Goal Generators: User sessions
VAR
VAR
GOALDEFINITION
GOAL RESULT=
usersession: { beatgame: true, score: 4747}
id: ‘finalResult’result: ‘beatgame && score > 3000’}
usersession: { beatgame: true, score: 4747, goals: { finalResult: true }}
=
+
+
PLAYER * GAME
Analyzing game results
SESSION
SESSION
VAR VAR
VAR VAR
GOAL RESULT
GOAL RESULT
SESSION
SESSION
VAR VAR
VAR VAR
GOAL RESULT
GOAL RESULT
VARIABLESGENERATO
R
GOALSGENERATO
R
VAR DEFINITI
ONVAR
DEFINITION
VAR DEFINITI
ONVAR
DEFINITION
VAR DEFINITI
ON
VAR DEFINITI
ON
GOALDEFINITI
ON
GOALDEFINITI
ONGOAL
DEFINITION
GOALDEFINITI
ON
eAdventure game platform
Open code authoring environment for the production of point-and-click adventure games & immersive learning simulations
Easy to include Learning Analytics in eAdventure games
http://www.chermug.eu
With MGH-Harvard Universityhttp://first-aid-game.e-ucm.es
With ONT, educ@ONT
eAdventure + Learning Analytics
Game Engine
CommunicationAPI
Logic
Input
Input
Logic
Input
{ type: 'input', timeStamp: some_timestamp, device: 'some_device', action: 'some_action', target: 'target_id', data: { key1: value, ...}}
{ type: 'logic', timeStamp: some_timestamp, event: 'some_event', target: 'some_id', data: { key1: value, ...}}
Learning Analytics Database
25
GLEANER
‣ GLEANER: Game Learning Analytics for education research
• Open code framework to capture game traces
Reference model in the EU NoE GALA
GLEANER Collector
‣ Implemented: Nodejs + Mongodb‣ A web server ready to receive traces‣ Cleans and sorts the data
GLEANER Analysis
‣ Reporter has access to the database, and presents its data through reports
• graphics, heat-maps, relational tables…
‣ Evaluator has access to the database and checks the educational defined goals in the assessment model
Example: A game to learn XML
http://gleaner.e-ucm.es/lostinspace/play/index.html
About the game...
‣ A basic platform game to acquire familiarity with XML documents
• Learn the syntax• Understand nesting and attributes• Gain agility writing and reading XML documents
‣ Designed as a complementary activity in a Web Programming Course
• For undergraduate computer science students
‣ Not developed with eAdventure
Understanding the game…
Goal
Main characte
r
Power-ups (new syntax elements)
Write XML snippets here to move the
main character to the goal
Educational goals
‣ Basic understanding of XML Syntax• Markup syntax• Basic documents• Attributes• Complex documents
‣ Basic programing skills• Sequencing• Loops
LA perspective
‣ What does GLEANER trace?• Higher level events• Generic traces & game specific traces• Only those relevant for our learning objectives
‣ The aggregator will filter and transmit:• Level completion events• Each XML fragment submitted by the player
Main dashboard (GLEANER Report)
33
Progress report
34
User Session
Phases complete
d
Current score
Learning achievements
Achievements are updated and
highlighted in real time
Detailed traces
35
Filters XML snipets
The fragments are updated in real time as submitted by the student. The instructor can filter to see only
mistakes, or all the fragments submitted by a specific students.
La Dama Boba : The Game
Based on The Foolish Lady by Lope de VegaThe game is available at http://damaboba.e-ucm.es/ (in Spanish)
Experiment game vs class‣ 757 students
‣ From 8 middle and high schools in Madrid region
‣ Control group and experimental group
39
Formal evaluation pre-post‣ Similar results with
Learning Analytics than with pre-post test?
learning analytics outcomes
41
42
Other issues in LA
‣ Ethical and legal aspects
‣ Security model
‣ Ownership of information• Informing the user
‣ Anonymization of information
‣ Aspects specially relevant if you are working with kids!
43
But there are new especifications and
develpments that could sistematize the work
44
ADL eXperience API (xAPI)‣ Result of Project Tin Can
‣ Tracks experiences, informal learning, real-world experiences (not just completions)
‣ Allows data storage AND retrieval (ex. 3rd party reporting and analytics tools)
‣ Enables tracking mobile, games, ITS, and virtual worlds experiences
‣ Developed by open source communityFrom Damon Regan (ADL) presentation at SINTICE2013
45
Activity Streams
‣ http://activitystrea.ms
‣ Collaboration between Google, Facebook, Microsoft and others
‣ Allows reporting of experiences, not just completions‣ Format: <Actor> <Verb> <Object> (I did this):
Simple Statement: I (actor) watched (verb) a video on protecting employee data (object)
Complex: in the context of [information assurance certification training] with result [timestamp:2013-0618T18:30:32.360Z ].
From Damon Regan (ADL) presentation at SINTICE2013
Reporting
Systems
Assessment
Services
Semantic Analysis
Statistical Services
xAPI Learning Record Store (LRS)
46
From Damon Regan (ADL) presentation at SINTICE2013
‣ Raw data can feed several systems• An LRS• A Learning Analytics System
eAdventure + LA with xAPI
Logic
Input
Logic
Input
Input
Input
Raw data
LRS
Learning Analytics System
StatementsAnalyzer
StatementsAnalyzer
EXPERIENCEAPI
EX
PE
RIE
NC
EA
PI
IMS Global Learning Analytics Interoperability Framework
http://www.imsglobal.org/IMSLearningAnalyticsWP.pdf
50
Conclusions
‣ LA in Serious Games has a great potential from the application and research perspective
• Simplify more complex and complete experiments
‣ LA in Serious Games should benefits from Games Analytics experience and work
‣ Still complex to implement LA in SG• Increases the (already high) cost of the games
‣ Frameworks and new standards specifications could greatly simplify LA implementation and adoption
Our current projects
Increasing patient safety using games
Modelling/teaching medical procedures atNational Transplant Organization