E19.2250 E19.2250 Research on Simulations and Research on Simulations and

Games Games for Educationfor Education

E19.2250 E19.2250 Research on Simulations and Research on Simulations and

Games Games for Educationfor Education

Jan L. Plass, DMDL

OverviewOverviewOverviewOverview

PlaytestingGoalsDesignMethodsSetup

Game GenresGame GenresGame GenresGame Genres

Goals for Games for LearningLiteracyFocus on social learning, identityExploration and knowledge creationLegitimate Participation in community of practiceApprenticeship (reciprocal)Learning through design & creating new artifacts

Game ResearchGame ResearchGame ResearchGame Research

Playtesting–Research GoalsGuide Design of New GamesPlaytest, Usability of Game PrototypesEfficacy, Evaluation of Completed GamesAdvancement of Theory

Games ResearchGames ResearchGames ResearchGames Research

PlaytestingObtain user opinions about a game Questions concern levels of difficulty, fun, engagementFollows in-house QA beta-testingTypically involves 100...103 participants

Games ResearchGames ResearchGames ResearchGames Research

Usability ResearchUsability Research: evaluate perceived and actual effectiveness of the design Provide guidance for specific design decisions using a variety of methods involving the target audienceTypically involves 101...102 participants

Games ResearchGames ResearchGames ResearchGames Research

Efficacy ResearchEvaluate the ability of the game to have the intended outcomes–cognitive, affective, or skills outcomesTypically involves >103 participants

Class ActivityClass ActivityClass ActivityClass Activity

What are the function and process of an Expert evaluation/ Heuristic review?

What research questions are addressed?What data is collected?What happens with the results?

PlaytestPlaytestPlaytestPlaytestResearch Design

Focussed Research QuestionsSampling of ParticipantsPlayer Group Sizes and CompositionLocation of Testing (Lab, field)StandardizationReference DataHeuristics (experienced-based methods)

Group ActivityGroup ActivityGroup ActivityGroup Activity

Play the Angles Game. Devise 3 research questions for playtesting of this game (15 min., Groups of 3–4)

What research questions are most important?What function would the answer serve?How could the question be addressed?

PlaytestPlaytestPlaytestPlaytest

Research MethodsQuestionnairesInterviewsThink-Aloud ProtocolsVideo ObservationsUser Logs (Events, User actions)

PlaytestPlaytestPlaytestPlaytestQuestionnaires

Learner/Player ProfileLearner VariablesLearner ResponsesConsider

Types of questionsQuestion contentQuestion wordingResponse formatQuestion placement and sequencing

PlaytestPlaytestPlaytestPlaytestQuestionnaires

Intrinsic/Extrinsic Motivation Scale

PlaytestPlaytestPlaytestPlaytestQuestionnaires

Self Esteem Scale

PlaytestPlaytestPlaytestPlaytestChild Questionnaires

PlaytestPlaytestPlaytestPlaytestParent Questionnaires

PlaytestPlaytestPlaytestPlaytestInterviews

Structured, Semi-Structured, or Open-EndedSemi-Structured

How much did you like this game?

Would you play this game again in the future?

What did you like most about this game?

What did you like the least?

Did you make enough progress?

PlaytestPlaytestPlaytestPlaytest

Think-Aloud ProtocolsLearner utterances during play (more on that next week)

PlaytestPlaytestPlaytestPlaytestVideo Observations

Video Recording of Player Behavior and Game PlayAllows for later analysis of a variety of issues

PlaytestPlaytestPlaytestPlaytestUser Logs

Instrumented simulation or game writes game events and user actions to a log file or databaseNeed to specify what should be recorded

Highest Level: Every detail, allowing play back of gameLowest Level: Only key events are recorded

PlaytestPlaytestPlaytestPlaytestUser Logs

id--Uniquely identifies each row of the tablegame-A string identifiying the name of the game (with it's version number)treatment-"Conceptual" or "Arithmetic"userid-the user's id.  This is not required to be unique so it's up to us to make sure we don't reuse these, timestamps can help disambiguateusername-The user's name that they typed in themselvestimestamp-The date/time the rows was inserted into the table, rounded to the nearest whole secondlocation--The chapter&level or screenname where the event took placewho-The name of the entity who issued the event (for example, "ANGLE EDF", or "RIGHT ANGLE BUTTON")what-The content of the message (for example, "SELECTED", or "Correct")extra- Additional values pertinent to the message, but is often unused

PlaytestPlaytestPlaytestPlaytestUser Logs

CyberBuch Böll Survey Monday, November 18, 1996

12:15:38 PM Start using stack

12:15:38 PM Environment: Annotations: TXTE PIC MOV Comprehension: Off Vocabulary: Off Forced lookup: On

12:15:38 PM User Name Clarke XXXX

12:15:40 PM Open Card 1

12:18:44 PM ärmlich TextEngl

12:18:54 PM ärmlich Picture

12:19:04 PM döst Picture

12:19:16 PM döst TextEngl

12:19:49 PM See Picture

12:19:56 PM See TextEngl

12:20:11 PM Wellenkämmen TextEngl

12:20:20 PM Wellenkämmen Movie

PlaytestPlaytestPlaytestPlaytestUser Logs

PlaytestPlaytestPlaytestPlaytestUser Logs

PlaytestPlaytestPlaytestPlaytestUser Logs

PlaytestPlaytestPlaytestPlaytest

Research SetupLab or Field settingEquipment to use for studyRole of ExperimenterInternet access

PlaytestPlaytestPlaytestPlaytestTechnical Setup

Data Capture from XBox, PS3

PlaytestPlaytestPlaytestPlaytestTechnical Setup

Data Capture from PC

PlaytestPlaytestPlaytestPlaytestPlaytest Research Plan

Research Questions Research Design (player groups & composition, location)Participant profile criteriaSampling PlanMethod (Survey Instruments, Procedure, Technical Setup)Analysis of Data

Group ActivityGroup ActivityGroup ActivityGroup Activity

Develop your own research plan for playtesting of the Angles Game! (15 min., Groups of 3–4)

Research Questions Research Design (player groups & composition, location)Participant profile criteriaSampling PlanMethod (Survey Instruments, Procedure, Technical Setup)Analysis of Data

StatisticsStatisticsStatisticsStatistics

Descriptive StatisticsFrequency DistributionMeasures of Central TendenciesVariabilityCorrelationsMean ComparisonStatistical Power and Effect Size

StatisticsStatisticsStatisticsStatisticsFrequency Distribution

StatisticsStatisticsStatisticsStatistics

Measures of Central Tendency

MeanModeMedian

StatisticsStatisticsStatisticsStatisticsVariability

Range, Variance, Standard Deviation

StatisticsStatisticsStatisticsStatisticsVariability

Range, Variance, Standard Deviation

StatisticsStatisticsStatisticsStatisticsMeans Comparison

T-test

StatisticsStatisticsStatisticsStatisticsStatistical Power and Effect Size

Out of Class ActivityOut of Class ActivityOut of Class ActivityOut of Class Activity

Play a game!Play a computer/console game at home

Study a game!What research questions do you find interesting exploring in the game you played?

Extra SlidesExtra SlidesExtra SlidesExtra Slides

Backup Slides to follow

Cognitive Design Cognitive Design FactorsFactors

Cognitive Design Cognitive Design FactorsFactors

Evidence-centered Design Define educational goals of gameDefine outcomes (e.g., desired knowledge of learners)Define and identify acceptable evidence for learningDesign activities to learn and provide evidence of learning

Game ResearchGame ResearchGame ResearchGame Research

Research Frameworks

Research FrameworkResearch FrameworkResearch FrameworkResearch Framework

Elements of InquiryElements of Inquiry Approaches to Approaches to ResearchResearch

Design Process of Design Process of ResearchResearch

Alternative Knowledge Claims

Qualitative Questions

Strategies of Inquiry Quantitative Theoretical Lens

Methods Mixed Methods Data Collection

Data Analysis

Report

Validation

Research FrameworkResearch FrameworkResearch FrameworkResearch Framework

Alternative Knowledge ClaimsPostpositivist knowledgeSocially constructed knowledgeAdvocacy/Participatory Knowledge Pragmatic Knowledge

Research FrameworkResearch FrameworkResearch FrameworkResearch FrameworkStrategies of Inquiry

QualitativeEthnographiesGrounded TheoryCase StudiesPhenomenological ResearchNarrative Research

QuantitativeExperimentsQuasi-ExperimentsCorrelational

Mixed Methods

Research FrameworkResearch FrameworkResearch FrameworkResearch Framework

MethodsSurveys & QuestionnairesField observationsOpen-ended interviewsKnowledge TestsPsychometric ScalesLog filesBiometricsEEG, EKG, EMG, NILS, GSR

OverviewOverviewOverviewOverviewLearning Game Design Model (Plass, 2010)

Formalize IdeasEvaluate Results

Test Ideas

Physical PrototypeDigital Prototype

Brainstorming

Design Documentation

ProductionRefine and PolishQuality Assurance

Educational Effectiveness

No problems Problems with Design

Analyze Requirements

Generate Learning

Ideas

Generate GamingIdeas