cpsx: measurement of collaborative problem …...cps x: measurement of collaborative problem solving...
TRANSCRIPT
![Page 1: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/1.jpg)
CPSX: Measurement of Collaborative Problem Solving Skills
Alejandro Andrade, Pravin Chopade, David Edwards,
Scott Pu and Bryan Maddox
![Page 2: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/2.jpg)
2
Competencies to Succeed in the 21st Century
![Page 3: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/3.jpg)
Challenge of Evidence Identification
3
![Page 4: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/4.jpg)
AI based innovations in Evidence Identification
4
Body Posture:
Relaxed
Facial Expression:
Smiling, Eye Contact
Voice:
Calm
Overall
State:
Engaged
Multimodal Analytics (NLP+CV+ML)
![Page 5: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/5.jpg)
AI based innovations in Evidence Identification
5
High-level inferences
Mid-level representations
Low-level features
SVM HMM
HOG MFCCEuler Angles
Multimodal data
Video 3D Audio
Dat
a Th
eory
Bayesian Networks, JHCRFs
Speech, Prosody, Facial Expressions, Gestures
Turn-taking, Engagement, Affect
Collaborative Problem Solving
Knowledge Assimilation
Positive Communication
Resource Sharing
Deep Hierarchical Inference Models
Khan (2017). In Innovative Assessment of Collaboration. Springer.
![Page 6: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/6.jpg)
Measuring Collaborative Problem Solving Skills
![Page 7: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/7.jpg)
Using Games for Education?
![Page 8: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/8.jpg)
Collaborative Game: Crisis in Space (CIS)
![Page 9: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/9.jpg)
Operator Engineer
Player Interfaces
![Page 10: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/10.jpg)
Gameplay SegmentsG
ame
Lau
nch Select Role
Create Match
Complete Tutorial
Mis
sio
n 1 Wires
Symbols
Intermission
Mis
sio
n 2 Wires
Symbols
Asteroid
Intermission
Mis
sio
n 3
Symbols
Asteroid
Maze
Intermission
Mis
sio
n 4 Wires
Asteroid
Maze
Solar Panel
Intermission
Mis
sio
n 5
Symbols
Asteroid
Maze
Satellites
Done
The game is divided into Five missions – a total of 16 tasks must be completed
Players alternate roles between missions
![Page 11: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/11.jpg)
Operator Engineer
4 Wires
CIS Task: Circuit Board
![Page 12: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/12.jpg)
13
• Involves two inter-related components:
• Cognitive: Traces the phenomenon of macro-cognition wherein team members transform data into information and then into knowledge that enables taking action and monitor problem solving.
• Socio-Emotional: Collaboration requires building and maintaining relationships among team members within a healthy and open emotional environment.
Collaboration in the CIS Game
![Page 13: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/13.jpg)
14
CPSX Skills Map
Cognitive Socio-Emotional
CPSX Skills
Leadership Cooperation Relationships ResilienceIndividual
Knowledge Building
Team Knowledge
Building
Decision Making
and Execution
Monitoring
![Page 14: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/14.jpg)
Experimental Design and Data Sources
CPS Human-Human (X) interaction equipment setup and telemetry data collection.
![Page 15: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/15.jpg)
Players During Gameplay Session
![Page 16: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/16.jpg)
17
Machine Learning based Evidence Identification and Inference
![Page 17: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/17.jpg)
18
Multimodal Data Fusion for Annotation and Feature Extraction
https://www.youtube.com/watch?v=QmuLeBKnvF0&list=PLw2QJrKWPUyJe0pS4g91DSEwjiReX03ut
UserCamera
Collaborator Skype Video
GameFrame
Gaze poin
ts
![Page 18: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/18.jpg)
Annotation of ‘high-level’ behaviors
![Page 19: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/19.jpg)
GRP13_E_P16
Tracking Player Affective State
![Page 20: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/20.jpg)
Okay i think i'm live and i see you in the corner of here so you know i think i'm good yeahstart recording the action start by clicking on this way have tio okay i r the wrenches in the erstarting from left bottom it's three to the from the left and then what okay i got a right corner one in one day okay go three down and three left to left ok now the fires and the upper right corner but it's one down okay go to write three up one right right that's right no thiss one doubt for me it's all the way to the left oh ok down too oh i can't do that okay where are you ?
Okay i think i'm live and i see you in the corner of here so you know i think i'm good yeah start recording the action start by clicking on this way have tio okay i r the wrenches in the er starting from left bottom it's three to the from the left and then what okay i got a right corner one in one day okay go three down and three left to left ok now the fires and the upper right corner but it's one down okay go to write three up one right right that's right no thiss one doubt for me it's all the way to the left oh ok down too oh i can't do that okay where are you ?
21
Automate Speech Recognition for Player Transcripts
![Page 21: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/21.jpg)
22
NLP Based Discourse Analysis
• Questions (“What did you say?”)
• Describe task to partner (“I see symbols”; “There is a tool and a fire”)
• Monitoring time and strikes (“We only have five seconds left”)
• Closing Loop (“Roger that”)
• Sentiment Analysis• Both from Keywords (+/-) and Waveform
(pitch, tonality, etc.)
Transformer Networks, e.g. BERT
Dyad Utterances
𝑔(𝑠) 𝑔(𝑡) cos 𝜃
USE
![Page 22: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/22.jpg)
23
CPSX Skills Map
Initiative Responsiveness AttentivenessDisplay of Emotions
EncouragingRemains Positive
Individual Information Gathering
Individual Information
Synthesis
Information Exchange
Shared Knowledge
Suggest Course of
Action
Implement Action
Evaluate Solution Quality
![Page 23: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/23.jpg)
CPSX Skills Map
![Page 24: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/24.jpg)
Individual Information Gathering
Individual Information
Synthesis
Information Exchange
Shared Knowledge
Suggest Course of
Action
Implement Action
Evaluate Solution Quality
• Name the Circuit task (O)• State number, color and
sequence of wires (O)
• Locate the Circuit task (E)• Ask question (E/O)
• Map number, color and sequence to rule (E)
• Share information from partner’s viewpoint (E/O)
• Cut wire (O)
• State wire to be cut (E)
• Assess solution (E/O)
• Monitor time and/or strikes (E/O)
Inference Model – Accumulate Evidence of CPS Skills
![Page 25: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/25.jpg)
Pilot Studies: CPSX Game Data Collection
Type of
Data
Attributes of Data
Survey
Data
• Demographics: Gender, Game-play experience
• CPSX Survey
• Tessera SEL
In-game
Data• Chat logs + Audio = Transcript
• Video (User camera & Screen Recording)
• Game Logs/Telemetry
• Eye Tracking & Areas of Interest
![Page 26: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/26.jpg)
CPSX Pilot 2: Iowa City area schools
![Page 27: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/27.jpg)
Post Gameplay Survey
High Very HighModerateLowVery Low
![Page 28: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/28.jpg)
Post Gameplay Survey
![Page 29: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/29.jpg)
Feedback & Reporting interface
![Page 30: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/30.jpg)
KCRG Coverage
https://www.kcrg.com/content/news/ACT-asks-students-to-solve-crisis-in-space-video-game-as-part-of-collaboration-study-509924761.html
![Page 31: CPSX: Measurement of Collaborative Problem …...CPS X: Measurement of Collaborative Problem Solving Skills saad.khan@act.org Alejandro Andrade, Pravin Chopade, David Edwards, Scott](https://reader033.vdocuments.net/reader033/viewer/2022042414/5f2e633f6e6487454a0ad67c/html5/thumbnails/31.jpg)
Contact: Saad KhanDirector AI and Machine Learning, ACTNext