give-me: gamification in virtual environments for ...in virtual environments for multimodal...
TRANSCRIPT
GIVE-ME: Gamification In Virtual Environments
for Multimodal Evaluation –
A FrameworkWAI KHOO
DEPARTMENT OF COMPUTER SCIENCE
THE GRADUATE CENTER,
CITY UNIVERSITY OF NEW YORK
Committee Members
Prof. Zhigang Zhu ◦ City College, Computer Science
Prof. Yingli Tian ◦ City College, Electrical Engineering
Prof. Tony Ro ◦ The Graduate Center, Psychology
Dr. Aries Arditi◦ Visibility Metrics LLC
2
Outline
§ Motivation§ Research questions§Approach§ Proposed framework§ 4 applications of the framework§ Conclusion & Future work
Funding supports:1. NSF Awards # ◦ Emerging Frontiers in Research
& Innovation (EFRI) 1137172◦ Chemical, Bioengineering,
Environmental, and Transport Systems (CBET) 1160046
◦ Industrial Innovation & Partnerships (IIP) 1416396
2. VentureWell (formerly NCIIA, through Award # 10087-12).
3. CUNY Graduate Center Science Fellowship (2009 –2014)
3
NYCPenn
Station
Source: Jason Gibbs. Retrieved from http://jasongibbs.com/pennstation/ on March 19, 2016
4
A B
Travel Aids
5
Guide Dog Talking GPS Miniguide White cane
Argus II, retinal implant
Brainport
Background
050100150200250300
2002 2014
WorldwideVisuallyImpairedPopulation(inmillion)
6
161 285
77%
Outline
§ Motivation
§ Research questions
§Approach
§ Proposed framework
§ 4 applications of the framework
§ Conclusion & Future work
7
Research Questions
1. How to establish a benchmark for heterogeneous systems?
2. How to provide a well-controlled and safe testing environment?
3. How to provide a robust evaluation and scientificcomparison of the effectiveness and friendliness of multimodal assistive technologies?
8
9
Inspiration
Jason Park, Helen MacRae, Laura J. Musselman, Peter Rossos, Stanley J. Hamstra, Stephen Wolman, Richard K. Reznick, Randomized controlled trial of virtual reality simulator training: transfer to live patients, The American Journal of Surgery, Volume 194, Issue 2, August 2007, Pages 205-211
Outline
§ Motivation
§ Research questions
§Approach
§ Proposed framework
§ 4 applications of the framework
§ Conclusion & Future work
10
Approach
VirtualReality
Gami-fication
Multi-modality
Unified formal evaluation and comparison approach.
Differ from Degara et al. (2013)[1]◦Only sounds
Differ from Lahav et.al (2012) [2] and Huang (2010) [3]◦ Focused on cognitive mapping in unknown space.
11
Virtual Reality
Use a game engine to design a virtual environment and simulate part of an assistive technology.
Benefits:◦Rapid prototyping◦ EARLY user involvement◦ Psychophysics evaluation◦ Safe & well-controlled environment for navigation tasks
12
Gamification
13
Use game design elements for research & evaluation.
Benefits:◦ Fun/engaging experiment sessions◦ Sustainable evaluation◦Crowd-sourcing data collection◦ Package designed VE as a simulation/training tool
Multimodality
Multimodal input and output of data
Benefits:◦ Enable alternative perception (sensory substitution)◦ Allows a mixture of input and output devices.
14
15
Who Benefits from My Research?
Researchers/developers
Assistive technology companies
Visually impaired users
Outline
§ Motivation
§ Research questions
§Approach
§ Proposed framework
§ 4 applications of the framework
§ Conclusion & Future work
16
Proposed Framework: Gamification in Virtual Environments for Multimodal Evaluation (GIVE-ME)
17
Framework: User Interface
18
Framework: Foundation
19
GIVE-ME Software Impl.
20
http://ccvcl.org/~khoo/GIVE_ME.unitypackage
• Minimal coding
• Click-&-drag
• Fully customizable
Package
Outline
§ Motivation
§ Research questions
§Approach
§ Proposed framework
§ 4 applications of the framework
§ Conclusion & Future work
21
Application 1: VibrotactileNav
22
Vista Wearable, Inc.
Application 1: VibrotactileNav
23
Application 1: VibrotactileNav
24
WaiL.Khoo,JoeyKnapp,FranklinPalmer,TonyRo,andZhigangZhu.Designingandtestingwearablerange-vibrotactiledevices.JournalofAssistiveTechnologies,7(2):102-117,2013.
Controller:Joystick/mouse
Stimulators:VibratorsSounds
Virtual sensor:Infrared
VibrotactileNav Results
25
VibrotactileNav Results
26
VibrotactileNav ResultsEASY HALLWAY
9 out of 18 succeeded
Succeeded:
Avg time: 280.10 sec
Avg bumps: 17.3
Failed:
Avg time: 288.65 sec
Avg bumps: 22.1
COMPLEX HALLWAY
3 out of 18 succeeded
Succeeded:
Avg time: 120.25 sec
Avg bumps: 12.7
Failed:
Avg time: 353.67 sec
Avg bumps: 42.727
EEG Data Collection
28
Application 2: BrainportNav
29
Wicab, Inc.
Application 2: BrainportNav
30
Margaret Vincent, Hao Tang, Wai L. Khoo, Zhigang Zhu, and Tony Ro. Shape discrimination using the tongue: Implications for a visual-to-tactile sensory substitution device. Multisensory Research, 2016.
Controller:Joystick
Stimulators:Electrode array
Virtual sensor:Camera in VEPathfinding
31
BrainportNav Results
Run1Time=151secondsAccuracy=0.95
Run2Time=91secondsAccuracy=0.95
32
BrainportNav Results
Avg accuracy:
83.33%
82.66%
93%
69.66%
33
Application 3: CrowdSourceNav
34
Application 3: CrowdSourceNav
OR
TCP/IP Conn
Stream Game View
Can be used for testing algorithms
UsabilityStudy
Wai L. Khoo, Greg Olmschenk, Zhigang Zhu and Tony Ro, "Evaluating Crowd Sourced Navigation for the Visually Impaired in a Virtual Environment," in Mobile Services (MS), 2015 IEEE International Conference on , pp.431-437, June 27 2015-July 2 2015
Controller:Joystick
Stimulator:Text-to-speech
Virtual Sensor:Camera in VE
Online crowd members
35
CrowdSourceNav Results
Maze 1Time = 514 secNumBump = 7
Maze 2Time = 345 secNumBump = 0
36
11 crowd members
Crowd completion time for either aggregation method is not significantly different (two-sample t-test, p=0.432 at 5% significance level, df=6)
CrowdSourceNav Results
37
Sample size of 11
Rating of 1 – 7 to the following statements1. It is useful2. It is easy to use3. It is user friendly4. I learned to use it quickly5. I am satisfied with it
CrowdSourceNav Results
CrowdSourceNav Real Exp.
38
5.5
m
0.5
m
3 m
0.5 m0.5 m
2 m
1 m
1 m
1 m
1 m
1 m
1 m1
m
1 m
1 m
1 m
2.235 m Similarities
• Simple average aggregation method
• Speech feedback
Differences
• Random obstacles
• Stream from cam’s view
Pros
• Skill transfer: “Preemptive” instructions
• Provided minimal training to crowd volunteer
Cons
• Various camera’s angle and height
• Various walking speed
Application 4: VistaNav
39
Vista Wearable, Inc.
Experiments
40
1. Configurations Experiment1. Four configs
2. Training Experiment1. Half with VE
training
VR setup
41
Controller:Xbox 360 game pad
Stimulators:VibratorsSounds
Virtual sensor:Infrared
VistaNav Results – Configurations
42
VistaNav Results - Configurations
43
VistaNav Results - Configurations
44
Two-way repeated measures ANOVA:
F(3,51) = 10.54, p = 0.00
Multiple comparisons w/ Bonferroni correction:
C3 vs. C4, C6 (p = 0.00)
C3 is the best!
VistaNav Results -Training
45
• Training session• Virtual hallway• 10 minutes• Free exploration• Audio & haptic feedback• 3 VISTA devices; one on each wrist and one on chest
• Testing session• Real U-shaped hallway (71 ft x 52 ft)• Sighted subjects are blindfolded• 2 VISTA devices; one on each wrist• Goal: reach destination w/o bumping into obstacles
VistaNav Results -Training
46
• 17 out of 21 subjects included in analysis• 2 outliers are trimmed from each group (training vs.
no training)
VE training significantly improved the performance in real hallway navigation.
t(15) = -1.91, p = 0.04, two-sample, one-tailed
Training Mean SD
YES 249.00s 92.45s
NO 333.78s 90.76s
VistaNav Results -Training
47
• A usability questionnaire is given at the end of the hallway• System Usability Scale (SUS)• 10 questions with 5 Likert-scale responses• Strongly disagree – strongly agree• 5 positive and 5 negative statements, which
alternate
VistaNav Results -Training
48
VistaNav Results -Training
49
Overall mean = 80.48
Overall SD = 13.62
Outline
§ Motivation
§ Research questions
§Approach
§ Proposed framework
§ 4 applications of the framework
§ Conclusion & Future work
50
GIVE-ME ContributionsUnified evaluation framework for multimodal navigational assistive technologies.
Sustainable evaluation that is fun and engaging.
Novel psychophysics evaluation for navigation tasks.
Novel collaborative environment that promotes early stakeholder involvement.
51
Future Work§ Improvement on framework implementation§New sensors, stimulators, and environments.
§ Determine metrics to evaluate ATs§ Survey population and experts
§ Brain data collection§Collect brain data to better understand how VIPs perform
visual task such as navigation
§ Comprehensive benchmark§ Large scale evaluation based on selected metrics
52
53
PublicationsBookChapters:1. E.Molina,W.L.Khoo,H.TangandZ.Zhu(2017).RegistrationofVideoImages.InA.A.
Goshtasby (Ed.),TheoryandApplicationofImageRegistration.(Invited)Hoboken,NJ:WileyPress.
Peer-reviewedJournals:1. M.Vincent,H.Tang,W.L.Khoo,Z.Zhu,T.Ro.ShapeDiscriminationusingtheTongue:
ImplicationsforaVisual-to-TactileSensorySubstitutionDevice.MultisensoryResearch(Pendingafinaldecisionafteraminorrevision)
2. W.L.Khoo,G.Olmschenk,Z.Zhu,H.Tong,W.H.Seiple,andT.Ro.DevelopmentandEvaluationofMobileCrowdAssistedNavigationfortheVisuallyImpaired.IEEETransactionsonServicesComputing(Pending;GOandWKequalcontribution).
3. W.L.KhooandZ.Zhu.MultimodalandAlternativePerceptionfortheVisuallyImpaired:ASurvey.JournalofAssistiveTechnologies10(1).pp.11-26.2016.
4. W.L.Khoo,J.Knapp,F.Palmer,T.Ro,andZ.Zhu.(2013).DesigningandTestingWearableRange-Vibrotactile Devices.JournalofAssistiveTechnologies,7(2).
PatentsPending&Provisional1. Z.Zhu,T.Ro,L.Ai,W.L.Khoo,E.Molina,F.Palmer.WearableNavigationAssistancefor
theVision-impaired,December27,2013.USPatentApp.14/141,742(pending)
54
PublicationsConferenceProceedings:1. Z.Zhu,W.L.Khoo,C.Santistevan,Y.Gosser,E.Molina,H.Tang,T.Ro,andY.Tian.EFRI-REMatCCNY:
ResearchExperienceandMentoringinMultimodalandAlternativePerceptionforVisuallyImpairedPeople.6thIEEEIntegratedSTEMEducationConference(ISEC'16),March5,2016,Princeton,NJ.
2. E.Molina,W.L.Khoo,F.Palmer,L.Ai,T.RoandZ.Zhu.VistaWearable:SeeingthroughWhole-BodyTouchwithoutContact.IEEE12thInternationalConferenceonUbiquitousIntelligenceandComputing,August10-14,2015,Beijing,China.
3. W.L.Khoo,G.Olmschenk,Z.Zhu,andT.Ro.Evaluatingcrowdsourcednavigationforthevisuallyimpairedinavirtualenvironment.InIEEE4thInternationalConferenceonMobileServices,pp.431-437.2015
4. W.L.Khoo,E.L.Seidel,andZ.Zhu.DesigningaVirtualEnvironmenttoEvaluateMultimodalSensorsforAssistingtheVisuallyImpaired.13thInternationalConferenceonComputersHelpingPeoplewithSpecialNeeds(ICCHP),7383,SpringerBerlinHeidelberg,July11-13,2012,Linz,Austria,573-580
5. A.Khan,J.Lopez,F.Moideen,W.L.Khoo,andZ.Zhu.KinDetect:KinectDetectingObjects.13thInternationalConferenceonComputersHelpingPeoplewithSpecialNeeds(ICCHP),7383,SpringerBerlinHeidelberg,July11-13,2012,Linz,Austria,588-595
6. Y.Qu,W.Khoo,E.Molina,andZ.Zhu.Multimodal3DPanoramicImagingUsingaPreciseRotatingPlatform.2010IEEE/ASMEInternationalConferenceonAdvancedIntelligentMechatronics,July6th- 9th,2010,260-265
7. W.Khoo,T.Jordan,D.Stork,andZ.Zhu.ReconstructionofaThree-DimensionalTableaufromaSingleRealistPainting,15thInternationalConferenceonVirtualSystemsandMultimedia,September9-12,2009,9-14
8. T.Jordan,D.Stork,W.Khoo,andZ.Zhu.FindingIntrinsicandExtrinsicViewingParametersfromaSingleRealistPainting,13thInternationalConferenceonComputerAnalysisofImagesandPatterns,5702,SpringerBerlinHeidelberg,September2-4,2009,293-300
55
http://ccvcl.org/~khoo/
Questions?
Starting as TT assistant professor at RPI in January 2017!
Any advice?
References
56
1. Norberto Degara, Frederik Nagel, and Thomas Hermann. “SonEX: an evaluation exchange framework for reproducible sonification.” In Proceedings of the 19th InternationalConference on Auditory Displays, 2013.
2. Ying Ying Huang. “Design and evaluation of 3D multimodal virtual environments for visually impaired people.“ PhD thesis, KTH, 2010.
3. Orly Lahav, David Schloerb, Siddarth Kumar, and Mandyam Srinivasan. “A virtual environment for people who are blind-a usability study.” Journal of Assistive Technologies, 6(1):38-52, 2012.
Definition
USA categories:◦ Low vision◦ 20/70 – 20/200
◦ Legal blindness◦ 20/200 or worse
57
VisualImpairmentVisualacuityof20/70orworseinthebetter,even
withcorrection.
Sensorysubstitution/alternativeperceptionTransformationofthecharacteristicsofonesensorymodalityintostimuli
ofanothersensorymodality.
58
Common Eye Disorders
Source: CDC Vision Health Initiative, Common Eye Disorders, http://www.cdc.gov/visionhealth/basics/ced/index.html, Mar 28, 2016
1) Controllers
59
1) Multimodal (Virtual) Sensors
Common multimodal sensors: infrared, sonar, and RGB-D
Infrared (IR) sensor◦ Light-based sensors with a very narrow beam angle◦ Specifications:◦ Minimum distance: 10 cm◦ Detection distance: 80 cm◦ Beam width: 12 cm -> about 8.5 degree beam angle
◦ Limitations:◦ Needs to be pointed exactly at object for detection.◦ Cross interference.◦ Noisy data
60
1) Multimodal (Virtual) Sensors
RGB-D (i.e., with image size width x height pixels)◦ Optical◦ Use the camera (game) view
◦ Depth◦ Use width x height raycasts at each pixel location.◦ Raycasts parallel to the avatar’s positive z-axis (right-hand rule).◦ Maximum range of 4 meters.
61
1) Multimodal (Virtual) Sensors: Transducing
Processing◦ Convert raw data to meaningful information◦ E.g., quantize/threshold range data into 3 intervals (close, near, and
far).
Transmission◦ Send meaningful info to stimulators of another modality◦ E.g., encode the interval with appropriate vibration levels◦ Communication protocols:◦ USB/serial port◦ Bluetooth Low Energy◦ TCP/IP
62
2) Game Mechanics
Provides a set of goals to achieve and define how the user can interact with the game.
Task definition◦ What needs to be done in order to finish.◦ Simple task: Navigate from point A to B, as fast and as few
errors as possible.◦ Complex task: Exploratory in nature with termination
conditions (e.g., time-out).
Avatar Behavior◦ What controller commands are valid.◦ E.g., constant walking speed or turn/rotate in place.◦ Define what audio cues are constantly audible and which are
in response to an action or proximity.
63
3) Virtual Environment Toolbox
Third party game engine◦ Unity3D: popular, excellent documentation, & tutorials◦ C#, Javascript. ◦ Multi-platform.
Controller setup (optional)◦ Standard input devices are compatible with Unity3D. Capture
specific key action events.◦ Other controllers (e.g., Microsoft Kinect) not natively
supported by Unity3D◦ Need an external/separate program capable of
communicating and exchanging data with Unity3D.
64
3) Virtual Environment Toolbox
Environment Design◦ Static/dynamic objects; objects interaction; placing sound cues and
collectibles.
65
4) Data Collection
Type of data:◦ Multimodal sensory data◦ E.g., range data generated by virtual sensors and sounds
◦ Control/action data◦ E.g., User inputs, events, and game state.
◦ Brain/behavioral measurement, as obtained from the Measurement Device◦ E.g., EEG readings and observations
These data contribute to task evaluation and ground truth establishment.
66
67
4) Data CollectionRecommend the following metrics for task evaluation:1. Acceptability: Design that is useful, reliable, robust, aesthetic, and has positive impact
on quality of life of user.2. Compatibility: Design that is compatible with lifestyle of user and other technologies.3. Adaptability: Design that can be easily adjusted (i.e., function, location).4. Friendly: Low learning curve for the system; easy to use.5. Performance: Overall performance
While these metrics can be posed as open-ended questions, it can also be presented as rating surveys. Data to assess friendly and performance features can also be collected and extrapolated from the VE. Data includes but not limited to:1. Time to completion2. Number of errors (e.g., bumping into obstacles, incorrect response)3. Game score4. User’s trajectory5. User’s brain data (e.g., EEG, fMRI)