mcsig chi 2008
DESCRIPTION
A presentation of the McSig system for allowing a teacher to work with a visually impaired student when learning to write. Presented at CHI 2008.TRANSCRIPT
![Page 1: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/1.jpg)
McSig - A Multimodal Collaborative Handwriting
Trainer for Visually-Impaired People
Beryl Plimmer, Rachel BlagojevicUniversity of AucklandAndrew Crossan, Stephen BrewsterUniversity of Glasgow
www.cs.auckland.ac.nz/~beryl www.dcs.gla.ac.uk/~stephen
![Page 2: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/2.jpg)
Outline
Motivation Why is writing important for a visually impaired person Why is a difficult thing to train
How can we train users to draw a letter shape One solution – haptic trajectory playback How good are people at recalling a trajectory they’ve been
dragged through
McSig Iterative Design Evaluation
Conclusions
2
![Page 3: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/3.jpg)
Why do Visually Impaired People Need to Write ?
Signature Difficult without visual
feedback Important for Job Applications,
Legal documents etc. One participant described her
signature as ‘resembling the meanderings of an inebriated fly’
More general problem with spatial data Presenting Creating
3
![Page 4: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/4.jpg)
Accessible Computing
Magnifiers Need some sight
Screen Readers Synthesized speech
reads text from the screen
Not good for non-text information
Dynamic Braille Non-text or spatial data
is very difficult to present non-visually Maps, charts, graphs,
diagrams, web pages
4
![Page 5: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/5.jpg)
Force Feedback Haptic – Relating to the sense of touch
Force feedback Tactile feedback
Force feedback Forces can be exerted through motors Can be arranged to form 3D objects Can be used to directly influence the movements of the
user We use a PHANTOM OMNI from SensAble
Provides 6 dof in, 3 dof out Produces rich feedback suitable for many general
purpose applications Max 3 Newton force so the user can overpower the
device5
![Page 6: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/6.jpg)
Force Feedback as an Accessibility Aid Non-visual communication channel Can directly affect the user’s exploration
Potential for collaborative applications Research question
Can trajectory playback techniques be used to communicate shape and trajectory information to visually impaired people ?
6
![Page 7: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/7.jpg)
Initial Studies Learning abstract trajectories through haptic
guidance drag a user through a set of pre-defined trajectories Initially significantly harder for visually impaired users &
congenitally blind participants found the task harder still But combining haptic guidance with a sonification of the
trajectory (pitch shift & panning a sinusoidal tone) significantly improved learning
7
Audio Pan
Pitch ShiftDrawing
Area
![Page 8: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/8.jpg)
McSig: Multimodal, collaborative handwriting and signature training tool Teacher and student work
synchronously on shared representation sat next to each other Haptic guidance and audio feedback Teacher guides the student to learn
letter shapes using words and actions Student holds PHANTOM Omni
pen, teacher uses a stylus & touchscreen Teacher can move student’s pen
around the character shape Audio feedback as shape is drawn
Left/right movements: pan, up/down movements: pitch
8
![Page 9: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/9.jpg)
McSig – system design Simulate standard school learning scenario Teacher can choose between three modes
Collaborative Playback mode: Student holds PHANTOM Omni pen student dragged through shape as it is drawn by teacher
Free Stencil mode: Teacher draws letter which is used as a virtual stencil which
thestudent can then explore Can reduce constraining forces as student gets more
experienced
Free drawing mode Student moves Omni pen freely within area. Movements echoed on teacher’s screen
9
![Page 10: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/10.jpg)
First evaluation
1 blind adult Feedback key to learning
User tried to feel what they had drawn on the paper
Added Dutch drawing board – shape is raised on the paper
Can be felt with other hand
Stencil didn’t work very well
10
![Page 11: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/11.jpg)
Usability testing 4 visually impaired adults (3 blind, 1 partially sighted)
Familiarization Letter set ‘o,c,a,d,e’ Started with playback mode and then moved on to stencil Finally drew the letter unsupported
Stencil mode hard to use Strengthened forces for shapes to give clearer path Still didn’t work very well
Audio feedback useful to some, teacher descriptions most useful Speech feedback error prone and reduced confidence of users Omni pen difficult to hold, plus pressing button whilst drawing
tricky Users not used to holding pens Gave some pen training before main study
11
![Page 12: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/12.jpg)
Evaluation Could McSig improve handwriting performance?
Task designed with teachers Some children almost no handwriting skills, some have
good skills 5 characters chosen after discussion with teachers
o, c, a, d, e Participants
8 children 11-17 years old, read Braille, no other major disabilities
3 partially sighted, 5 blind 4 stage study
Familiarization with McSig, then for each letter: Pre-test McSig training Post-test
12
![Page 13: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/13.jpg)
Familiarization Participants could feel setup,
PHANTOM, mat, PC Spatial orientation Drew circle, horizontal and vertical
lines Practised with the pen
Visually impaired people don’t commonly use pens
13
![Page 14: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/14.jpg)
14
![Page 15: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/15.jpg)
McSig: Pre-Test Training and Post-Test Pre-test: participants asked to draw each letter as
best they could Some unable to draw one or more of them
Training: teacher showed participant how to draw letter in Playback mode Experimenter wrote shape on screen, child felt it with
PHANTOM and scored line on tactile sheet Synchronous audio/haptic/tactile feedback Number of repeats based on child’s confidence
Post-test: participants draw character in free draw mode If participant could not draw it we trained and tested
again Time-out after 20 mins
Stopped earlier if all letters done
15
![Page 16: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/16.jpg)
Results – partially-sighted children
Participants All could read enlarged print All had deteriorating sight but had learned to write
when sight was better Did not write now as sight too bad
Familiarized very quickly, could all do circle, horizontal and vertical lines, no problem
One participant did all of our letters in the pre-test
Both of the others started ‘d’ in wrong place
Pre Post16
![Page 17: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/17.jpg)
Results – partially-sighted children
All letter correct except One did a normal ‘e’ in mirror image
Participants had eyes close to drawing surface but did not feel drawing surface with non-dominant hand Wanted to use their sight
All trained quickly and did all letters correctly in post-test Completed within 20 mins
Politely interested but not captivated
17
![Page 18: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/18.jpg)
Results – blind children
Participants 5 totally blind One lost her sight at 3 years, others blind from
birth Familiarization took much longer
Pressure on pen – too much/too little Interacted with drawing space very differently
Non-dominant hand for orientation in space All but one could draw circle and lines Before and after examples
18
![Page 19: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/19.jpg)
Before and after
19
![Page 20: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/20.jpg)
20
![Page 21: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/21.jpg)
Participant who had sight until age 3
21
![Page 22: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/22.jpg)
Discussion - Technology
Training Student independent
Input - teacher Stylus pen Verbalization
Output Visualization on teacher’s
display Phantom Force feedback Sound pan and pitch Tactile trace Recognition to voice
output
Input Phantom pen position
Output Visualization on teacher’s
display
Sound pan and pitch Tactile trace Recognition to voice output
22
![Page 23: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/23.jpg)
Discussion
Results suggest that McSig could help children to learn Especially blind children
Why didn’t stencil mode work? No tactile representation of the letter for non-
dominant hand Therefore an inconsistent interface
23
![Page 24: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/24.jpg)
Future Work Cursive handwriting and signatures
Support move from single letters to cursive A signature can be created and then practised to
keep it consistent over time A self-teaching tool Wider context
Could be used in any application where the teacher wants to guide student Geometry, 2D and 3D shapes Charts and graphs
24
![Page 25: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/25.jpg)
Conclusions Hard for visually-impaired people to learn to
handwrite Signatures difficult to learn and keep consistent Required for important aspects of life
McSig: a collaborative tool that allows a teacher to guide a student to handwrite letter shapes Dynamic haptic and audio feedback
Can improve handwriting in 20 minute session All blind students learned at least 2 new letters Enjoyed the experience
Now working on longer-term study to see how learning develops over time
25
![Page 26: McSig Chi 2008](https://reader035.vdocuments.net/reader035/viewer/2022070318/5575f04bd8b42af74e8b525e/html5/thumbnails/26.jpg)
McSig - A Multimodal Collaborative Handwriting
Trainer for Visually-Impaired People
Beryl Plimmer, Rachel Blagojevic
University of Auckland
Andrew Crossan, Stephen Brewster
University of Glasgow
www.cs.auckland.ac.nz/~beryl
www.dcs.gla.ac.uk/~stephen
Thanks to our participants, John Williamson and Malcolm Hall at Glasgow University and our sponsors University of Auckland and the EU FP6 MICOLE Project