sensorimotor control and braille reading€¦ · video-recording of finger movements in the 1980s,...
TRANSCRIPT
Sensorimotor control and braille reading
Barry HughesSchool of Psychology
Gunnar Jansson, Arend Van Gemmert, George Stelmach, Jeff Hamm
Amber McClelland, Dion Henare, Vania Glyn, Ash Mathur, Stewart D’Silva, Ayra Baes, Jay Patel, Devanshi Bhavsaar, Daniel Yeom
Pint
eres
t
David Katz, The World of Touch (1925)Chair of Pedagogy, 1937-1952, University of Stockholm.
The tactual properties of our surroundings do not chatter at us like their colors; they remain mute until we make them speak. . . Eye movements do not create color the way finger movements create touch.
Goodreads
Sensorimotor control and braille reading
The hand is a sophisticated perceptual system as well as being a multifunctional motor system.
Braille reading is one of hundreds of activities that involve the hapticperceptual-motor system at work.
But braille reading includes another important element that sets it apart from tactile perception or haptic exploration: language processing.
CB
M In
tern
atio
nal
Reading is not one skill. It is many.
Fluent reading involves the coordination of its component skills which are fast and subconscious.
Reading integrates many of the major constituent processes of the thinking brain: • perception and knowing, • pattern identification or recognition,• memory storage and retrieval, • attentional focus,• decision-making and prediction, • grammar processing and meaning extraction, • planning and controlling the movements of the eyes and body.
But print reading and braille reading engage these processes in different ways.
Print reading vs braille reading
Print reading is intermittent, discontinuous and selective,Braille reading is said to be constant, continuous and exhaustive.
The balloon with a white ribbon floated away..250 .360 .210 .305 .420 .395 .270
The balloon with the white ribbon floated away
The visual field is layered: foveal, parafoveal and peripheral fields and may offer preview.
In braille, only what is beneath the fingerpads is felt, but the number of fingers and hands can be increased.
Braille reading involves no fixations; at least one finger is always moving laterally. Why?
Braille reading is ‘like’ texture perception more generally:It uses an exploratory procedure similar to that deployed for surface roughness judgments, or raised line pictures.
How do people know how-to-move in order maximise perception?
This means that identification of letters is via a constant stream of cells with different patterns, crossing a patch of sensitive skin.
Does this mean braille reading is ‘like’ solving a ‘structure from motion’ problem in vision?
Or might it mean that braille is more ‘like’ listening to speech than reading?
Open questions
Lateral Position of Reading Finger(s)
Tim
e(s
)
Adapted from Bertelson et al (1985). A study of braille reading: 2. Patterns of hand activity in one-handed and two-handed reading. Quarterly J Experimental Psychology, 37A, 235-256.
Video-recording of finger movements
In the 1980s, Paul Bertelson’s group collected beautiful data from videotape.
Smooth movements
Reversals of direction
Scissoring of two hands
Movement Recording and Analysis:
The grip pentip for a digitizing tablet was fitted to a light-weight finger attachment and mounted on the reader’s single dominant reading finger.
Software sampled the position (x-y coordinates) of the pen tip at 100Hz and stored these coordinates for subsequent analyses.
All kinematic data were derived from finger position data: differentiating finger position with respect to time yields velocitydifferentiating finger velocity with respect to time yields acceleration
Understanding how the brain reads braille requires more precise recordings of how the reading finger moves
Left-RightVelocity
Left-RightDisplacement
5
10
15
20
0 1 2 3
x(cm
)
Time (s)
0
2
4
6
8
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5
g
vx(c
m/s
)
Time (s)
Finger velocity is neither smooth nor constant
0
2
4
6
8
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5
p g p
vx(c
m/s)
Time (s)
Why is the finger speed not smooth and not constant?
• Tactile perception: braille creates fluctuating coefficients of friction?
• Linguistic processing: readers adjust speed according to orthographic, lexical, syntactic, semantic demands?
• Motor control: forces for low-velocity movements are pulses not steady states?
Experiments with fluent readers
Participants were all experienced, fluent, daily readers of braille. All were required to read with a single finger. All sentences were novel and fit on a single line. All sentences to be read were rendered in Grade 2 (contracted)
braille . Reading was to be undertaken as fast as possible without sacrificing accuracy.
In order to derive statistically analysable measures of the reading performance we compute various kinematic dependent variables:
1. Mean velocity (cm/s) per sentence2. Intermittency of the velocity trace (technically, the number of acceleration zero-crossings)3. For both forward reading and any reversals (defined as any instant where velocity changes from having a positive to a negative value)
-6
-4
-2
0
2
4
5 10 15 20
vx(c
m/s
)
x(cm)
,Ty 3sid}$ x a will# 3nec;n4
They considered it a willing connection.
,Ty 3sid}$ x a typical faculty4
They considered it a typical faculty.
,Ty 3sid}$ x a 3d5s$ repres.n4
They considered it a condensed repression.
,Ty 3sid}$ x a noxi\s typhoid4
They considered it a noxious typhoid.
Left-Right position (cm)
0 2 4 6 8 10 12 14 16 18 20 22
Fing
er V
eloc
ity (c
m/s
)
-20
-10
0
10
20
30 HH
0 2 4 6 8 10 12 14 16 18 20 22
-20
-10
0
10
20
30 HL
0 2 4 6 8 10 12 14 16 18 20 22
-20
-10
0
10
20
30 LH
0 2 4 6 8 10 12 14 16 18 20 22
-20
-10
0
10
20
30 LL
,Ty 3sid}$ x a will# 3nec;n4
,Ty 3sid}$ x a typical faculty4
,Ty 3sid}$ x a 3d5s$ repres.n4
,Ty 3sid}$ x a noxi\s typhoid4
* Hughes (2011)
0
10
20
30
40
0
10
20
30
40
Left-
Rig
ht V
eloc
ity (c
m/s
)
0
10
20
30
40
Left-Right Position (cm)0 5 10 15 20 25 30
0
10
20
30
40
0 raised dots per cell
1 raised dot per cell
3 raised dots per cell
5 raised dots per cell
Continuous cell repetition
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
lllllllllllllllllllllllllllllllllllllllllllllllllll
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
0
10
20
30
40
Left-
Rig
ht V
eloc
ity (c
m/s
)
0
10
20
30
40
0 raised dots per cell
0
10
20
30
40
Left-Right Position (cm)0 5 10 15 20 25 30
0
10
20
30
40
1 raised dot per cell
3 raised dots per cell
5 raised dots per cell
Noncontinuous cell repetition
aaaaa aaaaa aaaaaa aaaaa aaaa aaaaa aaaaa
llll lllll llllll lllll lllll lllll llllll
yyyy yyyyy yyyyyy yyyyy yyyy yyyyyy yyyyyy
Scanning, not reading
* H
ughe
s, V
an G
emm
ert
& S
telm
ach
(201
1)
Mean Velocity (cm/s)2 4 6 8 10 12 14 16 18
Mea
n N
o. o
f A
ccel
erat
ion
Zero
-cro
ssin
gs
0
100
200
300
400
LN LMHNHM
Silent reading
Mean Velocity (cm/s)2 4 6 8 10 12 14 16 18
Mea
n N
o. o
f A
ccel
erat
ion
Zero
-cro
ssin
gs
0
100
200
300
400
LNLMHNHM
Oral readingOut of the tangle of velocity traces can emerge order:
The faster the average speed of the finger, the fewer fluctuations are observed in the velocity trace.
This is so• whether the text is
meaningless or not, • whether the words are
frequently or infrequently encountered in the language;
• whether the text is being read silently or aloud.
* H
ughe
s, V
an G
emm
ert
& S
telm
ach
(201
1)
Reading Reversals
Sentence ambiguity
Mean velocity (cm/s)
Mean no. zero-crossings (per cm)
Mean velocity (cm/s)
Mean no. zero-crossings (per cm)
Syntactic 3.84 (+0.38) 4.52 (+0.46) 5.47 (+0.27) 2.57 (+0.16)
Lexical 4.02 (+0.42) 4.33 (+0.47) 5.21 (+0.26) 2.96 (+0.21)
None 4.34 (+0.49) 4.10 (+0.48) 4.77 (+0.26) 2.66 (+0.17)
Forward reading and reversals with ‘garden-path’ sentences
* McClelland, Henare & Hughes, 2013
The rich people mansions ...Syntactically ambiguousThe rich inhabit mansions ...Lexically ambiguousThe rich people bought yachts ...No ambiguity
Mean velocity and intermittency of velocity are tightly linked regardless of sentence ambiguity and the direction of finger movements
Mean Reading Velocity (cm/s)0 5 10 15 20
No.
Acc
eler
atio
n Ze
ro-C
ross
ings
(per
cm
trav
elle
d)
0
5
10
15
20
25
Semantic ambiguityLexical ambiguityNone
Syntactic ambiguity Lexical ambiguity No ambiguity
On reversals
Approximately 15% of eye movements by print readers (of English) are from right to left, resulting in re-reading of words already encountered.
Reversals of reading direction occur frequently in braille reading also. Why?
Reversals in braille reading have been called recovery mechanism, but recovery from what?• are they regressions to recover from explicit errors of
comprehension?• are they movements that serve to synchronise the finger position
with comprehension (the ‘finger-mind span’)?• temporary loss of concentration?
We asked readers to silently read sentences to completion, at which point a symbol would indicate which one of the sentence’s two nouns should be reversed to as fast and as accurately as possible.
The sentences came in three forms:
normal: grammatical and coherent The sickly maid took the pill.The motorists smashed the glowing headlights.
reversed: grammatical but not semantically coherentThe tasty eggplants consumed the goat.The aroma smelt the sneaky predators.
scrambled: neither grammatical nor coherentFrail held snapshots the the historian.Costly teens the the admired yacht.
The target noun could be:located early or late in the sentence (i.e., in the noun phrase or the
verb phrase of a normal sentence)short or long in length (4 cells or 8 cells in Grade 2 Braille)
Noun Target (Location and Length)
First Short First Long Second Short Second Long
Mea
n R
eadi
ng V
eloc
ity (c
m/s
)
0
2
4
6
8
10NormalReversedScrambled
First Short First Long Second ShortSecond Long
-10
-8
-6
-4
-2
0
Mea
n R
ever
sal V
eloc
ity (c
m/s
)
A
B
Mean reading velocity differs littleaccording to sentence structure
Mean reversal velocity is faster to nouns whose first letter is farther from the sentence end. But otherwise they differ little.
,! SICKLY MAID TOOK ! PILL4
The sickly maid took the pill.
0 5 10 15 20 25
Left-
Rig
ht F
inge
r Vel
ocity
(cm
/s)
-60
-50
-40
-30
-20
-10
0
10
20
30
Left-Right Position (cm)0 5 10 15 20 25
-60
-50
-40
-30
-20
-10
0
10
20
30
Left-
Rig
ht F
inge
r Vel
ocity
(cm
/s)
The sickly maid took the pill.,! SICKLY MAID TOOK ! PILL4
Normal sentences:grammatical and coherent
0 5 10 15 20 25
Left-
Rig
ht V
eloc
ity (c
m/s
)
-60
-50
-40
-30
-20
-10
0
10
20
30
Left-Right Position (cm)0 5 10 15 20 25
-60
-50
-40
-30
-20
-10
0
10
20
30
Left-
Rig
ht V
eloc
ity (c
m/s
)
,! TA/Y E7PLANTS 3SUME$ ! GOAT4
,! TA/Y E7PLANTS 3SUME$ ! GOAT4
The tasty eggplants consum ed the goat.
The tasty eggplants consum ed the goat.
Reversed sentences:grammatical but not coherent
Left-
Rig
ht V
eloc
ity (c
m/s
)
,Frail held snap%ots ! ! hi/orian4
Frail held snapshots the the historian.
Left-Right Position (cm)
0 5 10 15 20 25-60
-50
-40
-30
-20
-10
0
10
20
30
,Frail held snap%ots ! ! hi/orian4
0 5 10 15 20 25
Left-
Rig
ht V
eloc
ity (c
m/s
)
-60
-50
-40
-30
-20
-10
0
10
20
30
Frail held snapshots the the historian.
Scrambled sentences:Not grammatical, not coherent
Mean Left to Right Reading Velocity (cm/s)0 2 4 6 8 10
Mea
n N
o. A
ccel
erat
ion
Zero
-Cro
ssin
gs
0
50
100
150
200
250
Normal ReversedScrambled
A
Mean Right to Left Reversal Velocity (cm/s)-18-16-14-12-10-8-6-4-20
Mea
n N
o. A
ccel
erat
ion
Zero
-Cro
ssin
gs
(per
cm
reve
rsed
)
0
2
4
6
8
10
12B
The same decaying exponential function captures the relationship between mean reading speed and intermittency
The looser decaying exponential function captures the relationship between mean reversal speed and intermittency
Novices and braille:Sensitivity to numerosity of raised dots
Blindfolded, sighted participants, with no knowledge of braille.
Each was asked to scan a line of braille text –-left to right without stopping or reversing-- and decide whether one cell in a target zone of seven cells, contained a different number of raised dots.
''''''''''YYYYY.Y''''''''''
''''''''''YYYYYYY''''''''''
In one experiment, a single finger was used at self-selected or experimenter-selected velocities.In another, the right index finger alone, or paired with the right middle finger ofr the left index could be used at self-selected velocities.
Condition In guide Target zone Out guide
6:1''''''''''=@=====''''''''''
''''''''''=======''''''''''
1:6''''''''''@@@=@@@''''''''''
''''''''''@@@@@@@''''''''''
5:2 ''''''''''YYYYY.Y''''''''''
''''''''''YYYYYYY''''''''''
2:5 ''''''''''.Y.....''''''''''
''''''''''.......''''''''''
4:3 ''''''''''XXX[XXX''''''''''
''''''''''XXXXXXX''''''''''
3:4''''''''''[[[[[X[''''''''''
''''''''''[[[[[[[''''''''''
A change in raised dot numerosity could occur at one of three locations within the zone.
Left-right position (cm)
0 2 4 6 8 10 12 14 16
Left-
right
vel
ocity
(cm
/s)
0
2
4
6
8
10
12
14
0 2 4 6 8 10 12 14 16
0
2
4
6
8
10
12
14
Left-right position (cm)
0 2 4 6 8 10 12 14 16
0
2
4
6
8
10
12
14
Left-
right
vel
ocity
(cm
/s)
Left-right position (cm)0 2 4 6 8 10 12 14 16
0
2
4
6
8
10
12
14
Left-right position (cm)
0 2 4 6 8 10 12 14 16
0
2
4
6
8
10
12
14
0 2 4 6 8 10 12 14 16
0
2
4
6
8
10
12
14
Left-
right
vel
ocity
(cm
/s)
Left-right position (cm) Left-right position (cm)
''''''''''===@==='''''''''' ''''''''''@@@=@@@''''''''''
''''''''''YYY.YYY'''''''''' ''''''''''...Y...''''''''''
''''''''''XXX[XXX''''''''''
''''''''''[[[X[[[''''''''''
6:1 1:6
5:2
The movements made across the target zone are highly intermittent in velocity, even if more stable on average
Sample traces self-selected velocities
D’Silva, Baes & Hughes, 2016
Numerosity Difference
1 3 5
Perc
eptu
al S
ensi
tivity
(mea
n d'
)
0
1
2
3
4
5Target > Distractors Target < Distractors
1 3 5 1 3 5
1: Self-paced 2: Self-paced mean*2.0 3: Self-paced mean*0.5
1 3 50
1
2
3
4
5
1 3 5 1 3 5
1: Self-paced 2: Self-paced 3: Self-paced
Novices are:
not equally sensitive to all numerosity changes: one dot differences are not easily detected
sensitive to the direction of the numerosity difference: a target with fewer dots than the distractors is slightly more easily detectable than one with more
Mean Scanning Velocity (cm/s)
0 2 4 6 8 10 12 14 16
No.
Acc
eler
atio
n Ze
ro-C
ross
ings
0
50
100
150
200
250
300
350
0 2 4 6 8 10 12 14 16 0 2 4 6 8 10 12 14 16
Self-selected velocity Self-selected velocity * 2.0 Self-selected velocity * 0.5
0 2 4 6 8 10 12 14 160
50
100
150
200
250
300
350
No.
Acc
eler
atio
n Ze
ro-C
ross
ings
0 2 4 6 8 10 12 14 16
Mean Scanning Velocity (cm/s)
0 2 4 6 8 10 12 14 16
Self-selected velocity Self-selected velocity Self-selected velocity
y = 261.3190* exp(-0.3476*x)RSQ = 0.7184
Even when the mean velocity of the scans changes, the underlying relationship to intermittency remains
D’Silva, Baes & Hughes, 2016
The benefit of adding an additional reading finger is negligible.
D’Silva, Baes & Hughes, 2016
Conclusions
• The smoothness of braille reading is illusory.
• Braille reading speeds are slow (3-7 cm/s) but are not of constant velocity. Readers of all degrees of fluency show the same thing.
• This intermittent velocity emerges in the reading many types of sentences, including those with no meaning at all. And when making reversals.
• Braille readers are intermittent in a way print readers are not. There are no saccades and fixations in braille.
Open questions
Is an intermittent velocity trace relevant to the perception of braille patterns?
If so, how is the changing speed of the finger ‘taken into account’ when reading?
Does intermittency contribute to letter confusions in braille readers?
If faster movements are smoother movements, should learners move faster?
How and when does increasing the number of reading fingers benefit the reader?