new datacodingof,andtransitiontimein, asl fingerspellingpubs.jonkeane.com/pdfs/keane2011ac.pdf ·...
TRANSCRIPT
-
Introduction Methods and Coding Results Conclusions References
DATACODINGOF, AND TRANSITION TIME IN,ASL FINGERSPELLING
Jonathan Keane
University of Chicago
-
Introduction Methods and Coding Results Conclusions References
OutlineIntroductionMethods and Coding
Coding PrinciplesData collectionOur coding methodCoding Principles againTransitions
ResultsSpeedSignerWord typePositionIndividual Letters
Transitions againConclusions
-
Introduction Methods and Coding Results Conclusions References
Background – Fingerspelling
All handshapes are static except for -- and --.
Fingerspelling makes up anywhere from– of discourse.(Padden, ; Padden and Gunsauls, )
zj
-
Introduction Methods and Coding Results Conclusions References
The phonetics of Fingerspelling
Wilcox () looks at about words and describes some of thedynamics of hand motion.
Tyrone et al. () looks at fingerspelling by parkinsoniansigners from a phonetic perspective.
Brentari and Padden (); Cormier et al. () both look atthe nativization process for fingerspelled words.
Quinto-Pozos () described the rate of fingerspelling for twosigners within fluent discourse.
-
Introduction Methods and Coding Results Conclusions References
Coding Principles
What data looks like
data.mpg
-
Introduction Methods and Coding Results Conclusions References
Coding Principles
Data coding needs to be
Accurate
Accurate, detailed data is necessary for any linguistic analysis.
Reproducible
Coding should be able to be reproduced, and individual codersshould form some sort of consensus.
Quick
Coding time is oen directly related to the amount of data we cancollect.
Easy
A coding system that requires little specialized training is betterthan one that requires experts to use. (All else being equal)
-
Introduction Methods and Coding Results Conclusions References
Data collection
Recording specifications
Signers
signers, both are deaf of deaf parents, and native users.
Video
video cameras recording at .
We collected sessions for each signer at a normal, conversational speed, and at a careful speed.
�ere were non-English words, names, nouns.
Each word was fingerspelled twice in each speed.
�e video was then post processed and compressed for coding.
-
Introduction Methods and Coding Results Conclusions References
Data collection
Session details
Careful elicitation and data collection allowed us to maximize thedata we started with.
. Output a logfile with the order of the words as they werepresented to the signer (although this could be improved further)
. Segmentation – green button. First pass error detection – red button
-
Introduction Methods and Coding Results Conclusions References
Our coding method
3–4 humans hand coded apogees
Using ELAN, coders watched the videos at – speed.Told to press a button whenever they thought there was an apogee.
Described as the point where the hand was maximally orminimally open.
Could be described as the minimum instantaneous velocity of allof the articulators.
Use discretion when coding apogees with movement, but beconsistent.
Not defined as the canonical form
-
Introduction Methods and Coding Results Conclusions References
Our coding method
The position of each apogee wasalgorithmically determined.
Minimized the mean absolute distance between the points for eachword.
We accounted for errant, and missing presses by assigning aviolation cost for every apogee that was deleted or added.
�e coders were already fairly close together.Mean absolute deviation:. msec for all letters. msec for letters with movement
-
Introduction Methods and Coding Results Conclusions References
Our coding method
Leveraging known data
A first guess at the letter of each apogee was added using le edgeforced alignment.
Although the letters it assigns are not accurate, they are close.
Finally someone trained in fingerspelling went through and verifiedthe location, and letter of each apogee. �e vast majority of apogeesare unchanged.
-
Introduction Methods and Coding Results Conclusions References
Our coding method
Example
-
Introduction Methods and Coding Results Conclusions References
Our coding method
Verification apogees of in normal are unchanged. (∼ %)
−200 −100 0 100 200
0.00
0.02
0.04
0.06
0.08
0.10
msec
Den
sity
-
Introduction Methods and Coding Results Conclusions References
Our coding method
VerificationOf the changed apogees, they are oen shied back by – frames.
−200 −100 0 100 200
0.00
00.
004
0.00
8
msec
Den
sity
-
Introduction Methods and Coding Results Conclusions References
Coding Principles again
How does our method look?Accurate
Much less chance for transcription or other errors compared totraditional methods.
Reproducible�e first stage of coding is incredibly reproducible, to a highdegree of accuracy. �e second step might be (we’re working ontesting this)
QuickVaried, but each clip chunk (– min of data) took min for coders, min of which can be done simultaneously by coders,so min at the fastest.
EasyOur initial pass for coding can be done with very little training.�e verification task requires a bit of training in fingerspelling,but since it’s more confirmation, it’s much easier and quicker thantraditional methods.
-
Introduction Methods and Coding Results Conclusions References
Transitions
Example
staxi.mp4
Figure: ---, normal speed.
-
Introduction Methods and Coding Results Conclusions References
Transitions
Example
staxi2.mp4
Figure: ---, normal speed, slow motion
-
Introduction Methods and Coding Results Conclusions References
Transitions
Example, revisited
ms ms ms| | | |
Figure: durations for ---
-
Introduction Methods and Coding Results Conclusions References
Transitions
---, ---, ---, ---, and --- ms ms ms
| | | |
ms ms ms| | | |
ms ms ms| | | |
ms ms ms| | | |
ms ms ms| | | |
Figure: durations for ---, ---, ---, ---, and ---(signer: s speed: normal)
-
Introduction Methods and Coding Results Conclusions References
Questions
. When asked to sign at two different speeds, how much of adifference is there between them?
. Is there individual variation?. Does the type of word affect the speed of fingerspelling?. Does the position of a letter in a word affect transition time?. Do letters with movement take longer to execute?. Does articulatory complexity change transition time?
-
Introduction Methods and Coding Results Conclusions References
table
Df Sum Sq Mean Sq F value Pr(>F)wordtype . . . .speed . . . .signer . . . .wordtype:speed . . . .wordtype:signer . . . .speed:signer . . . .Residuals . .
Table: table for log(time)
-
Introduction Methods and Coding Results Conclusions References
Speed
between speeds
speed
time
(mse
c)
200
400
600
800
careful normal
+
++
++
+
+
+
+
+
+
+
+
++
++
+
++
+
+
+
+
+
+++
+
++
+++
+
+
+
+
+
+++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
++++
+
++++
+
++
+
++
+++
++
++
+
+
+
++
++
+
+
+
++
+
+
+
+
+
+
+
+
+
++
++
+
+
+++
+
++
+
+++
+
+
+
+
++
+
+
+
+
+++
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
++
+++++
+
+
+
+
+
+
++
+
++
++
+
+
+
+
+++
+
++
++
+
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+++
+
+++
++
+
+
+
++
+++
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
++
+
+
+
+++
+
+
+
+
+
++
+
+
+
+
++
+++++
+
+
++
+
+
+
+
+
+
+
+
+
++++
+
+++
+
+
+
+
++++
+
+
+
++
+
+
+
+
+
+++
+
+
+
+
+++++
++
+
+
+
+
+
+
++
+
-
Introduction Methods and Coding Results Conclusions References
Speed
table
Df Sum Sq Mean Sq F value Pr(>F)wordtype . . . .speed . . . .signer . . . .wordtype:speed . . . .wordtype:signer . . . .speed:signer . . . .Residuals . .
Table: table for log(time)
-
Introduction Methods and Coding Results Conclusions References
Signer
between signers, by speed
signer
time
(mse
c)
200
400
600
800
s1 s2
++
+
+
+
+
+
+++++++
+
+++
+
+
++
+++
++++
+
+
+
+
+
+
+
+
+
+++
+
+
+++
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
++
+
++
+
++
+
+
+
+++
+
++
++
+
+
+
+
++
+
+
++
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
++
++
+
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+++
+
+++++
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
++
+
+
+
+
+
+
++
careful
s1 s2
+
++
+
++
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
++++++
+
+
+
+
+
+
+
+
+
+++
++
+
++
+
+
++
+
++
+
+
+
+
++
+
+
+
+
+
+
+
++
++
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++++
+
+
+
++++
+
++++
+
++
+
+
+
+
+
+
+
+
+
++
+
+
++
+
++
+
+
+
+
+
+
+
+
++
++++
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+++++
+
+++
++
++
++
++
+
+
+
+
+
+
++
+
+
+
+
+
+
+
+
++++
+
+
++
+
+
++
+
+
++
+
+
++
+
+
+
+
++
+
+++
+
++++
+
+
+
+
++
+
+
+
++++
+
+
+
+
++
normal
-
Introduction Methods and Coding Results Conclusions References
Signer
table
Df Sum Sq Mean Sq F value Pr(>F)wordtype . . . .speed . . . .signer . . . .wordtype:speed . . . .wordtype:signer . . . .speed:signer . . . .Residuals . .
Table: table for log(time)
-
Introduction Methods and Coding Results Conclusions References
Word type
between wordtypes, by speed
wordtype
time
(mse
c)
200
400
600
800
foreign name noun
+++
++
++
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+++++
+
++
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
+
+
++
+
+
+
+
++
++
+
++
+
+
+
+
+
+
+
+
+
++
+
++
++
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
++
+
++++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
++
+
+
+
+
+
+
+
+
++
+
+
+
+
++
++
+
+
+
+
+
+
+
+++
+
++
+
++
+
+++
++
+
+
+
+
+++
+
+
+
+
++
+
+
+
+
+
+
careful
foreign name noun
+
++
+
+
+
++
+
+
+
+
+
+
++
++
+
+
+
+++
+
+++++++++
+
+
+
+
+
+
+
+
+
+
+
++++
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++++++++
+++
+
+
+
++
+
+
++
+
+
+
+
+
+
+
+
+
++
+
+
++
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+++
++
+
+++
+
+
+
+
+
+
+++
+
+++
+
++++++
+
+
+
++
++
+
++
+
+
+
+
++
+
+
+
+++
+
+
+
++
+
+
+
+
+
+
normal
-
Introduction Methods and Coding Results Conclusions References
Word type
table
Df Sum Sq Mean Sq F value Pr(>F)wordtype . . . .speed . . . .signer . . . .wordtype:speed . . . .wordtype:signer . . . .speed:signer . . . .Residuals . .
Table: table for log(time)
-
Introduction Methods and Coding Results Conclusions References
Position
Short words (3 – 6 le�ers) – s1, normal
position
time
(mse
c)
50
100
150
200
250
300
1 2 3 4 5
+++
+
+
+
+
+
+++
+
+
+
+
+
+
+
-
Introduction Methods and Coding Results Conclusions References
Position
Long words (8 – 10 le�ers) – s1, normal
position
time
(mse
c)
100
150
200
250
300
1 2 3 4 5 6 7 8 9 10
+
+++
+
+
+
+
+++
+ +
+
+++
+
++
++
++
+
+
+
+
+
+
+
+
+
-
Introduction Methods and Coding Results Conclusions References
Position
Possible explanations
. memory limitations. articulation limitations. phonological chunking
letters ∼= movements ∼= sign
-
Introduction Methods and Coding Results Conclusions References
Position
careful is different
position
time
(mse
c)
300
400
500
600
700
800
1 2 3 4 5 6 7 8 9 10 11 12
+++
+++
+
+
+
+
+
+
+
++
+
+
+
++
+
+
++
++
+
+
+
+
++
+
+++
++
+
+
+
+
+
++
+
+
+
+
-
Introduction Methods and Coding Results Conclusions References
Position
non-english is different
position
time
(mse
c)
100
150
200
250
300
1 2 3 4 5 6 7 8 9 10
+
+
+
++
++
+
+
+++
++
++
+
-
Introduction Methods and Coding Results Conclusions References
Individual Letters
---, ---, ---, ---, and --- ms ms ms
| | | |
ms ms ms| | | |
ms ms ms| | | |
ms ms ms| | | |
ms ms ms| | | |
Figure: durations for ---, ---, ---, ---, and ---(signer: s speed: normal)
-
Introduction Methods and Coding Results Conclusions References
Individual Letters
---, ---, ---, ---, and --- ms ms mst a x i
ms ms msl a m b
ms ms msf r e d
ms ms msc a r p
ms ms msp u h u
Figure: durations for ---, ---, ---, ---, and ---(signer: s speed: normal)
-
Introduction Methods and Coding Results Conclusions References
Individual Letters
transitions for high frequency le�ers
time
(mse
c)
−400
−200
0
200
400
i o a t e
+
+
+
+
+
+
+
+
+
+
+
++
+
+
++
+
+
+
+
+
+
++
++
+
++
+
++
+
+
+
++
+
+++
+
+
++
++
+
+
+
+
+
++
++
+
+
++++
299 212 369 146 360
+
+
+
+
+++++
+
+
+
+
++++
+
+
+
+
+
+ +
++
++
+
+
+
++
+
+
+
+
+
++
+
+
++++
+
+
+
++
+
+++++++
273 197 340 144 299
-
Introduction Methods and Coding Results Conclusions References
Individual Letters
transitions for for low frequency le�ers
time
(mse
c)
−400
−200
0
200
400
z j q x y++
++
++
+
+
43 26 21 43 80
+
++
+
+
+59 46 42 52 42
-
Introduction Methods and Coding Results Conclusions References
Individual Letters
Movement in --
sNormal.mp4
Figure: �e first instances of -- – not at the end of the word
-
Introduction Methods and Coding Results Conclusions References
Individual Letters
82 bigrams with more than 50 instances
letter
time
(mse
c)
100
200
300
400
ac ad al aman ar as at ax be ca ce co de ea ee eg ek el en er es ex fi ge ho ia ic ie il in is it ji ka ke ko la le li mamemi na nd ne ng ni no nt ol on or osowpe po pr qu ra re ri sa se si ss st ta te ti to ua ue ui ur us va ve vi wi xe ze
+
+ +
+
+
+
+
+
+
+
+
++
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+++
+
+
+
++ +
+
++
+
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
32 36 52 4416560 36 48 25 36 56 33 31 36 28 21 28 24 8510510135 36 32 36 27 56 39 35 57 78 41 48 27 25 28 28 66 57 60 27 28 44 32 28 48 36 56 31 35 28 64 29 37 27 28 36 32 83 70 57 37 35 35 47 22 60 72 52 43 36 33 32 28 26 27 36 27 32 32 26 31
-
Introduction Methods and Coding Results Conclusions References
Individual Letters
Transition time by articulatory complexity
complexity score
mea
n tr
ansi
tion
time
200
250
300
0 100 200 300 400 500
-
Introduction Methods and Coding Results Conclusions References
Conclusions
. When asked to fingerspell at different speeds, the spread issignificant.
. �ere is individual variation overall and in speed, but notwordtype.
. Signers fingerspell slower on non-English words.. Signers seem to chunk their production into - letter chunks
with longer words.. Letters with movement take longer to execute.. �e class of letters that have movement might need redefining:
-- and possibly --.. Transition time does not seem to correlate with articulatory
complexity.
-
Introduction Methods and Coding Results Conclusions References
Future directions
. More sophisticated modeling. Quantification of other articulatory features. Recognition related tasks. More signers (in progress!)
-
Introduction Methods and Coding Results Conclusions References
Thank you for coming.
I must also acknowledge the contributions of many whocontributed in ways big and small:
Fingerspelling dataDrucilla Ronchen and Andy Gabel
Main advisorsJason Riggle and Diane Brentari
Other researchersSusan Rizzo, Karen Livescu, Greg Shakhnarovich, RaquelUrtasun, Erin Dhalgren, and Katie Henry.
-
Introduction Methods and Coding Results Conclusions References
Brentari, Diane, and C. Padden. . Foreign vocabulary in sign languages:A cross-linguistic investigation of word formation, chap. Native and foreignvocabulary in American Sign Language: A lexicon with multiple origins,–. Mahwah, NJ: Lawrence Erlbaum.
Cormier, Kearsy, Adam Schembri, and Martha E. Tyrone. . One hand ortwo? nativisation of fingerspelling in asl and banzsl. Sign Language &Linguistics .–.
Padden, Carol. . �eoretical issues in sign language research, chap.�eAcquisition of Fingerspelling by Deaf Children, –. �e University ofChicago press.
Padden, Carol, and Darline Clark Gunsauls. . How the alphabet came tobe used in a sign language. Sign Language Studies .–.
Quinto-Pozos, David. . Rates of fingerspelling in american signlanguage. Poster at .
Tyrone, M.E., J. Kegl, and H. Poizner. . Interarticulator co-ordination indeaf signers with parkinson’s disease. Neuropsychologia .–.
Wilcox, Sherman. . �e phonetics of fingerspelling. John BenjaminsPublishing Company.
-
Introduction Methods and Coding Results Conclusions References
between wordtypes, by speed and by signer
wordtype
time
(mse
c)
200
400
600
800
foreign name noun
++
+
+++
+
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+
++
+
++
+
++++
++
+
+
++
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+++
+
+
+
+
++
+
+
+
carefuls1
foreign name noun
++
+
+
+
+++
+
+
+
+
+
++
+
++
+
+
++
+
+++++
+
++
+
+
+
+
+++
++
+
+
++
+
+
++
+
+
+
+
++
+
++
+
+
++
+
+
+
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+++++++
+
+
+
+
+
+
++
+
+
+
++++
+
++
+
+
+
+
+
+
normals1
foreign name noun
++
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
++
+
+
++
+
+
+
+
+
+
+
+
+
+
+++++
+
+
+
+
+
+
+
+
+
+++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
+
+
++
+
+
+
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+
+++
++
+
++
+
+
+
+
++
carefuls2
foreign name noun
+
+
+
+
++
+
+
+
+
++
+
+
+
+
+++++
+
+
+
+
+
+
+
++++
+
+
++
+
+
+
+
+
+
+++++
+++
+
+
+
++
++
+
+
+
+
+
+
++
+
++
+
+
+++
+
+
+
+
+++++
++
+
++
+
+
+
+
+
++
+
+
+
++
+
+++++
+
+
+
++
++
+
+
+
+
++
+
++++
++
+
+
+
normals2