ON THE ROLE OF TRANSITIONS IN SLOR:

WHAT’S WRONG WITH THE SIGN?

Tommi Jantunen, University of Jyväskylä, Finland

Theoretical Issues in Sign Language Research (TISLR10), Purdue University (IN, USA), September 30 – October 2, 2010

1

Outline of the talk

• Introduction

• Signs, transitions & the characteristics of transitions

• Experiments with transitions

(i) Distribution of acceleration peaks in continuous signing

(ii) Comprehension of signing without signs

• Implications for the notion of sign

• Conclusion

2

The sign - temporal perspective

• Segmental phonology

A sequence of dynamic and static segments, prototypically containing at least one dynamic segment during which the hand produces a path movement (e.g. Liddell & Johnson 1989; Sandler 1989; Perlmutter 1992)

• Prosodic phonology

Prototypically one syllable, understood generally as one dynamic phonological movement unit (e.g. Perlmutter 1992; Sandler 1993; Brentari 1998; Jantunen & Takkinen 2010)

• Corpus linguistics

Prototypically a series of video frames that identifies with a semantically coherent sequence of sign stream during which the hand(s) move from the initial location of the sign to the final location of the sign, both marked by a change in the direction of the movement (e.g. Crasborn & Zwitserlood 2008; Johnston 2009; Mesch 2010)

3

Sign identification method

• A sign begins...

on the video frame that immediately precedes the frame in which the dominant hand first shows movement away from the initial location of the sign; if the sign includes only a local movement, the beginning of a sign corresponds to the frame that immediately precedes the frame in which the initial handshape or orientation of the dominant hand first starts to change

• A sign ends...

at the frame immediately following the frame in which the path movement of the dominant hand has reached its end or in which the dominant hand still holds a posture or a hand configuration of the sign

4

Transitions

• Two types of transitions (see e.g. Blondel & Miller 2001: 41-43):

• Sign external transitions, i.e., movements of the hand(s) that occur inbetween signs and transfer the hand(s) from the end location of one sign to the start location of the next sign

• Sign internal transitions, i.e., movements of the hand(s) that occur inside repeated signs and transfer the hand(s) from the end location of the first part of the sign back to the start location of the second part of the sign

• Standard theory treats transitions as nonlinguistic, unintentional, meaningless, automatic, nonsalient, unmodifiable, holistic, etc. (e.g. Wilbur 1990a, Perlmutter 1990, Wilcox 1992, van der Hulst 1993)

5

Research into transitions & key findings

• Phonetics

Acceleration peaks, the maximally perceivable and salient moments in the sign stream (e.g. Shaw & Cutting 1980, Wilbur 1990b, Luck & Sloboda 2008, Wolfe & al. 2008), can associate with transitions - typically with their beginnings and/or ends - making transitions the most salient moments in the sign stream (e.g. Wilbur 1990ab, Wilcox 1992, Jantunen & al. 2010)

• Poetry

Transitions are modifiable, also linguistically, e.g. for the purpose of expressing emphatic stress (e.g. Blondel & Miller 2001, Sutton-Spence & Kaneko 2007; see also Wilbur 1990a)

• Sign recognition

The lexical recognition point of signs is located within the pre-sign transition (e.g. Grosjean 1981, Clark & Grosjean 1982, Emmorey & Corina 1990, Arendsen, van Doorn & de Ridder 2007, ten Holt & al. 2009)

6

Experiment 1 - Introduction

• The aim: to demonstrate the distribution of acceleration peaks in continuous signing with a new software tool developed in cooperation with the Aalto University and JyU for the purpose of visualizing and analyzing signed language motion from a digital video (Jantunen & al. 2010; see also Koskela & al. 2008)

• The basis of the tool is quantitative computer vision analysis (Tomasi & Kanade 1991)

• The tool includes four phases (see the next slide)

• Data: semi-pre-rehearsed FinSL monologue/interview; native Deaf signer; 25fps

7

SL Motion Analysis Tool phases

Text1: face detection 2: skin color filtering 3: body part segmentation

6: active shape model5: point distribution model4: feature detection and tracking(data from Jantunen & al. 2010)

8

DemonstrationThe distribution of acceleration peaks (dom. hand only)

9

DemonstrationThe distribution of acceleration peaks (dom. hand only)

10

Experiment 2 - Introduction

• The aim of the experiment: to test the role the linear sign (as currently understood) has in the comprehension of linguistic meaning in continuous signing

• Question: Can continuous signing be understood only on the basis of transitions?

• Method: a video test in which testees were shown a series of short superficially signing-like video clips, edited to contain only transitions

• Participants & data: eight (8) native FinSL signers; five (5) randomly selected video clips from Suvi, the Online dictionary of FinSL

• The task: to tell whether a clip is understandable or not; if judged understandable, testees were asked to repeat the signs in the clip

11

How the ‘signless’ video clips were created

Transition

Sign1.Sign frames were identified from the video (see demo 1 for the identification method)

2.Sign frames were deleted from the video

3.Transition frames were edited into a single sequence

12

Results

a b c d e

1

2

3

4

5

6

7

8

Clip

Signer Understood

13

Discussion -Implications for the sign

From the perspective of research into transitions:

1. What are the defining features of the linguistic symbol?

2. What’s wrong with the sign?

3. How should we re-think the sign?

14

...includesthe maximally

salient part(s) of the signal

Linguistic symbol

form and

meaning

...is a combination of

&

15

Words & salience

Time (s)

1.844 2.361-0.107

0.1172

0

...[sit.te]...

Intensity curve (dB) Pitch (Hz)

The peaks of salience in spoken signal are always word-internal (and syllable-internal)

16

Words & meaningMeaning always attaches to the form of the word

ja sitte ‘and then’

ja ‘and’

(ja)’...’(sitte)

17

Signs as linguistic symbols?

‘depends’

...DEPENDS...

‘children’

...CHILDREN...

18

My argument

• The prevailing conception of the sign is misguided

• Signs are ontologically longer and fuzzier units than the prevailing view acknowledges

• The ontological re-segmentation and re-definition of the linear sign makes it possible (i) to count the “transitional peaks of salience” as internal parts of signs and (ii) to say that the (lexical) meaning is conveyed through linguistic symbols also in signed language

19

Towards the ontologically revised sign

Form

Meaning

Current Revised

‘black’ ‘black’

20

Conclusion

• The data from research into transitions (in continuous signing) suggests that the prevailing ontology concerning the sign and transitions is misguided

• The sign needs ontological revision into a longer and fuzzier unit

• The consequences, e.g., for SL phonology (how the revised sign, and especially the movement component, should be represented) and corpus work (how signs should be identified from the video) need to be further investigated

21

References (1)• Arendsen, J., van Doorn, A. & de Ridder, H. (2007). When and how well do people see the onset of gestures? Gesture

7:305-342.

• Blondel, M. & Miller, C. (2001). Movement and rhythm in nursery rhymes in LSF. Sign Language Studies 2:24-61.

• Brentari, D. (1998). A prosodic model of sign language phonology. Cambridge, MA: MIT Press.

• Clark, L. & Grosjean, F. (1982). Sign recognition processes in American Sign Language: the effect of context. Language and Speech 25:325-340.

• Crasborn, O. & Zwitserlood, I. (2008). Annotation of the video data in the "Corpus NGT". Dept. of Linguistics & Centre for Language Studies, Radboud University Nijmegen, The Netherlands. Online publ. http://hdl.handle.net/1839/00-0000-0000-000A-3F63-4

• Emmorey, K. & Corina, D. (1990). Lexical recognition in sign language - effects of phonetic structure and morphology. Perceptual and Motor Skills 71:1227-1252.

• Green, K. (1984). Sign boundaries in American Sign Language. Sign Language Studies 42:65-91.

• Grosjean, F. (1981). Sign & word recognition: a first comparison. Sign Language Studies 32:98-125.

• ten Holt, G., van Doorn, A., de Ridder, H., Reinders, M. & Hendriks, E. (2009). Which fragments of a sign enable its recognition? Sign Language Studies 9:211-239.

• van der Hulst, H. (1993). Units in the analysis of signs. Phonology 10:109-241.

• Jantunen, T., Koskela, M., Laaksonen, J. & Rainò, P. (2010). Towards the automated visualization and analysis of signed language motion: method and linguistic issues. [In proceedings of the 5th International Conference on] Speech Prosody 2010 100006:1-4.

• Jantunen, T. & Takkinen, R. (2010). Syllable structure in sign language phonology. In D. Brentari (Ed.), Sign Languages, pp. 312-331. Cambridge Language Surveys. Cambridge: Cambridge University Press.

• Johnston, T. (2009). Guidelines for annotation of the video data in the Auslan Corpus. Dept. of Linguistics, Macquarie University, Sydney, Australia. Online publ. http://media.auslan.org.au/media/upload/attachments/Annotation_Guidelines_Auslan_CorpusT5.pdf

• Koskela, M., Laaksonen, J., Jantunen & al. (2008). Content-based video analysis and access for Finnish Sign Language - a multidisciplinary research project. In O. Crasborn, E. Efthimiou, T. Hanke, E. D. Thoutenhoofd & I. Zwitserlood (Eds.), Construction and exploitation of sign language corpora, pp. 101-104. Paris: ELRA.

22

References (2)• Liddell, S. & Johnson, R. (1989). American Sign Language: the phonological base. Sign Language Studies 64:195–277.

• Luck, G. & Sloboda, J. (2008). Exploring the spatio-temporal properties of simple conducting gestures using a synchronization task. Music Perception 25:225-239.

• Mesch, J. (2010). Viittomien glossit ja ajalliset pituudet: annotointityöskentelyyn liittyviä kysymyksiä [Glosses and temporal durations of signs: questions relating to sign language annotation]. In T. Jantunen (Ed.), Näkökulmia viittomaan ja viittomistoon [Perspectives on sign and lexicon], pp. 42-54. Theory and practice in applied linguistics 5. Jyväskylä: University of Jyväskylä.

• Perlmutter, D. (1990). On the segmental representation of transitional and bidirectional movements in ASL phonology. In S. Fischer & P. Siple (Eds.), Theoretical issues in sign language research. Volume 1: linguistics, pp. 67-80. Chicago: The University of Chicago Press.

• Perlmutter, D. (1992). Sonority and syllable structure in American Sign Language. Linguistic Inquiry 23:407–442.

• Sandler, W. (1989). Phonological representation of the sign: linearity and nonlinearity in ASL. Dordrecht: Foris.

• Sandler, W. (1993). A sonority cycle in American Sign Language. Phonology 10:243–279.

• Shaw, R. & Cutting, J. (1980). Clues from an ecological theory of event perception. In U. Bellugi & M. Studdert-Kennedy (Eds.), Signed and spoken languages: biological constraints on linguistic form, pp. 57-84. Weinheim: Verlag Chemie.

• Sutton-Spence, R. & Kaneko, M. (2007). Symmetry in sign language poetry. Sign Language Studies 7:284-318.

• Suvi = Suvi — Suomalaisen viittomakielen verkkosanakirja [The online dictionary of FinSL]. [Helsinki:] Kuurojen Liitto ry, 2003. Online publication: http://suvi.viittomat.net.

• Tomasi, C. & Kanade, T. (1991). Detection and tracking of point features, Carnegio-Mellon University Tech. Rep. CMU-CS-91-132, Apr. 1991.

• Wilbur, R. (1990a). An experimental investigation of stressed sign production. International Journal of Sign Language 1:41-59.

• Wilbur, R. (1990b). Why syllables? What the notion means for ASL research. In S. Fischer & P. Siple (Eds.), Theoretical issues in sign language research. Volume 1: linguistics, pp. 81-108. Chicago: The University of Chicago Press.

• Wilcox, S. (1992). The phonetics of fingerspelling. Studies in speech pathology and clinical linguistics 4. Amsterdam: John Benjamins.

• Wolfe, J., Kluender, K., Levi, D. & al. (2008). Sensation & perception. Second edition. Sunherland, MA: Sinauer Associates Inc.

23

ACKNOWLEDGEMENTS

Aalto University School of Science and Technology (Department of Information and Computer Science, CoE in Adaptive Informatics Research): Matti Karppa, Jorma Laaksonen, Markus Koskela

Finnish Association of the Deaf: Päivi Rainò, Päivi Mäntylä

University of Jyväskylä: All colleagues & students, especially Tuija Wainio, Elina Tapio, Eleanor Underwood, Danny De Weerdt

The financial support of the Academy of Finland is gratefully acknowledged

24