shari r. speer ohio state university
DESCRIPTION
Intonationally marked contrast in instructed visual search: Intersective color vs. subsective scalar adjectives. Shari R. Speer Ohio State University. Acknowledgements. Co-I, Kiwako Ito. NSF BCS-018464 NSF BCS-0617609 NIDCD R01-DC007090 Ping Bai Allison Blodgett Laurie Maynell - PowerPoint PPT PresentationTRANSCRIPT
Intonationally marked contrast in instructed visual search: Intersective
color vs. subsective scalar adjectives
Shari R. Speer
Ohio State University
Acknowledgements
NSF BCS-018464
NSF BCS-0617609NIDCD R01-
DC007090
Ping Bai
Allison Blodgett
Laurie Maynell
Mary Beckman
Co-I, Kiwako Ito
Contrast Similarities and differences comparisons
Bolinger (1961) “… cases where one or more individual items are
singled out from a larger (but limited) set as being true as regards some relationship whereas others in the same set are untrue…”
Zeevat (2004)Contrastors (alternatives)… “must be obtainable from the actual utterance by substituting something else for the intonationally prominent constituent.”
Contrast
Discourse coherence
prerequisite for anaphora resolution Must be in focus domainOften accompanied by structural
parallelism
Matter of degree?
Accentuation for expressing contrast
B (vs. A) accent (Bolinger, 1961; Jackendoff, 1972)
L+H* (vs. H*) (Pierrehumbert, 1980; ToBI)
Pierrehumbert & Hirschberg (1990)
“the accented item - and not some alternative related item - should be mutually believed (p 296.)”
L+H*: +AGREED, theme accent (Steedman, 2003)
Intonationally marked contrast in sentence/discourse processing
Faster comprehension of short contrastive discourse when prominent accents were appropriately located (vs. inappropriate) (Bock & Mazella, 1983)
Faster comprehension and higher acceptance of appropriately accented Q-A pairs (English pitch accent: Birch & Clifton, 1995; Japanese, Basque pitch range expansion/peak alignment: Ito, 2002).
Faster phoneme monitoring when the contrastive entity was negated with prominent accent (vs. no accent). Intonationally marked narrow focus in negation generates contrast set (Davidson, 2001).
Accentuation shows immediate effects on instructed visual search(Dahan, Tanenhaus & Chambers, 2002)
Prominent accent interpreted non-anaphorically, lack of prominent accent interpreted anaphorically.
1. “Put the candle/candy below the triangle.”
2. “NOW, put the CANDLE above the square.” “NOW, put the candle ABOVE THE SQUARE.”
fixation to fixation to
candycandyfixation to fixation to candlecandle
candle candle “ “CANDLECANDLE” ” candle candle “ “candycandy” ”
L+H* on a modifier evokes a contrast set(Ito & Speer, 2008)
Manipulated pitch accent type on a modifier adjective or modified noun
L+H*: Evokes a contrast set that contains alternative related item(s) (Pierrehumbert & Hirschberg, 1990)
GREENGREEN ballballGREENGREEN ballballblue ballblue ball
Holiday Tree task
(Ito & Speer, 2008)
Eyemovements monitored as naïve participants followed recorded audio instruction to decorate Christmas trees.
(ASL e5000) at 60Hz.
Visual Stimuli (Ito & Speer, 2008)
44 ornaments displayed by type on a grid with 11 cells (8 target + 3 filler)
Sound Stimuli (Ito & Speer, 2008)
Accentual patterns of stimuli were selected based on the analysis of spontaneous speech in a similar task (Ito, Speer & Beckman, 2003).
A trained female phonetician produced stimuli with the intended pitch accent patterns. Adjective-noun pairs were presented in discourse context.
Recorded 44.1KHz, 16bits All stimuli were ToBI-annotated by two native speakers
of American English. Items on which they did not agree were re-recorded.
Experiment 1: Stimuli
Felicitous L+H* Patternsgreen onion ORANGE onion
L+H* no accent brown ball brown ANGEL
H* L+H* Infelicitous L+H* Patterns
gray stocking brown STOCKING H* L+H*
orange candy ORANGE onion L+H* no accent
Experiment 1 Results: Felicitous vs. Infelicitous L+H*
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
-600 -300 0 300 600 900 1200 1500
Time from noun onset (ms)
Fixation proportion to the target object cell
Felicitous Adj Contrast BLUE drum
Infelicitous Adj Contrast blue DRUM
Felicitous Noun Contrast blue DRUM
Infelicitous Noun Contrast BLUE drum
blue
BLUE
DRUM
drum
(Ito & Speer, 2008)(Ito & Speer, 2008)
Experiment 1: Summary
Felicitous L+H* on the color term in contrastive environments (green drum, BLUE drum) facilitated visual search compared to infelicitous L+H* on non-contrastive noun (green drum, blue DRUM).
Felicitous L+H* on the object noun in contrastive environments (blue egg, blue DRUM) showed a smaller advantage over infelicitous L+H* on the adjective (blue egg, BLUE drum).
Listeners ‘tune’ their sensitivity to contrastive accent on the basis of the visual task?
Experiment 2: stimuli
Does L+H* on the contrastive adjective lead to anticipatory eye-movement compared to H*? e.g. brown drum --> RED/red drum
Does infelicitous Does infelicitous L+H*L+H* on the color adjective lead on the color adjective lead to an expectation strong enough to lead a to an expectation strong enough to lead a “garden-path” in eye-movement?“garden-path” in eye-movement?
e.g. red onion --> e.g. red onion --> GREENGREEN drum drum
Experiment 2 Results: Felicitous/Infelicitous L+H* vs. H*
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
-600 -300 0 300 600 900 1200 1500
Time from noun onset (ms)
Fixation proportion to the target object cell
Felicitous L+H* Contrast brown drum...GREEN drum
H*!H* Adj Contrast brown drum...green drum
H*!H* Both Already red onion...green drum
Infelicitous L+H* red onion...GREEN drum
GREEN drum
green drum
Experiment 2 results: “Garden-path” due to L+H*
Experiment 2 results: Refixation of previous cell w/out L+H*?
The effect of infelicitous L+H*
QuickTime™ and aDV/DVCPRO - NTSC decompressor
are needed to see this picture.
Ito & Speer 2008: Summary
For cases with repetition of the immediately preceding noun, L+H* showed a processing advantage as compared to H* on the adjective (brown drum --> RED/red drum)
For cases without repetition of the noun, infelicitous L+H* led to incorrect anticipatory fixations to the most-recently mentioned ornament type.
significant delay in fixations to the real target significant delay in fixations to the real target
NO incorrect anticipatory fixationsNO incorrect anticipatory fixations were observed when were observed when non-repeated non-repeated targets were felicitously presented with [H* !H*] targets were felicitously presented with [H* !H*]
Discussion Dahan et al results showed that L+H* accent led to
the expectation of contrast. Contrastive accent on a (potentially) repeated word led listeners to fixate the object type that had *not* just been mentioned.
Results here show contrast effects from L+H* accent on the modifier that preceded the noun. L+H* led listeners to restrict the set of expected referents to those of the object type that *had* been most recently mentioned.
Discussion
Evidence suggests that L+H* evokes generation of a contrast set in the listener’s discourse representation. The set is based on the accented word interpreted within the structure of the utterance. Here, contrastive accent on the adjective restricted expectations about the coming head noun to the set of objects specified by the immediately preceding noun.
Facilitative anticipatory fixations and prosodic ‘garden-path’ effects due to prominence and have also been shown for German (Weber et al, 2006) and Japanese (Ito et al, under review).
Remaining questions
In Ito&Speer 2008, rapid integration of contrastive L+H* involved refixation of the immediately preceding ornament cell.
Were facilitation effects of L+H* dependent on the visual layout, where contrastive trials required re-fixation of a ‘just-visited’ ornament cell? Was predictive use of L+H* in the ‘garden-pathing’ trials dependent on the organization of the ornament array by objects (nouns)?
Are primary color adjectives, which describe a context- independent distinction between modified objects, more likely to require intonational contrast than other types of adjectives (e.g. scalars)?
Followup studies:
We used adjective-sorted arrays, where contrastive sequences (e.g., red ball green ball, small ball medium ball) require fixations to a new cell (instead of returning fixations).
We compared 2 adjective types: Color (context-independent, intersective) Size (scalar,subsective) Because relative size terms are inherently contrastive and context-dependent, prominent pitch accent may be redundant as a cue to contrast.
Exp3 Visual stimuli - Colors
Exp4 Visual stimuli - Sizes
Sound stimuli
Articles rather than determiners, e.g. ‘Hang a green ball,’ ‘Hang a large ball,’ as there were multiple like ornaments.
Same native English ToBI-trained speaker Same procedures for annotation, presentation
and eyemovement monitoring Four conditions:
Contrastive vs. non-contrastive
H* !H* vs. L+H* no accent
Anticipatory effect w/out ‘refixation’
Exp3, color adjectives:
“Hang a yellow/YELLOW star”
Exp4, scalar adjectives:
“Hang a large/LARGE tree”
Garden-path effects
Color:
fixations to green tree hearing ‘GREEN ball’
Scalar:
fixations to large tree hearing “LARGE ball”
QuickTime™ and aH.264 decompressor
are needed to see this picture.
Summary: Fixation proportions L+H* facilitates eyemovements to the targets for
both color and size adjectives in contrastive sequences.
Visually more complex size-sorted boards led to slower eye movements to the target than color-sorted boards.
L+H* led to more frequent fixations to the incorrect targets for both color and size adjectives in non-contrastive sequences.
Non-prominent size adjective (with H*) did not lead to looks to contrastive cells, i.e., size adjectives are not automatically interpreted contrastively.
Prominent pitch accent (L+H*) on a pre-nominal modifier evokes a contrast set and facilitates reference resolution with both intersective color adjectives and subsective scalar adjectives.
L+H* evokes the anticipation of contrast, and thus may lead to intonation-based “garden-pathing” regardless of the adjective type used.
The interpretation of pre-nominal adjectives depends on their efficacies within each referential environment rather than on their inherent semantics. Intonational prominence is used immediately and predictively for reference resolution regardless of the modifier type.
Summary
Current directions
L+H* vs. H* controversy: Phonetic analyses of stimuli from 4
comprehension experiments underway Correlation of first fixation latency and acoustic
measures Parallel analysis of spontaneous speech from
production version of holiday tree task
Issue on categorical distinction:Is L+H* a kind of H*?
Color adj with H* weaker interpretation of contrast? Categorical interpretation but non-categorical perception
(Ladd & Morton, 1997) Great overlap between H* and L+H* in pitch scale, shape &
alignment (Tayler, 2000) Frequent uncertainty between H* and L+H* in expert-ToBI
labeling (Brugos et al. 2008). More frequent use of H* than L+H* to mention contrastive
discourse entities in story continuation (Metusalem & Ito, 2008).
1. How can we define categories of prosodic prominence?2. What factors contribute to recognition & processing of
contrastiveness?
Step-wise Multiple Linear Regressions:Predicting First fixation latency: L+H*
Step Var t p
1 subject -2.91 <.01
2 F0 diff -2.76 <.01
Total R2: .051
F(2,256) = 7.96, p<.001
Step Var t p
1 F0 diff -5.66 <.001
2 F0 peak 4.44 <.001
3 color -2.57 <.05
4 size 2.53 <.05
Total R2: .245
F(2,296) = 25.38, p<.0001
EXP1: Color-sorted EXP2: Size-sorted
Step-wise Multiple Regressions:Predicting First fixation latency: H*
Step Var t p
1 size -4.78 <.0001
2 color 4.03 <.0001
3 subject -2.97 <.01
Total R2: .108
F(3,257) = 11.59, p<.0001
Step Var t p
1 size -2.44 <.05
2 F0 diff -3.21 <.01
3 color -2.64 <.01
Total R2: .088
F(3,293) = 10.57, p<.0001
EXP1: Color-sorted EXP2: Size-sorted
Further exploration needed:
Different dependent variable?Additional phonetic variables:
Normalized F0 scaling Peak, ‘low elbow’ alignment from vowel onset Word/syllable intensity Vowel quality (F1, F2, breathiness, etc.) Following noun’s phonetic status
F0 prominence intensity vowel quality, etc.