parsing acoustic variability as a mechanism for feature abstraction

Parsing acoustic variability as a mechanism for feature abstraction

Jennifer Cole Bob McMurrayGary Linebaugh Cheyenne MunsonUniversity of Illinois University of Iowa

www.psychology.uiowa.edu/faculty/mcmurray

Phonetic precursors to phonological sound patterns

• Many phonological sound patterns are claimed to have precursors in systematic phonetic variation that arises due to coarticulation

• Assimilation– Vowel harmony from V-to-V coarticulation

(Ohala 1994; Beddor et al. 2001)– Palatalization from V-to-C coarticulation

(Ohala 1994)– Nasal Place assimilation (-mb, -nd, -ŋg) from C-to-

C coarticulation (Browman & Goldstein 1991)

• Assimilation• Epenthesis

– Epenthetic stops from C-C coarticulation: sen[t]se (Ohala 1998)

• Assimilation• Epenthesis• Deletion

– Consonant cluster simplification via deletion from C-C coarticulation: perfec(t) memory

(Browman & Goldstein 1991)

The role of the listener

Phonologization: • when acoustic properties that arise due to

coarticulation are interpreted by the listener as primary phonological properties of the target sound.

• generalization over variable acoustic input that results in a new constraint on sound patterning.

Speaker 3 All vowels

2

3

4

5

6

76.07.08.09.010.011.012.013.014.015.016.0

F2 (Bark)

F1 (B

ark)

iuoe


• From V-to-V coarticulation …

ɛ

ʌ

ɑ

i


2

3

4

5

6

76.07.08.09.010.011.012.013.014.015.016.0

F2 (Bark)

F1 (B

ark)

iuoe


• From V-to-V coarticulation … […ɛi…i…]

ɛ

ʌ

ɑ

i

[…ɛɑ…ɑ…]


2

3

4

5

6

76.07.08.09.010.011.012.013.014.015.016.0

F2 (Bark)

F1 (B

ark)

iuoe


• Perception may yield vowel assimilation […ɛi…i…]

ɛ

ʌ

ɑ

i

[…ɛɑ…ɑ…]i ɑ


2

3

4

5

6

76.07.08.09.010.011.012.013.014.015.016.0

F2 (Bark)

F1 (B

ark)

iuoe


• But – distinct factors can produce similar variants: […ɛi…i…]

ɛ

ʌ

ɑ

i

[…ɛ ŋ…]

From perception to phonology

• What is the mechanism for mapping from continuous perceptual features to phonological categories?

ɛi mid and highcentral and front-peripheral

ɛɑ mid and low central and back

From perception to phonology


ɛi mid and highcentral and front-peripheral

ɛɑ mid and low central and back

The problem: • The perceptual system is confronted with uncertainty due to variation arising from multiple sources.

• Yet, patterns of variation must get associated with individual features of the context vowel (e.g,. high, front) if coarticulation serves as a precursor to phonological assimilation.

• How do lawful, categorical patterns emerge from ambiguous, variable input?

…the lack of invariance problem!

Our claims


Our claims: Variability is retained. Acoustic variability is parsed

into components related to the target segment and the local context.

Feature abstraction through parsing. Acoustic parsing provides a mechanism for the emergence of phonological features from patterned variation in fine phonetic detail.

Variability is retained

• Listeners are sensitive to fine-grained acoustic variation. (Goldinger 2000; Hay 2000; Pierrehumbert 2003)

Variability is retained, not discarded

Consistent with exemplar models of the lexicon, phonetic detail is encoded and stored, and can inform subsequent categorization of new sound tokens.

• Variability due to coarticulation is subtracted to identify the “underlying” target sound.

(Fowler 1984; Beddor et al. 2001, 2002; Gow 2003)

Variability is retained

• Variability is useful for the identification of sounds in contexts of coarticulation.

• The perceptual system uses information about variability to identify a sound and its context, in parallel.

• Variability due to coarticulation is exploited to facilitate perception.

-- Listeners benefit from the presence of anticipatory coarticulation in predicting the identity of the upcoming sound.

(Martin & Bunnell 1982; Fowler 1981, 1984; Gow 2001, 2003; Munson, this conference)

Variability and perceptual facilitation

Perceptual facilitation from V-to-V coarticulation is expected to occur only if:

• The effects of coarticulation are systematic—an influencing vowel conditions a consistent acoustic effect on target vowels;

• The listener can recognize coarticulatory effects on the target vowel;

• The listener can isolate the effects of context vowel from other sources of variation, and attribute those effects to the context vowel.

Feature abstraction through parsing

More specifically…under coarticulation of vowel height and backness,

• The listener must parse out the portion of the variance in F1 and F2 that is due to coarticulation, and base their perception of the target vowel on the residual values.

• Acoustic parsing isolates the effects of context vowel on F1 and F2.


• The parsed acoustic variance defines features of the context vowel, over which new generalizations can be formed. phonologization

[ɛ] + [i] [ɛ] + [i]

[ɛ] + [high]

ɛi



[ɛ] + [i] [ɛ] + [i]

[ɛ] + [high]

phonologized to [i]

i



Question: Why phonologization? If target and context vowels can both be identified from the fine phonetic detail…. What’s the force driving phonologization?

Testing the model

The acoustic parsing model of speech perception requires that there is a robust and systematic pattern of acoustic variation from V-to-V coarticulation.

This paper: we present supporting evidence from an acoustic study of coarticulation.

• We examine a range of V-to-V coarticulatory effects in VCV contexts that cross a word boundary, where coarticulation cannot be attributed to lexicalized phonetic patterns.

Key Questions

Extent of phenomenon• Does V-to-V coarticulation cross word boundaries?• Does V-to-V coarticulation affect both F1 and F2?• Relative strength of V-to-V effects vs. other forms of

coarticulation?

Usefulness of phenomenon• How could V-to-V effects translate to perceptual

inferences?• Is the information by V-to-V coarticulation different

when other sources of variation are explained?

Methods

Target vowels: ɛ ʌ

Measure coarticulation

Context vowels: i æ ɑInduce Coarticulation

i

æ ɑʌɛ

• /u/ excluded from contexts (rounded + fronted)• intervening consonant varied in

- place (labial, coronal, velar)- voicing- /ɛg/ excluded (tends to be raised)

Methods

bed actor tech afternoon web addicteagle evening ecologistevergreen elevator educatorostrich Oxygen Offer

wet Afro deck alligator step AdmiralEaster Bunny easter basket eastEskimo elephant exitOxen octopus obstacle

mud apple bug astronaut pub advertisementeater evil easelumpire underwear undergradobservation optician operator

cut abdomen duck athlete cup appetizerevenly eating eavesdroppingonion usher ovenOlive officer occupant

Methods

Methods• 10 University of Illinois students.• 48 phrases x 3 repetitions.• Sentences embedded in neutral carrier sentences

/ɛ/ He said ‘_______’ all the time/ʌ/ I love ‘_______’ as a title

Coding• F1, F2, F3

- Converted to Bark for analysis• LPC (Burg Method)• Outliers / misproductions inspected by hand

Analysis

Target x Voicing x Context

F1 F2Voicing p=.033 p=.001Target p=.005 p=.001Context p=.001 p=.001Interactions n.s. n.s.

Target x Place x Context

F1 F2Place n.s. p=.001Target p=.01 p=.001Context p=.001 p=.001Interactions some some

V-to-V coarticulation crosses word boundaries.

Clear effects of coarticulatory contexton both F1 and F2.

Analysis

400

500

600

700

800

900

1000100015002000

F2 (Hz)

F1 (H

z)

æi

ɑSame

High

Low

Front Back

Male

Female

A lot of unexplained variance…

• How does the perceptual system “get to” the V-to-V coarticulation?

• How useful is V-to-V coarticulation?

• Does accounting for other sources of variance in the signal improve the usefulness of V-to-V?

Strategy

Need to systematically account for sources of variance prior to evaluating V-to-V coarticulation.

F2

ɛʌ

1431 hz 1801 hz

iɑ ?

ɑ-coarticulated ɛ?or

i-coarticulated ʌ?

Strategy

F2

ɛʌ

1431 hz 1801 hz

i?

A slightly i-coarticulated ɛ? or

A really i-coarticulated ʌ?


Strategy

F2

ɛʌ

1431 hz 1801 hz

iɑ ?

If you knew the category…If ʌ, then expect iIf ɛ then expect ɑ

? - ʌ: Positive (more i-like)? - ɛ: Negative (more ɑ-like)

F2? – F2category = coarticulation direction


Strategy

Target – F2? = coarticulation direction

F2

ɛʌ

1431 hz 1801 hz

iɑ ?

F2

ɛʌ

1431 hz 1801 hzF2

ɛʌ

1431 hz 1801 hz

iiɑɑ ??

Strategy: 1) Compute mean of a source of variance 2) Subtract that mean from F1/F23) Residual is coarticulation direction.4) Repeat for each source of variance (speaker, target

vowel, place, voicing).

Strategy

F1predicted = 1 * target + 0

If target = 0 for /ʌ/ and 1 for /ɛ/… ʌ) F1predicted = 1 * 0 + 0 Mean /ʌ/ = 0 ɛ) F1predicted = 1 * 1 + 0 Mean /ɛ/ = 0 + 1

Hierarchical Regression can do exactly these things.

1) Compute mean of a source of variance

Strategy


1) Compute mean of a source of variance.2) Subtract that mean from F1/F23) Residual is coarticulation direction.

ResidualF1actual - F1predicted = F1actual - (1 · target + 0)

ʌ) Residtarget = F1actual - 0ɛ) Residtarget = F1actual - (0 + 1)

Strategy


1) Compute mean of a source of variance.2) Subtract that mean from F1/F23) Residual is coarticulation direction.4) Repeat for each source of variance (speaker, target

vowel, place, voicing).

Residtarget = 2 * Place + 0

Residplace = 3 * Voicing+ 0

F1 = 0 * Target+ 0

Residvoicing = 4 * V-to-V + 0

Strategy

Construct a hierarchical regression to systematically account for known sources of variance from F1 and F2

• Speaker• Target vowel• Place (intervening C)• Voicing (intervening C)• Interactions between target, place & voicing

After partialing out these factors, how much variance does vowel context (V-to-V) account for?

3

4

5

6

7

889101112131415

F2 (Bark)

F1 (B

ark)

iæɑSame

Regression F2

1) Raw DataMale

Female

Regression F2

1) Raw Data

Partialed Out2) Subject

-1.5

-1

-0.5

0

0.5

1

1.5-3-2-1012

F2 Resid (Bark)

F1 R

esid

(Bar

k)i

æɑSame

ʌɛ

Regression F2

1) Raw Data

Partialed Out2) Subject3) Target Vowel

-1.5

-1

-0.5

0

0.5

1-2-1012

F2 Resid (Bark)

F1 R

esid

(Bar

k)iA

oSame

Regression F2

1) Raw Data

Partialed Out2) Subject3) Target Vowel4) Consonant

-1

-0.5

0

0.5

1-1-0.500.51

F2 Resid (Bark)

F1 R

esid

(Bar

k)iæɑSame

Regression F2

1) Raw Data

Partialed Out2) Subject3) Target Vowel4) Consonant5) Interactions

-1

-0.5

0

0.5

1-1-0.500.51

F2 Resid (Bark)

F1 R

esid

(Bar

k)iæɑSame

t

Regression F1

Step Variables R2change P

1 Subjects (10) .824 ***

Regression F1


1 Subjects (10) .824 ***2 Vowel .009 ***3 Voicing .018 ***4 Place (2) .003 **

Regression F1


1 Subjects (10) .824 ***2 Vowel .009 ***3 Voicing .018 ***4 Place (2) .003 **5 Vowel x Voicing .000 -6 Vowel x Place (2) .002 *7 Voicing x Place (2) .012 ***

Total R2=.884

Post-hoc analysis: height only.

Regression F1


1 Subjects (10) .824 ***2 Vowel .009 ***3 Voicing .018 ***4 Place (2) .003 **5 Vowel x Voicing .000 -6 Vowel x Place (2) .002 *7 Voicing x Place (2) .012 ***8 ContextVl (3) .012 ***9 ContextVL interactions (12) .003 -

Total R2=.884

Post-hoc analysis: height only.

Regression F2


1 Subjects (10) .409 ***

Regression F2


1 Subjects (10) .409 ***2 Vowel .412 ***3 Voicing .034 ***4 Place (2) .050 ***

Regression F2


1 Subjects (10) .409 ***2 Vowel .412 ***3 Voicing .034 ***4 Place (2) .050 ***5 Vowel x Voicing .008 ***6 Vowel x Place (2) .015 ***7 Voicing x Place (2) .004 ***

Regression F2


1 Subjects (10) .409 ***2 Vowel .412 ***3 Voicing .034 ***4 Place (2) .050 ***5 Vowel x Voicing .008 ***6 Vowel x Place (2) .015 ***7 Voicing x Place (2) .004 ***8 ContextVl (3) .008 ***9 ContextVL interactions (12) .001 -

Total R2=.940

Post-hoc analysis: height + backness.

Regression Summary

Progressively accounting for variance is powerfulF1: 88% of varianceF2: 94% of variance

using only known sources of variance

V-to-V coarticulation is readily apparent when other sources of variance are explained.

How useful would this be?

Effect of V-to-V coarticulation has a similar size to place/voicing effects.

Predicting Vowel Identity

Multinomial Logistic Regression (MLR)• Classification algorithm• Predict category membership from multiple variables.• Categories do not have to be binary

Samei

Sameɑ

Sameæ

Context Vowel


Samei

Sameɑ

Sameæ

Context Vowel

Samei

Samei

Sameɑ

Sameɑ

Sameæ

Sameæ

Context Vowel

• Assumes optimal listener.

• Computes % correct.• How much well could a

listener do under ideal circumstances with information provided.

Multinomial Logistic Regression (MLR)• Classification algorithm• Predict category membership from multiple variables.• Categories do not have to be binary


0

10

20

30

40

50

60

i ɑ æ Same

Vowel

% C

orre

ct

Partialed outSubjectVowelPlaceVoicingInteractions

Model does quite well at predicting all vowels but the identity.


-12

-10

-8

-6

-4

-2

0

2

4

6-42814202632

F2 (Z)

F1 (Z

)

ʌ-i

ʌ-æʌ-ɑ i

æ

ɑ

-10

-8

-6

-4

-2

0

2

4

6-18-12-6061218

F2 (Z)

F1 (Z

)

ɛ-i

ɛ-æɛ-ɑ i

æ

ɑ


Does partialing out other sources of variance improve the utility of V-to-V coarticulation?

- Use linear regression to partial out variance. - Use F1, F2 residuals to predict vowels.

FULL: Partial out everythingRAW: No parsingSPEAKER: Partial out speaker variation only. Assume

speaker normalization, but no interactions between consonant, or vowel and V-to V.

VOWEL: Partial out effects of everything heard at the target vowel (speaker + target)

NO-SPKR: Assume no normalization, but interactions between consonants.


FULL: about 4% better than others.VOWEL: parsing out consonant may not be necessarySPEAKER: Effect of speaker and phonetic cues similar.RAW: V-to-V not useful without some parsing.

2527293133353739414345

FULL VOWEL SPEAKER NO-SPKR RAW

% C

orre

ct


2) Regressively compensate for consonant coarticulation

targetvowel consonant context

vowelpreceding

context

3) Use residuals to predict context vowel

1) Parse out speaker effects on target

Suggests a 3-stage parsing process to maximally useV-to-V modifications.

Key Questions

Extent of phenomenon• Word boundaries?• Both F1 and F2?• Relative strength of V-to-V effects?

Usefulness of phenomenon• Perceptual inferences?• Parsing our variability?

Summary: Extent

• Clear evidence for V-to-V coarticulation across word boundaries—not lexicalized.

• V-to-V in both formants (height + backness).

• Strength is similar to that of place and voicing.

• Known sources of variance (speaker, vowel, consonant, V-to-V) can account for most of the variability in vowel production.

- Problem of lack of invariance?

- Identifying multiple categories at once may be easier than identifying one.

Summary: Usefulness

• Idealized listener (+ parsing) could identify upcoming vowel at 40% correct given only V-to-V coarticulation.

- Near 50% for /i/ and /ɑ/

• Parsing dramatically improves predictive power of V-to-V coarticulation

• Do you need perfect categorization of variance sources (e.g. speaker, target vowel, voicing…)?

- Imperfect categorization enhances need for multiple cues.- Simultaneously evaluating multiple features (e.g. V1, C, V2)

yields correct parse.

• How do you determine the order of parsing?- Temporal order of information arrival?

Future Directions

How do you identify the components you will be parsing?• See Toscano poster.

Does the model actually describe perception?• Parsing is a temporal process.• Visual world paradigm to time-course of processing

(e.g. McMurray, Clayards, Tanenhaus, in prep; McMurray, Tanenhaus & Aslin, 2002; McMurray, Munson & Gow, submitted).

Parsing as part of word recognition.• Lexical structure can contribute to inferences.• Interactive activation models (McClelland & Elman,

1986) could implement this.

Conclusions

Where do features come from?• Emerge out of progressively accounting for sources of

variance from signal.• Any “chunk” (segment) of the input can provide

multiple features.• Speaker normalization may work by same process.

Why phonologize?• Eliminates one step of parsing.

How does the system balance need for features with utility of fine-grained detail?

• Features provide tag to parse variance and utilize continuous detail.

parsing acoustic variability as a mechanism for feature abstraction

Documents

cc coarticulation

c coarticulation ohala

v coarticulation ohala

v coarticulation ichart14

listenerfrom v

target sound

sound patterning

acoustic properties