Experimental Design Experimental Design for Linguistsfor Linguists

Charles Clifton, Jr.Charles Clifton, Jr.

University of Massachusetts University of Massachusetts AmherstAmherst

Slides available at http://people.umass.edu/cec/teaching.html

Goals of CourseGoals of Course

►Why should linguists do experiments?Why should linguists do experiments?►How should linguists do experiments?How should linguists do experiments?

Part 1: General principles of experimental Part 1: General principles of experimental designdesign

►How should linguists do experiments?How should linguists do experiments? Part 2: Specific techniques for Part 2: Specific techniques for

(psycho)linguistic experiments(psycho)linguistic experiments

Schütze, C. (1996). The empirical basis of linguistics. Chicago: University of Chicago Press.

Cowart, W. (1997). Experimental syntax: Applying objective methods to sentence judgments. Thousand Oaks, CA: Sage Publications Inc.

Myers, J. L., & Well, A. D. (in preparation). Research design and statistical analysis (3d ed.). Mahwah, NJ: Erlbaum.

1. Acceptability judgments1. Acceptability judgments

►Check theorists’ intuitions about Check theorists’ intuitions about acceptability of sentencesacceptability of sentences Acceptability, grammaticality, Acceptability, grammaticality,

naturalness, comprehensibility, felicity, naturalness, comprehensibility, felicity, appropriateness…appropriateness…

►Aren’t theorists’ intuitions solid?Aren’t theorists’ intuitions solid?

Example of acceptability Example of acceptability judgment: Cowart, 1997judgment: Cowart, 1997

► Subject extraction: Subject extraction: I wonder who you think I wonder who you think (that) likes John.(that) likes John.

► Object extraction: Object extraction: I wonder who you think I wonder who you think (that) John likes.(that) John likes.

No-That No-That

That

That

-0.8

-0.6-0.4

-0.2

0

0.20.4

0.6

Subject Extraction Object Extraction

Mea

n ju

dged

acc

epta

bilit

y (z

-sco

re)

Stability of ratings Stability of ratings (Cowart,1997)(Cowart,1997)

2. Sometimes linguists are 2. Sometimes linguists are wrong…wrong…

►Superiority effectsSuperiority effects I’d like to know who hid it where.I’d like to know who hid it where. *I’d like to know where who hid it.*I’d like to know where who hid it.

►Ameliorated by a third wh-phrase?Ameliorated by a third wh-phrase? ?I’d like to know where who hid it when.?I’d like to know where who hid it when.

……maybe. Paired-comparison maybe. Paired-comparison preference judgmentspreference judgments

a. I’d like to know who hid it where. 86%b. (*)I’d like to know where who hid it. 14% 76%c. (*)I’d like to know where who hid it when. 24%.49%d. I’d like to know who hid it where when.51%

a-b basic superiority violation

b-c heads-on comparison, extra wh “when” hurts, doesn’t help

c-d the “ameliorated” superiority violation, c, seems good when compared to its non-superiority-violation counterpart

Clifton, C. Jr., Fanselow, G., & Frazier, L. (2006). Amnestying superiority violations: Processing multiple questions. Linguistic Inguiry, 37(51-68).

Another instance…Another instance…

Question: is the antecedent of an ellipsis a syntactic or a semantic object? Why is (a) good and (b) bad?

(a) The problem was to have been looked into, but obviously nobody did.

(b) #The problem was looked into by John, and Bob did too.

Andrew Kehler’s suggestion: semantic objects for cause-effect discourse relations, syntactic objects for resemblance relations. Corpus data bear his suggestion out.

But an experimental But an experimental approach…approach…

Kim looked into the problem even though Lee did. (causal, syntactic parallel)

Kim looked into the problem just like Lee did. (resemblance)

The problem was looked into by Kim even though Lee did. (causal, nonparallel)

The problem was looked into by Kim just like Lee did. (resemblance)

2.5

3

3.5

4

4.5

Parallel NonParallel

Mean Acceptability Rating (5 = good)

CausalResemblance

Frazier, L., & Clifton, C. J. (2006). Ellipsis and discourse coherence. Linguistics and Philosophy, 29, 315-346.

Context effectsContext effects

►Linguists: think of minimal pairsLinguists: think of minimal pairs►The contrast between a pair may The contrast between a pair may

affect judgmentsaffect judgments►Hirotani: Production of Japanese Hirotani: Production of Japanese

sentencessentences The experimental context in which The experimental context in which

sentences are produced affects their sentences are produced affects their prosodyprosody

Hirotani experimentHirotani experimenta. Embedded wh-question (ka associated to na’ni-o) (# = Major phrase

boundary)

Mi’nako-san-wa Ya’tabe-kun-ga na’ni-o moyasita’ka (#) gumon-sita’-nokai?

Minako-Ms.-TOP Yatabe-Mr.-NOM what-ACC burned-Q stupid question-did-Q (-wh)

‘Did Minako ask stupidly what Yatabe burned?’ (Yes, it seems (she) asked such a question.’)

b. Matrix wh-question (ndai associated to na’ni-o)

Mi’nako-san-wa Ya’tabe-kun-ga na’ni-o moyasita’ka (#) gumon-sita’-ndai?

Minako-Ms.-TOP Yatabe-Mr.-NOM what-ACC burned-Q stupid question-did-Q (+wh)

‘What did Minako ask stupidly whether Yatabe burned?” (‘The letters (he) received from (his) ex-girlfriend.’)

Hirotani resultsHirotani results

Initial Block Initial Block (pure)(pure)

Final Block Final Block (pair (pair contrast)contrast)

Embedded Embedded questionquestion

100%100% 100%100%

MatrixMatrix

questionquestion57%57% 15%15%

Percentage of insertion of MaP before phrase with question particle

Hirotani, Mako. (submitted). Prosodic phrasing of wh-questions in Japanese

3. Unacceptable 3. Unacceptable grammaticalitygrammaticality

► Old multiple self-embedding sentence Old multiple self-embedding sentence experimentsexperiments Miller & Isard 1964: sentence recall, right-Miller & Isard 1964: sentence recall, right-

branching vs. self-embedded (1-4)branching vs. self-embedded (1-4)► She liked the man that visited the jeweler that made She liked the man that visited the jeweler that made

the ring that won the prize that was given at the fair.the ring that won the prize that was given at the fair.► The prize that the ring that the jeweler that the man The prize that the ring that the jeweler that the man

that she liked visited made won was given at the fair.that she liked visited made won was given at the fair.► Median trial of first perfect recall: 2.25 vs neverMedian trial of first perfect recall: 2.25 vs never

Stolz 1967, clausal paraphrases: subjects never Stolz 1967, clausal paraphrases: subjects never understood the self-embedded sentences understood the self-embedded sentences anywayanyway

Miller, G. A., & Isard, S. (1964). Free recall of self-embedded English sentences. Information and Control, 4, 292-303. Stolz, W. (1967). A study of the ability to decode grammatically novel sentences. Journal of verbal Learning and verbal Behavior, 6, 867-873..

3’. Acceptable 3’. Acceptable ungrammaticalityungrammaticality

Speeded acceptability judgment and acceptability rating

%OKRating

a. OK

None of the astronomers saw the comet, but John did. 83%4.36

B. Embedded VP

Seeing the comet was nearly impossible, but John did. 66%3.71

C. VP w/ trace

The comet was nearly impossible to see, / but John did. 44%3.27

D. Neg adj

The comet was nearly unseeable, / but John did. 17%2.21

Arregui, A., Clifton, C. J., Frazier, L., & Moulton, K. (2006). Processing elided verb phrases with flawed antecedents: The recycling hypothesis. Journal of Memory and Language, 55, 232-246.

4. Provide additional 4. Provide additional evidence about linguistic evidence about linguistic

structurestructure►A direct experimental reflex of A direct experimental reflex of

structure would be nicestructure would be nice But we don’t have oneBut we don’t have one

►Are traces real?Are traces real? Filled gap effect: reading slowed at Filled gap effect: reading slowed at usus in in

My brother wanted to know who Ruth will My brother wanted to know who Ruth will bring (t) bring (t) us us home to at Christmas.home to at Christmas.

Compared to Compared to My brother wanted to know My brother wanted to know if Ruth will bring if Ruth will bring us us home to at Christmas.home to at Christmas.

Stowe, L. (1986). Parsing wh-constructions: Evidence for on-line gap location. Language and Cognitive Processes, 1, 227-246.

Are traces real, cont.Are traces real, cont.

► Pickering and Barry. “no.”Pickering and Barry. “no.”► Possible evidencePossible evidence

That’s the pistol with which the heartless killer That’s the pistol with which the heartless killer shotshot the hapless man yesterday afternoon the hapless man yesterday afternoon tt..

That’s the garage with which the heartless killer That’s the garage with which the heartless killer shotshot the hapless man yesterday afternoon the hapless man yesterday afternoon tt..

► Reading disrupted at Reading disrupted at shotshot in the second in the second example, far before the trace positionexample, far before the trace position But who’s to say that the parser has to wait to But who’s to say that the parser has to wait to

project the trace?project the trace?Pickering, M., & Barry, G. (1991). Sentence processing without empty categories. Language and Cognitive Processes, 6, 229-259.

Traxler, M. J., & Pickering, M. J. (1996). Plausibility and the processing of unbounded dependencies: An eye-tracking study. Journal of Memory and Language, 35, 454-475.

5. Is grammatical knowledge 5. Is grammatical knowledge used?used?

►Serious question early onSerious question early on ““psychological reality” experimentspsychological reality” experiments

►Direct experimental attack did not Direct experimental attack did not succeedsucceed Derivational theory of complexityDerivational theory of complexity

► Indirect experimental attack has Indirect experimental attack has succeededsucceeded Build experimentally-based theory of Build experimentally-based theory of

processingprocessing

6. Test theories of how 6. Test theories of how grammatical knowledge is grammatical knowledge is

usedused► Moving beyond modularity debate – more Moving beyond modularity debate – more

articulated questions about real-time use of articulated questions about real-time use of grammar grammar

► Phillips: parasitic gaps, selfpaced readingPhillips: parasitic gaps, selfpaced reading The superintendent learned which The superintendent learned which schools/studentsschools/students

the plan to the plan to expand _expand _ … overburdened _. (slowed at … overburdened _. (slowed at expandexpand after after students – students – plausibility effect) plausibility effect)

The superintendent learned which The superintendent learned which schools/studentsschools/students the plan that the plan that expanded _expanded _ … overburdened _. (no … overburdened _. (no differential slowing at differential slowing at expand – expand – no plausibility no plausibility effect)effect)

Phillips, C. (2006) The real-time status of island phenomena. Language, 82, 795-823.

II: How to do experiments. II: How to do experiments. Part 1, General design Part 1, General design

principlesprinciples►Dictum 1: Formulate your question Dictum 1: Formulate your question

clearlyclearly►Dictum 2: Keep everything constant that Dictum 2: Keep everything constant that

you don’t want to varyyou don’t want to vary►Dictum 3: Know how to deal with Dictum 3: Know how to deal with

unavoidable extraneous variabilityunavoidable extraneous variability►Dictum 4: Have enough power in your Dictum 4: Have enough power in your

experimentexperiment►Dictum 5: Pay attention to your data, not Dictum 5: Pay attention to your data, not

just your statistical testsjust your statistical tests

Dictum 1: Formulate your Dictum 1: Formulate your question clearlyquestion clearly

► Independent variable: variation Independent variable: variation controlled be experimenter, not by controlled be experimenter, not by what subject doeswhat subject does

►Dependent variable: variation Dependent variable: variation observed in subject’s behavior, observed in subject’s behavior, perhaps dependent on IVperhaps dependent on IV

►Operationalization of variablesOperationalization of variables

Formulate your questionFormulate your question

►Question: Do you identify a focused Question: Do you identify a focused word faster than a non-focused word?word faster than a non-focused word? Must clarify: Syntactic focus? Prosodic Must clarify: Syntactic focus? Prosodic

focus? Semantic focus?focus? Semantic focus? Must operationalizeMust operationalize

►Syntactic focus – Clefting? Fronting? Other Syntactic focus – Clefting? Fronting? Other device?device?

►Prosodic focus – Natural speech? Manipulated Prosodic focus – Natural speech? Manipulated speech? Synthetic speech? Target word or speech? Synthetic speech? Target word or context?context?

Formulate your questionFormulate your question

►Question: does discourse context Question: does discourse context guide or filter parsing decisions?guide or filter parsing decisions? Clarify question: does discourse satisfy Clarify question: does discourse satisfy

reference? establish plausibility? set up reference? establish plausibility? set up pragmatic implications? create syntactic pragmatic implications? create syntactic structure biases?structure biases?

Operationalize IV: Operationalize IV: LotsLots of choices here of choices here►But also have to worry about dependent But also have to worry about dependent

variable…variable…

Choose appropriate task, DVChoose appropriate task, DV

► Question about focus: need measure of Question about focus: need measure of speed of word identificationspeed of word identification Conventional possibilities: lexical decision, Conventional possibilities: lexical decision,

naming, phoneme detection, reading timenaming, phoneme detection, reading time► Question about “guide vs filter:” probably Question about “guide vs filter:” probably

need explicit theory of your taskneed explicit theory of your task Tanenhaus: linking hypothesisTanenhaus: linking hypothesis E.g. eye movements in reading: tempting to E.g. eye movements in reading: tempting to

think that “guide” implicates “early measures,” think that “guide” implicates “early measures,” “filter” implicated “late measures.”“filter” implicated “late measures.”

► But what’s early, what’s late? Need model of eye But what’s early, what’s late? Need model of eye movement control in parsing.movement control in parsing.

Subdictum A: Never leave your Subdictum A: Never leave your subjects to their own devicessubjects to their own devices

► It may not matter a lotIt may not matter a lot Cowart example: 5-point acceptability Cowart example: 5-point acceptability

ratingrating►A. “….base your responses solely on your gut A. “….base your responses solely on your gut

reaction”reaction”►B. “…would you expect the professor to B. “…would you expect the professor to

accept this sentence [for a term paper in an accept this sentence [for a term paper in an advanced English course]?”advanced English course]?”

►But sometimes it does matter…But sometimes it does matter…

Cowart 1997Cowart 1997

Dictum 2: Try to keep Dictum 2: Try to keep everything constant except everything constant except

what you want to varywhat you want to vary►Try to hold extraneous variables Try to hold extraneous variables

constant through norms, pretests, constant through norms, pretests, corpora…corpora…

►When you can’t hold them constant, When you can’t hold them constant, make sure they are not associated make sure they are not associated (confounded) with your IV(confounded) with your IV

An example: Staub, in pressAn example: Staub, in press

Staub, A. (in press). The parser doesn't ignore intransitivity, after all. Journal of Experimental Psychology: Learning, Memory and Cognition.

Eyetracking: does the reader honor intransitivity? Compare unaccusative (a), unergative (b), and optionally transitive)

a. When the dog arrived the vet1 and his new assistant took off the muzzle2.b. When the dog struggled the vet1 and his new assistant took off the muzzle2.c. When the dog scratched the vet1 and his new assistant took off the muzzle2.

Critical regions: held constant (the vet…; took off the muzzle).

Manipulated variable (verb): conditions equated on average length and average word frequency of occurrence.

Better: match on additional factors (number of stressed syllables, concreteness, plausibility as intransitive, ….)

Better: don’t just have overall match, but match the items in each triple.

Another example: NP vs S-comp Another example: NP vs S-comp biasbias

Kennison, S. M. (2001). Limitations on the use of verb information during sentence comprehension. Psychonomic Bulletin & Review, 8, 132-137.

Kennison (2001), eyetracking during reading of sentences like:

a. The athlete admitted/revealed (that) his problem worried his parents….

b. The athlete admitted/revealed his problem because his parents worried…

Conflicting results from previous research (Ferreira & Henderson, 1990; Trueswell, Tanenhaus, & Kello, 1993): does a bias toward use as S-complement (admit) reduce the disruption at the disambiguating word worried?

Problems in previous research: plausibility of direct object analysis not controlled (e.g., Trueswell et al., ambiguous NP (his problem) rated as implausible as direct object of S-biased verb)

Kennison, normed material, equated plausibility of subject-verb-object fragment for NP- and S-comp biased verbs; found reading disrupted equally at disambiguating verb worried for both types of verbs.

What happens when there is What happens when there is unavoidable variation?unavoidable variation?

► Subdictum B: When in doubt, randomizeSubdictum B: When in doubt, randomize Random assignment of subjects to conditionsRandom assignment of subjects to conditions Questionnaire: order of presentation of items?Questionnaire: order of presentation of items?

► Single randomization: problemsSingle randomization: problems► Different randomization for each subjectDifferent randomization for each subject► Constrained randomizationsConstrained randomizations

► Equate confounds by balancing and Equate confounds by balancing and counterbalancingcounterbalancing Alternative to random assignment of subject to Alternative to random assignment of subject to

conditions: match squads of subjectsconditions: match squads of subjects

Counterbalancing of Counterbalancing of materialsmaterials

►CounterbalancingCounterbalancing Ensure that each item is tested equally Ensure that each item is tested equally

often in each condition.often in each condition. Ensure that each subject receives an Ensure that each subject receives an

equal number of items in each condition.equal number of items in each condition.►Why is it necessary?Why is it necessary?

Since items and subjects may differ in Since items and subjects may differ in ways that affect your DV, you can’t have ways that affect your DV, you can’t have some items (or subjects) contribute more some items (or subjects) contribute more to one level of your IV than another level.to one level of your IV than another level.

Sometimes you don’t have to Sometimes you don’t have to counterbalancecounterbalance

► If you can test each subject on each item in If you can test each subject on each item in each condition, life is sweeteach condition, life is sweet

► E.g., Ganong effect (identification of E.g., Ganong effect (identification of consonant in context)consonant in context) Vary VOT in 8 5-ms stepsVary VOT in 8 5-ms steps

► /dais/ - /tais//dais/ - /tais/► /daip/ - /taip//daip/ - /taip/

Classify initial segment as /d/ or /t/Classify initial segment as /d/ or /t/► Present each of the 80 items to each subject 10 timesPresent each of the 80 items to each subject 10 times► Ganong effect: biased toward /t/ in “type,” /d/ in “dice”Ganong effect: biased toward /t/ in “type,” /d/ in “dice”

Connine, C. M., & Clifton, C., Jr. (1987). Interactive use of information in speech perception. Journal of Experimental Psychology: Human Perception and Performance, 13, 291-299.

If you have to If you have to counterbalance…counterbalance…

► Simple exampleSimple example Questionnaire, 2 conditions, N itemsQuestionnaire, 2 conditions, N items Need 2 versions, each with N items, N/2 in Need 2 versions, each with N items, N/2 in

condition 1, remaining half in condition 2condition 1, remaining half in condition 2► Versions 1 and 2, opposite assignment of items to Versions 1 and 2, opposite assignment of items to

conditionsconditions

► More general versionMore general version M conditions, need some multiple of M items, M conditions, need some multiple of M items,

and need M different versionsand need M different versions► Embarrassing if you have 15 items, 4 conditions…Embarrassing if you have 15 items, 4 conditions…► That means that some subjects contributed more to That means that some subjects contributed more to

some conditions than others did; bad, if there are true some conditions than others did; bad, if there are true differences among subjectsdifferences among subjects

Counterbalancing things Counterbalancing things besides itemsbesides items

► Order of testingOrder of testing Don’t test all Ss in one condition, then the next Don’t test all Ss in one condition, then the next

condition…condition… At least, cycle through one condition before At least, cycle through one condition before

testing a second subjecttesting a second subject Fancier, latin squareFancier, latin square

► Avoid minor confound if always test cond 1 before cond 2 Avoid minor confound if always test cond 1 before cond 2 etc.etc.

► N x n square, sequence x squad, containing condition N x n square, sequence x squad, containing condition numbers, such that each condition occurs once in each numbers, such that each condition occurs once in each column, each ordercolumn, each order

► Location of testingLocation of testing E.g., 2 experiment stationsE.g., 2 experiment stations

Experimental Design Experimental Design for Linguistsfor Linguists

Charles Clifton, Jr.Charles Clifton, Jr.

University of Massachusetts University of Massachusetts AmherstAmherst

Slides available at http://people.umass.edu/cec/teaching.html

and at

http://coursework.stanford.edu

Goals of CourseGoals of Course

►Why should linguists do experiments?Why should linguists do experiments?►How should linguists do experiments?How should linguists do experiments?

Part 1: General principles of experimental Part 1: General principles of experimental designdesign

►How should linguists do experiments?How should linguists do experiments? Part 2: Specific techniques for Part 2: Specific techniques for

(psycho)linguistic experiments(psycho)linguistic experiments

Schütze, C. (1996). The empirical basis of linguistics. Chicago: University of Chicago Press.

Cowart, W. (1997). Experimental syntax: Applying objective methods to sentence judgments. Thousand Oaks, CA: Sage Publications Inc.

Myers, J. L., & Well, A. D. (in preparation). Research design and statistical analysis (3d ed.). Mahwah, NJ: Erlbaum.

II: How to do experiments. II: How to do experiments. Part 1, General design Part 1, General design

principlesprinciples►Dictum 1: Formulate your question Dictum 1: Formulate your question

clearlyclearly►Dictum 2: Keep everything constant that Dictum 2: Keep everything constant that

you don’t want to varyyou don’t want to vary►Dictum 3: Know how to deal with Dictum 3: Know how to deal with

unavoidable extraneous variabilityunavoidable extraneous variability►Dictum 4: Have enough power in your Dictum 4: Have enough power in your

experimentexperiment►Dictum 5: Pay attention to your data, not Dictum 5: Pay attention to your data, not

just your statistical testsjust your statistical tests

So how do you randomize?So how do you randomize?

►E-mail me (E-mail me (cec@psych.umass.educec@psych.umass.edu) ) and I’ll send you a powerful programand I’ll send you a powerful program

►But for most purposes, check outBut for most purposes, check outhttp://www-users.york.ac.uk/~mb55/guide/randsery.htmhttp://www-users.york.ac.uk/~mb55/guide/randsery.htm

OrOr

http://www.randomizer.org/index.htmhttp://www.randomizer.org/index.htm

Factor out confoundsFactor out confounds

►Factorial designFactorial design An example, discussed earlier: Arregui et An example, discussed earlier: Arregui et

al., 2006al., 2006 Initial experiment contained a confound; Initial experiment contained a confound;

corrected in second experiment by adding corrected in second experiment by adding a second factora second factor

Arregui et al., rating studyArregui et al., rating studyAcceptability rating

RatingRating

clause 1

a. OK

None of the astronomers saw the comet, but John did. 4.36 4.53

B. Embedded VP

Seeing the comet was nearly impossible, but John did. 3.71 4.41

C. VP w/ trace

The comet was nearly impossible to see, but John did. 3.27 4.81

D. Neg adj

The comet was nearly unseeable, but John did. 2.21 4.39Arregui, A., Clifton, C. J., Frazier, L., & Moulton, K. (2006). Processing elided verb phrases with flawed antecedents: The recycling hypothesis. Journal of Memory and Language, 55, 232-246.

Factorial DesignFactorial Design

First clauseFirst clause Ellipsis absentEllipsis absent Ellipsis Ellipsis presentpresent

Syntactically Syntactically OKOK

None of the astronomers saw the comet.

..but John did...but John did.

Embedded VPEmbedded VP Seeing the comet was nearly impossible.

..but John did...but John did.

VP with traceVP with trace The comet was nearly impossible to see.

..but John did...but John did.

NominalizationNominalization The comet was nearly unseeable.

..but John did...but John did.

Factor 1: syntactic form of initial clause (4 levels)

Factor 2: presence or absence of ellipsis (2 levels)

An interactionAn interaction

0

1

2

3

4

5

Ellipsis absent Ellipsispresent

Mean Acceptability Rating

OKEmbed VPVP traceNominal

Interaction: The size of the effect of one factor differs among the different levels of the other factor.

Factorial Designs in Factorial Designs in Hypothesis TestingHypothesis Testing

► Cowart (1997), that-trace effectCowart (1997), that-trace effect Question: is it bad to extract a subject over Question: is it bad to extract a subject over thatthat

► ?I wonder who you think (that) t likes John.?I wonder who you think (that) t likes John. Acceptability judgment: worse with Acceptability judgment: worse with thatthat

► But: underlying theory talks just about But: underlying theory talks just about extracting a extracting a subjectsubject. . Does acceptability suffer with extraction of object Does acceptability suffer with extraction of object

over over thatthat? ? ► I wonder who you think (that) John likes t.I wonder who you think (that) John likes t.

Need to do factorial experimentNeed to do factorial experiment► Factor 1: presence vs. absence of Factor 1: presence vs. absence of thatthat► Factor 2: subject vs. object extractionFactor 2: subject vs. object extraction

The results (from before)The results (from before)

No-That No-That

That

That

-0.8

-0.6-0.4

-0.2

0

0.20.4

0.6

Subject Extraction Object Extraction

Mea

n ju

dged

acc

epta

bilit

y (z

-sco

re)

A clear interaction.

A worry about scalesA worry about scales

► Interactions of the form “the effect of Factor A Interactions of the form “the effect of Factor A is bigger at Level 1 than at Level 2 of Factor B.is bigger at Level 1 than at Level 2 of Factor B. Cowart, effect of Cowart, effect of thatthat bigger at subject than object bigger at subject than object

extractionextraction► Types of scalesTypes of scales

Ratio: true zero, equal intervals, can talk about Ratio: true zero, equal intervals, can talk about ratios (time, distance, weight)ratios (time, distance, weight)

Interval: equal intervals, but no true zero Interval: equal intervals, but no true zero (temperature, dates on a calendar)(temperature, dates on a calendar)

Ordinal: only more or less (ratings on rating scale, Ordinal: only more or less (ratings on rating scale, measures of acceptability, measures of difficulty)measures of acceptability, measures of difficulty)

0

1

2

3

Factor 2-1 Factor 2-2

Log scale

Factor 1-1Factor 1-2

0

200

400

600

800

1000

Factor 2-1 Factor 2-2

Original Scale

Factor 1-1Factor 1-2

Is there really an interaction?

Disordinal and crossover Disordinal and crossover interactionsinteractions

0

5

10

15

20

25

30

Factor 2-1 Factor 2-2

Factor 1-1 Factor 1-2

0

5

10

15

20

25

30

35

Factor 2-1 Factor 2-2

Factor 1-1 Factor 1-2

An example of an important but An example of an important but problematic experiment: Frazier & problematic experiment: Frazier &

Rayner, 1982Rayner, 1982

Frazier, L., & Rayner, K. (1982). Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14, 178-210.

Closure:LC: Since Jay always jogs a mile and a half this seems like a short distance to

him.40 40 ms/ch

EC: Since Jay always jogs a mile and a half seems like a very short distance to him.

35 54 ms/chAttachment:

MA: The lawyers think his second wife will claim the entire family inheritance.36 ms/ch

NMA: The second wife will claim the entire family inheritance belongs to her.37 51 ms/ch

Data shown: ms/character first pass times for the colored regions.

Problems???

Dictum 3: Know how to deal Dictum 3: Know how to deal with unavoidable extraneous with unavoidable extraneous

variabilityvariability► i.e., know some statisticsi.e., know some statistics►Measures of central tendency (“typical”)Measures of central tendency (“typical”)

Mean (average, sum/N)Mean (average, sum/N) Median (middle value)Median (middle value) Mode (most frequent value)Mode (most frequent value)

►Measures of variabilityMeasures of variability Variance (Average squared deviation from Variance (Average squared deviation from

mean)mean) Average deviation (Average absolute Average deviation (Average absolute

deviation from median)deviation from median)

Computation of VarianceComputation of VarianceDistr 1Distr 1 XX Mean-Mean-

XXSq’dSq’d Distr 2Distr 2 XX Mean-Mean-

XXSq’dSq’d

77 99 8181 1212 44 1616

1010 66 3636 1414 22 44

…… ……

3030 -14-14 196196 2121 -5-5 2525

1717 -1-1 11 1717 -1-1 11SumSum 6464 314314 6464 4646MeanMean 1616 VariancVarianc

ee78.578.5 1616 VariancVarianc

ee11.511.5

Variance in an experimentVariance in an experiment

►Systematic variance: variability due to Systematic variance: variability due to manipulation of IV and other variables manipulation of IV and other variables you can identifyyou can identify

►Random variance: variability whose Random variance: variability whose origin you’re ignorant oforigin you’re ignorant of

►Point of inferential statistics: is there Point of inferential statistics: is there really variability associated with IV, on really variability associated with IV, on top of other variability?top of other variability? Is there a signal in the noise?Is there a signal in the noise?

Best way to deal with Best way to deal with extraneous variability: Minimize extraneous variability: Minimize

it!it!►Keep everything constantKeep everything constant

Reduce experimental noiseReduce experimental noise►See the signal easierSee the signal easier

Keep environment, instructions, Keep environment, instructions, distractions, experimenter, response distractions, experimenter, response manipulanda, etc. constantmanipulanda, etc. constant

Pretest subjects and select homogeneous Pretest subjects and select homogeneous ones, if that suits your purposesones, if that suits your purposes

One way to minimize One way to minimize extraneous variance: Within-extraneous variance: Within-

subject designssubject designs► Subjects differSubjects differ

……a lot, in some measures, eg. Reading speed, reaction timea lot, in some measures, eg. Reading speed, reaction time► Present all levels of your IV to each subjectPresent all levels of your IV to each subject

Assume the subject effect is a constant across all the levels.Assume the subject effect is a constant across all the levels. Differences among conditions thus abstracted from subject Differences among conditions thus abstracted from subject

differencesdifferences► Counterbalancing necessaryCounterbalancing necessary

Test each item in each condition for an equal number of Test each item in each condition for an equal number of subjects.subjects.

► Worry about experience changing what your subject Worry about experience changing what your subject diddid E.g., will reading an unreduced relative clause (E.g., will reading an unreduced relative clause (The horse The horse

that was raced past the barn fellthat was raced past the barn fell) affect reading of a reduced ) affect reading of a reduced relative clause sentence?relative clause sentence?

Statistical tests/statistical Statistical tests/statistical inferenceinference

►Never expect observed condition means Never expect observed condition means to be exactly the sameto be exactly the same Just noise? Or signal + noise?Just noise? Or signal + noise?

►Statistical inference: is there really a Statistical inference: is there really a signal?signal? p value: the probability you’d obtain a p value: the probability you’d obtain a

difference among the means that is as large difference among the means that is as large as what you observed, if the true signal is as what you observed, if the true signal is zerozero

““null hypothesis” testnull hypothesis” test

Basic logic of statistical tests (t, Basic logic of statistical tests (t, F, etc.)F, etc.)

► Get one estimate of the variabilty due to Get one estimate of the variabilty due to noise + any signalnoise + any signal Estimate from the variation among the observed Estimate from the variation among the observed

mean values in the different conditionsmean values in the different conditions► Get another estimate of the variabilty due to Get another estimate of the variabilty due to

noise alonenoise alone Estimate from how much variation there is Estimate from how much variation there is

among subjects, within a conditionamong subjects, within a condition► If signal = 0, ratio is expected to be 1If signal = 0, ratio is expected to be 1

If it’s enough bigger than 1, then the signal is If it’s enough bigger than 1, then the signal is likely to be non-zerolikely to be non-zero

Underlying modelUnderlying model

► Subjects are a random sample from some Subjects are a random sample from some populationpopulation

► You can make inferences about variability in You can make inferences about variability in the population from the observed variability in the population from the observed variability in the samplethe sample

► Logical inference: “if the size of the signal in Logical inference: “if the size of the signal in the population is zero, the probability of the population is zero, the probability of getting a difference among the means that is getting a difference among the means that is as big as we observed is as big as we observed is pp” where ” where pp is the is the level of significancelevel of significance If If pp is small enough, reject the proposal that the is small enough, reject the proposal that the

population signal is zeropopulation signal is zero

Between-subject designBetween-subject design

►Estimate of signal + noise: variability Estimate of signal + noise: variability among the condition meansamong the condition means

►Estimate of noise alone: variability Estimate of noise alone: variability among the subject means in each among the subject means in each conditioncondition

►F = MSF = MSbetween condsbetween conds/MS/MSwithin condwithin cond

MS, not exactly variance; must divide sum MS, not exactly variance; must divide sum of squares by df, not by Nof squares by df, not by N

Within-subject designWithin-subject design

► Estimate of signal + noise: variability Estimate of signal + noise: variability between the condition meansbetween the condition means

► Estimate of noise aloneEstimate of noise alone Get a measure of the variability among condition Get a measure of the variability among condition

means for each subjectmeans for each subject Calculate the variability among these measuresCalculate the variability among these measures Subjects x treatment interactionSubjects x treatment interaction

► How much the the size of the treatment effect differs How much the the size of the treatment effect differs among subjects is an estimate of error variability.among subjects is an estimate of error variability.

► F = MSF = MSbetween conditionsbetween conditions/MS/MSsubjects x treatmentssubjects x treatments

Advanced topicsAdvanced topics

► Multi-factor designs, tests for interactionsMulti-factor designs, tests for interactions► Treat counterbalancing factors as factors in Treat counterbalancing factors as factors in

ANOVAANOVA E.g., if have 4 conditions, 4 counterbalancing E.g., if have 4 conditions, 4 counterbalancing

groups, differing in assignment of items to groups, differing in assignment of items to conditions, you can treat groups as a between-conditions, you can treat groups as a between-subject factor and pull out variability due to items subject factor and pull out variability due to items from the subjects x treatment error termfrom the subjects x treatment error term

► Statistical accommodation of extraneous Statistical accommodation of extraneous variationvariation Analysis of covarianceAnalysis of covariance Multi-level, hierarchical designsMulti-level, hierarchical designs

Pollatsek, A., & Well, A. D. (1995). On the use of counterbalanced designs in cognitive research: A suggestion for a better and more powerful analysis. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21, 785-794.

Forthcoming special issue of the Journal of Memory and Language on new and alternative data analyses.

Dictum 4: Have enough power Dictum 4: Have enough power to overcome extraneous to overcome extraneous

variabilityvariability►Add more data!Add more data!

Minimizes noise component of differences Minimizes noise component of differences among condition meansamong condition means

►Law of large numbersLaw of large numbers The larger the sample size, the more The larger the sample size, the more

probable it is that the sample mean comes probable it is that the sample mean comes arbitrarily close to the population meanarbitrarily close to the population mean

If you’re (almost) looking at population If you’re (almost) looking at population means, any differences have to be real – means, any differences have to be real – not sampling errornot sampling error

Law of large numbersLaw of large numbers

► Imagine a population with a variance Imagine a population with a variance vv22..► Imagine you take a bunch of Imagine you take a bunch of

independent samples from this independent samples from this population, each sample of size population, each sample of size N.N.

►Each sample will have a mean value.Each sample will have a mean value.►These mean values will have a variance, These mean values will have a variance,

which turns out to be which turns out to be vv22/N./N.►This variance will be smaller as N gets This variance will be smaller as N gets

larger.larger.

http://onlinestatbook.com/stat_sim/index.html

A sampling simulationA sampling simulation►The effect of sample size on the The effect of sample size on the

variability of sample meansvariability of sample means Bigger samples, smaller variabilityBigger samples, smaller variability Standard deviation = square root of Standard deviation = square root of

variancevariance

N = 5 N = 25

Means from larger Ns have less Means from larger Ns have less noisenoise

►Holds for subject meansHolds for subject means More subjects, means reflect vagaries of More subjects, means reflect vagaries of

sample less; means have less noisesample less; means have less noise

►Holds for item means tooHolds for item means too More items, means less affected by More items, means less affected by

peculiarity of individual itemspeculiarity of individual items

►OK, you can have too many items and OK, you can have too many items and burn out your subjectsburn out your subjects

Have enough power….Have enough power….

►Back to holding everything constantBack to holding everything constant First reason: don’t want variables First reason: don’t want variables

confounded with our independent variableconfounded with our independent variable Second reason: minimize noise. Less Second reason: minimize noise. Less

noise, more power.noise, more power.

Dictum 5: Pay attention to your Dictum 5: Pay attention to your data, not just your statistical data, not just your statistical

teststests►Look at your data, graph them, try to Look at your data, graph them, try to

make sense out of themmake sense out of them Don’t just look for p < .05!Don’t just look for p < .05!

►Examine confidence intervalsExamine confidence intervals►Look at your data distributionsLook at your data distributions

Stem and leaf graphsStem and leaf graphs By subjects…By subjects…

Confidence intervalsConfidence intervals

► Confidence intervals (of means over items and Confidence intervals (of means over items and subjects)subjects) If you have a sample mean and you know the true If you have a sample mean and you know the true

population standard deviation of the sample population standard deviation of the sample σσMM , , you can say that there is a 95% chance that the you can say that there is a 95% chance that the true population mean is within +/- 1.96 * true population mean is within +/- 1.96 * σσMM your your sample mean.sample mean.

But of course you don’t know But of course you don’t know σσM M so you have to so you have to estimate it from your data and use the t estimate it from your data and use the t distribution. distribution.

But then you can present your means as X +/- CIBut then you can present your means as X +/- CI► A simulation: A simulation: http://onlinestatbook.com/stat_sim/index.html

Confidence IntervalsConfidence Intervals

► Do you want to look at individual item data?Do you want to look at individual item data? Don’t make too much of the tea leavesDon’t make too much of the tea leaves Consider getting a confidence interval on the Consider getting a confidence interval on the

individual item meansindividual item means Example: Self-paced reading timeExample: Self-paced reading time

► Cond 1: This table is slightly dirty Cond 1: This table is slightly dirty and the manager and the manager wants it removedwants it removed. (minimum standard adjective). (minimum standard adjective)

► Cond 1: This table is slightly clean Cond 1: This table is slightly clean and the manager and the manager wants it removed. wants it removed. (maximum standard adjective)(maximum standard adjective)

► Reading time, clause 2, slower for maximum than Reading time, clause 2, slower for maximum than minimum standard adjective minimum standard adjective

Are some items more effective than others?Are some items more effective than others?

Confidence intervals, Confidence intervals, individual itemsindividual items

►Each item has 12 different Each item has 12 different observations (different subjects) in observations (different subjects) in each condition.each condition.

►Can measure the variability among Can measure the variability among these subject data points for max std these subject data points for max std and min std adjectiveand min std adjective And from that, estimate the variability of And from that, estimate the variability of

the difference, and from that, the the difference, and from that, the confidence interval of the differenceconfidence interval of the difference

Max std Max std adjadj

Min std Min std adjadj

Mean MaxMean Max Mean MinMean Min DiffDiff 95% CI 95% CI DiffDiff

CleanClean DirtyDirty 19601960 14861486 474474 +/- 662+/- 662

SafeSafe DangerouDangerouss

15721572 12581258 314314 +/- 303+/- 303

HealthyHealthy SickSick 16351635 11641164 471471 +/- 485+/- 485

DryDry WetWet 11301130 12291229 -99-99 +/- 427+/- 427

CompleteComplete IncompletIncompletee

11961196 16171617 -421-421 +/- 621+/- 621

Dictum 5: Pay attention to your Dictum 5: Pay attention to your data, not just your statistical data, not just your statistical

teststests►Graph your data Graph your data ►Examine confidence intervalsExamine confidence intervals►Look at the distributions of your meansLook at the distributions of your means

Stem and leaf graphsStem and leaf graphs By subjects…and by itemsBy subjects…and by items

Frequency Stem & Leaf

1.00 0 . 9 13.00 1 . 1112223334444 22.00 1 . 5555555666677777777788 9.00 2 . 000111344 1.00 2 . 7 2.00 Extremes (>=3120)

Stem width: 1000.00 Each leaf: 1 case(s)

Frequency Stem & Leaf

3.00 0 . 778 17.00 1 . 01111222333334444 15.00 1 . 556677788889999 11.00 2 . 00011122333 1.00 2 . 7 1.00 Extremes (>=3286)

Stem width: 1000.00 Each leaf: 1 case(s)

Maria asked Bob to invite Fred or Sam to the barbecue. She didn't have enough room to invite both.

Maria asked Bob not to invite Fred or Sam to the barbecue. She didn't have enough room to invite both.

Frequency Stem & Leaf

1.00 Extremes (=<950) 4.00 1 . 3334 14.00 1 . 56666777777888 4.00 2 . 1222 1.00 Extremes (>=2562)

Stem width: 1000.00 Each leaf: 1 case(s)

Frequency Stem & Leaf

7.00 1 . 1122234 12.00 1 . 556666667788 4.00 2 . 1123 1.00 Extremes (>=2629)

Stem width: 1000 Each leaf: 1 case(s)

Maria asked Bob to invite Fred or Sam to the barbecue. She didn't have enough room to invite both.

Maria asked Bob not to invite Fred or Sam to the barbecue. She didn't have enough room to invite both.

By items

Variation among itemsVariation among items► Treat items as a random sample from some Treat items as a random sample from some

population.population. Just like we treat subjects as a random sampleJust like we treat subjects as a random sample

► Then do statistical tests to generalize to this Then do statistical tests to generalize to this population of items.population of items. ““F1” and “F2”F1” and “F2”

► CriticismsCriticisms Should generalize simultaneously to subjects and items, Should generalize simultaneously to subjects and items,

using F’.using F’.► But must estimate F’ unless every you have data from every But must estimate F’ unless every you have data from every

subject on every condition of every item (min F’; Clark, 1973)subject on every condition of every item (min F’; Clark, 1973) We’re fooling ourselves when we view items as We’re fooling ourselves when we view items as

anything like a random sample from a population.anything like a random sample from a population.

Alternatives to F1 and F2Alternatives to F1 and F2

►Some conventional ANOVA designs do Some conventional ANOVA designs do permit generalization to subjects and permit generalization to subjects and items without full dataitems without full data But generally lack powerBut generally lack power

►Coming trend: multilevel, hierarchical Coming trend: multilevel, hierarchical designsdesigns Complex regression-based analyses of Complex regression-based analyses of

individual data points, not subject- or individual data points, not subject- or item-means.item-means.

But what if you recognize But what if you recognize that random sampling from a that random sampling from a population of items is nutty?population of items is nutty?

►What you really want is to show that What you really want is to show that your effects hold for most or all of your your effects hold for most or all of your items and aren’t due to a couple of items and aren’t due to a couple of oddballsoddballs F2 tests a crude attempt to do this.F2 tests a crude attempt to do this.

►People struggling to get a better way.People struggling to get a better way. One possibility, from Ken Forster: plot One possibility, from Ken Forster: plot

effect size vs effect rank, see if it is effect size vs effect rank, see if it is pleasingly regular.pleasingly regular.

Forster, “What is F2 good Forster, “What is F2 good for”for”► Plot effect size (difference between two conditions) Plot effect size (difference between two conditions)

against rank of effect size (suggested by Peter against rank of effect size (suggested by Peter Killeen)Killeen) Both cases: a 5 msec mean effect sizeBoth cases: a 5 msec mean effect size Left panel: a limited effect (add 100 ms to 5 items)Left panel: a limited effect (add 100 ms to 5 items) Right panel: a general effect (add 5 ms to 100 items)Right panel: a general effect (add 5 ms to 100 items)

R2 = 0.4491

-40

-20

0

20

40

60

80

100

120

0 10 20 30 40 50

RANK

EF

FE

CT

SIZ

E

R2 = 0.9502

-20

-10

0

10

20

0 10 20 30 40 50

RANK

EF

FE

CT

SIZ

E

Forster, K. (2007). What is F2 good for? Round 2. Unpublished ms, University of Arizona.

Bogartz, 2007Bogartz, 2007► Effect size vs. rank effect size, Clifton et al. JML 2003Effect size vs. rank effect size, Clifton et al. JML 2003

Effect of ambiguity (absence of relative pronoun) on Effect of ambiguity (absence of relative pronoun) on sentences with relative clauses (sentences with relative clauses (The man [who was] paid by The man [who was] paid by the parents was unreasonable)the parents was unreasonable)

Contrasted with Monte Carlo data based on same mean and Contrasted with Monte Carlo data based on same mean and variance as experimental datavariance as experimental data

Bogartz, R. (2007). Fixed vs. random effects, extrastatistical inference, and multilevel modeling. Unpublished manuscript, University of Massachusetts.

III. How to do experiments, III. How to do experiments, Part 2: Experimental Part 2: Experimental

proceduresprocedures► Acceptability judgmentAcceptability judgment► Interpretive choicesInterpretive choices► Stops making senseStops making sense► Self-paced readingSelf-paced reading► Eyetracking during readingEyetracking during reading► ERPERP► Secondary tasksSecondary tasks► Speed-accuracy tradeoff tasksSpeed-accuracy tradeoff tasks► Eyetracking during listening (“visual world”)Eyetracking during listening (“visual world”)

Choose task that is Choose task that is appropriate for your questionappropriate for your question

► Is this really a sentence of English?Is this really a sentence of English?► Does some variable affect how a sentence is Does some variable affect how a sentence is

understood?understood?► Is there some difficulty in understanding this Is there some difficulty in understanding this

sentence?sentence?► Just where in the sentence does the difficulty Just where in the sentence does the difficulty

appear?appear?► Where in processing does the difficulty appear?Where in processing does the difficulty appear?► Can we observe consequences of processing Can we observe consequences of processing

other that difficulty?other that difficulty?► and more….and more….

Acceptability judgmentAcceptability judgment

►Simple written questionnaireSimple written questionnaire See SchSee Schütze, Cowart for lots of examplesütze, Cowart for lots of examples Worry about instructionsWorry about instructions Rating scalesRating scales

►Is seven the magical number?Is seven the magical number?

►Magnitude estimationMagnitude estimation Basis in psychophysics – attempt to build Basis in psychophysics – attempt to build

an interval scalean interval scale

Magnitude estimation: an Magnitude estimation: an exampleexample

Which man did you wonder when to meet?

Assign an arbitrary number to that item, greater than zero.

Now, for each of the following items, assign a number. If the item is better than the first one, use a larger number; if it’s worse, smaller. Make the item proportional to how much better or worse the item is than the original – if twice as good, make the number 2x the start; if 1/3 as good, make the number 1/3 as big as the start.

Magnitude estimation : an Magnitude estimation : an exampleexample

► Which man did you wonder when to meet?Which man did you wonder when to meet? Assign an arbitrary number, greater than 0, to this first item.Assign an arbitrary number, greater than 0, to this first item. Now, for each successive item, assign a number – bigger if Now, for each successive item, assign a number – bigger if

the item is better, smaller if worse, and proportional – if the the item is better, smaller if worse, and proportional – if the item is 2x as good, make the number 2x the original; if ¼ as item is 2x as good, make the number 2x the original; if ¼ as good, make the number ¼ as big as the original.good, make the number ¼ as big as the original.

► Which book would you recommend reading?Which book would you recommend reading?► When do you know the man whom Mary invited?When do you know the man whom Mary invited?► This is a paper that we need someone who This is a paper that we need someone who

understands.understands.► With which pen do you wonder when to write.With which pen do you wonder when to write.► Who did Bill buy the car to please?Who did Bill buy the car to please?

Bard, E. G., Robertson, D., & Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 72.

On-line and web-based On-line and web-based questionnairesquestionnaires

►WebExp: WebExp: http://www.webexp.infohttp://www.webexp.info►Subject scheduling systems optionSubject scheduling systems option►Advantages: Big N, easy, broader Advantages: Big N, easy, broader

populationpopulation►Disadvantages: you have to worry Disadvantages: you have to worry

about controlabout control

Speeded acceptability Speeded acceptability judgmentjudgment

►Time pressure; discourage navel-Time pressure; discourage navel-examiningexamining

►Measure reaction time and Measure reaction time and acceptabilityacceptability

►Example: is given-new order more Example: is given-new order more acceptable than new-given?acceptable than new-given? Maybe so. Maybe not always.Maybe so. Maybe not always.

Given-New: DefNP-IndefNP

• All the players were watching an umpire. The pitcher threw the umpire a ball.

New-Given: IndefNP-DefNP

b. The catcher tossed a ball to the mound. The pitcher threw an umpire the ball.

Given-New: DefNP-IndefPP

c. The catcher tossed a ball to the mound. The pitcher threw the ball to an umpire.

New-Given: IndefNP-DefPP

d. All the players were watching an umpire. The pitcher threw a ball to the umpire.

2000

2400

2800

3200

3600

Reaction Time, ms

NP-NP NP-PP

Given-New New-Given

60

70

80

90

100

Percent Accepted

NP-NP NP-PP

Given-New New-Given

Clifton, C. J., & Frazier, L. (2004). Should given information come before new? Yes and no. Memory & Cognition, 32, 886-895.

Choice of interpretationChoice of interpretation

► Paper and pencil or speededPaper and pencil or speeded► Multiple-choice or paraphraseMultiple-choice or paraphrase► Example: interpretation of ellipsisExample: interpretation of ellipsis

Full stop effectFull stop effect► Auditory questionnaireAuditory questionnaire

Relative size of intonational phrase boundaryRelative size of intonational phrase boundary► Strengths: does indicate whether a variable has Strengths: does indicate whether a variable has

an effect or notan effect or not► Weaknesses: don’t know when the effect Weaknesses: don’t know when the effect

operatesoperates Worst case: subject says sentence to self, mulls it over, Worst case: subject says sentence to self, mulls it over,

reacts to the prosody s/he happened to imposereacts to the prosody s/he happened to impose

Example of interpretation Example of interpretation questionnaire: VPEquestionnaire: VPE

John said Fred went to Europe and Mary did too.

What did Mary do?

…went to Europe 60%

…said Fred went to Europe 40%

John said Fred went to Europe. Mary did too.

What did Mary do?

…went to Europe 45%

…said Fred went to Europe 55%

Frazier, L., & Clifton, C. Jr. (2005). The syntax-discourse divide: Processing ellipsis. Syntax, 8, 154-207.

Clifton, C. J., Carlson, K., & Frazier, L. (2002). Informative prosodic boundaries. Language and Speech, 45, 87-114.

Who arrived? Johnny and Sharon’sip inlaws. (0 ip)

Who arrived? Johnnyip and Sharon’sip inlaws (ip ip)

Who arrived? JohnnyIPh and Sharon’sip inlaws (IPh ip)

Alternative answers: Sharon’s inlaws and Johnny; Sharon’s and Johnny’s inlaws

Stops-making-sense taskStops-making-sense task

►Word-by-word, self-paced, but each Word-by-word, self-paced, but each word make one of two responses: OK, word make one of two responses: OK, BADBAD

►Get cumulative proportion of BAD Get cumulative proportion of BAD responses and OK RTresponses and OK RT

►Sensitive to point of difficulty in a Sensitive to point of difficulty in a sentencesentence

Example of SMSExample of SMS

Boland, J., Tanenhaus, M., Garnsey, S., & Carlson, G. (1995). Verb argument structure in parsing and interpretation: Evidence from wh-questions. Journal of Memory and Language, 34, 774-806.

Which client/prize did the salesman visit while in the city? (transitive)

Which child/movie did your brother remind to watch the show? (object control)

Stops-making-sense taskStops-making-sense task

►StrengthsStrengths Begins to address processing dynamics Begins to address processing dynamics

questionsquestions Can get both time and choice as relevant Can get both time and choice as relevant

datadata

►WeaknessesWeaknesses Very slow reading time – 500 to 900 ms/word Very slow reading time – 500 to 900 ms/word

typicallytypically Permits more analysis than is done in normal Permits more analysis than is done in normal

readingreading

Self-paced readingSelf-paced reading

►Word by word self-paced readingWord by word self-paced reading Generally noncumulativeGenerally noncumulative Sometimes in place (“RSVP”), sometimes Sometimes in place (“RSVP”), sometimes

moving across screenmoving across screen Time strongly affected by length of word, Time strongly affected by length of word,

frequency of wordfrequency of word►Can statistically adjustCan statistically adjust

►Variant: phrase by phrase self-paced Variant: phrase by phrase self-paced reading.reading.

SPR methodsSPR methods

►Computer programsComputer programs E-prime (E-prime (www.pstnet.comwww.pstnet.com)) Dmastr/DMDX (Dmastr/DMDX (

http://www.u.arizona.edu/~kforster/dmasthttp://www.u.arizona.edu/~kforster/dmastr/dmastr.htmr/dmastr.htm))

Others (PsyScope, Superlab, various Others (PsyScope, Superlab, various home-made systems)home-made systems)

SPR EvaluationSPR Evaluation

► Cheap and effectiveCheap and effective Don Mitchell, trailblazing techniqueDon Mitchell, trailblazing technique

► Slower than normal readingSlower than normal reading Perhaps 180 words per minute readingPerhaps 180 words per minute reading Unless reader clicks fast and buffers….Unless reader clicks fast and buffers….

► Often get effect on word following critical Often get effect on word following critical wordword SpilloverSpillover

► Phrase-by-phrase: overcomes these Phrase-by-phrase: overcomes these difficulties, but you lose precisiondifficulties, but you lose precision

Mitchell, D. C. (2004). On-line methods in language processing: Introduction and historical review. In M. Carreiras & C. J. Clifton (Eds.), The on-line study of sentence comprehension: Eyetracking, ERPs, and beyond. Brighton, UK: Psychology Press.

More SPR evaluationMore SPR evaluation

► Does SPR hide subtle detailsDoes SPR hide subtle details Maybe: Clifton, Speer, & Abney 1991 JML; SchMaybe: Clifton, Speer, & Abney 1991 JML; Schütze & ütze &

Gibson 1999 JMLGibson 1999 JML► Verb attachment: The man expressed his interest Verb attachment: The man expressed his interest in a hurryin a hurry

during the storewide sale… (VP adjunct)during the storewide sale… (VP adjunct)► NP attachment: The man expressed his interest NP attachment: The man expressed his interest in a walletin a wallet

during the storewide sale… (NP argument)during the storewide sale… (NP argument) Clifton et al: eyetracking, slow first-pass time in NP-Clifton et al: eyetracking, slow first-pass time in NP-

attached PP (followed by faster reading for argument attached PP (followed by faster reading for argument than adjunct)than adjunct)

SchSchütze & Gibson, word by word SPR, only the ütze & Gibson, word by word SPR, only the argument advantageargument advantage

► Better materialsBetter materials► Worse techniqueWorse technique

Even more SPR evaluationEven more SPR evaluation

► Does SPR introduce unnatural effects?Does SPR introduce unnatural effects? Maybe: Tabor, Galantucci, Richardson, 2004, Maybe: Tabor, Galantucci, Richardson, 2004,

local coherence effectslocal coherence effects The coach smiled at the player The coach smiled at the player tossed/thrown thetossed/thrown the

frisbee by the…frisbee by the…► Result: slowed reading at Result: slowed reading at tossedtossed as if reader as if reader

considering grammatically illegal main clause considering grammatically illegal main clause interpretation of “the player tossed the…”interpretation of “the player tossed the…”

But: scuttlebutt, may not show up in eyetrackingBut: scuttlebutt, may not show up in eyetracking► Global SPR reading speed, 412 ms/word, 145 wpmGlobal SPR reading speed, 412 ms/word, 145 wpm

Eyetracking during readingEyetracking during reading

►Eye movement measurementEye movement measurement Fixations and saccadesFixations and saccades Reading time affected by word length, Reading time affected by word length,

frequency, other lexical factorsfrequency, other lexical factors

Word-based measures of eye Word-based measures of eye movements (ms)movements (ms)

Most cowboys hate to live in houses so they 1 2 3 4 6 5 7

223 235 178 301 179 267 199

cowboys hate houses

SFD: 301 ms 267 ms

FFD: 235 ms 301 ms 267 ms

GAZE: 413 ms 301 ms 267 ms

Go-P: 413 ms 301 ms 436 ms

Region-based measuresRegion-based measures

While Mary/ was mending/ the sock/ fell off/ * * * * * * * * * 1 2 3 6 4 7 5 8 9 277 213 233 277 445 289 401 233 314

First pass: 510 ms 445 msSecond pass: 401 ms 547 msGo-Past: 510 ms 1393 msTotal Time: 911 ms 992 ms

Interpretation of the Interpretation of the measuresmeasures

► ““Early” vs. “late” measuresEarly” vs. “late” measures Debates about modularityDebates about modularity Some measures clearly late – second pass timeSome measures clearly late – second pass time But early: need explicit model of eye movement But early: need explicit model of eye movement

controlcontrol

► Rayner, Pollatsek, Reichle, colleagues – EZ Rayner, Pollatsek, Reichle, colleagues – EZ ReaderReader Good model of lexical effectsGood model of lexical effects Says little or nothing about parsing & Says little or nothing about parsing &

intepretationintepretation

ERP (event-related ERP (event-related potentials)potentials)

►Measure electrical activity on scalpMeasure electrical activity on scalp Reflect electrical activity of bundles of Reflect electrical activity of bundles of

cortical neuronscortical neurons Good time resolution, questionable spatial Good time resolution, questionable spatial

resolutionresolution

►Standard effects: LAN, N400, P600Standard effects: LAN, N400, P600 Typical peak time, polarityTypical peak time, polarity

““Standard” ERP effects Standard” ERP effects (Osterhout, 2004)(Osterhout, 2004)

The cat will EATThe cat will BAKE

N400

The cat will EAT

*The cat will EATING

P600

Osterhout, L. et al. (2004). Sentences in the brain…. In M. Carreiras and C. Clifton, Jr., The on-line study of sentence comprehension. New York: Psychology Press, pp 271-308.

Secondary tasks: Load Secondary tasks: Load effectseffects

► Limited capacity modelsLimited capacity models► Desire: measure of auditory processing Desire: measure of auditory processing

difficultydifficulty► Phoneme monitoringPhoneme monitoring

Eg: Cutler & Fodor, 1979Eg: Cutler & Fodor, 1979► Which man was wearing the hat? The man on the Which man was wearing the hat? The man on the cornercorner

was wearing the blue hat.was wearing the blue hat.► Which hat was the man wearing? The man on the corner Which hat was the man wearing? The man on the corner

was wearing the was wearing the blueblue hat. hat.► Target: /k/ or /b/; when target started focused word, 360 Target: /k/ or /b/; when target started focused word, 360

ms; when started non-focused word, 403 ms.ms; when started non-focused word, 403 ms.

Interpretive difficultiesInterpretive difficulties

Secondary tasks: Load Secondary tasks: Load effects IIeffects II

►Lexical decision (or naming, or Lexical decision (or naming, or semantic decision, or….)semantic decision, or….) Word unrelated to sentence; measure of Word unrelated to sentence; measure of

available capacityavailable capacity PiPiñango et al., auditory presentation, ñango et al., auditory presentation,

visual probevisual probe►The man examined the little bundle of fur for a The man examined the little bundle of fur for a

long time long time aspectaspect to see if it was… 743 ms to see if it was… 743 ms►The man kicked the little bundle of fur for a The man kicked the little bundle of fur for a

long time long time aspectaspect to see if it was… 782 ms to see if it was… 782 ms

Pinango, M. M., Zurif, E., & Jackendoff, R. (1999). Real-time processing implications of enriched composition at the syntax-semantics interface. Journal of Psycholinguistic Research, 28, 395-414.

Secondary tasks: Probe for Secondary tasks: Probe for activationactivation

►Auditory (or visual) presentationAuditory (or visual) presentation Probe semantically related to word in Probe semantically related to word in

sentence whose activiation you want to sentence whose activiation you want to measuremeasure

►E.g., activation of “filler” at “gap” in E.g., activation of “filler” at “gap” in long-distance dependencylong-distance dependency The policeman saw the boy who the crowd The policeman saw the boy who the crowd

at the partyat the party11 accused accused22 of the crime. of the crime.►Present probe Present probe girlgirl or matched unrelated word at or matched unrelated word at

point 1 or 2; point 1 or 2; girlgirl faster at 2. faster at 2. Worries, criticisms…Worries, criticisms…

Nicol, J., Swinney, D., Love, T., & Hald, L. (2006). The on-line study of sentence comprehension: An examination of dual task paradigms. Journal of Psycholinguistic Research, 35, 215-231.

Speed-accuracy tradeoffSpeed-accuracy tradeoff

► Present sentence (usually RSVP), subject to Present sentence (usually RSVP), subject to make judgment (grammaticality, etc.)make judgment (grammaticality, etc.)

► But judgment is made in response to a signal But judgment is made in response to a signal that is presented some time after a critical that is presented some time after a critical point.point.

► Accuracy increases with time after the critical Accuracy increases with time after the critical point.point.

► Note, current procedure, multiple signals and Note, current procedure, multiple signals and multiple responses, e.g., every 350 msmultiple responses, e.g., every 350 ms Early procedure: just one signal, one response, per Early procedure: just one signal, one response, per

trialtrial

McElree, B., Pylkkanen, L., Pickering, M., & Traxler, M. (2006). A time course analysis of enriched composition. Psychonomic Bulletin & Review, 13, 53-59.

McElree et al. dataMcElree et al. dataBest fit: coercion lowered asymptote and lowered rate of approach to asymptote.

Visual World (“head-mounted Visual World (“head-mounted eyetracking”)eyetracking”)

►Measure where you look when you are Measure where you look when you are listening to speech.listening to speech. Cooper, 1974. About 40% probability of Cooper, 1974. About 40% probability of

fixating on referent, 30% fixating on related fixating on referent, 30% fixating on related picturepicture

►About 10% in control group.About 10% in control group.

►Permits on-line measure of processing Permits on-line measure of processing during listening.during listening. Not just difficulty – actual contentNot just difficulty – actual content Both incidental looks and controlled reachingBoth incidental looks and controlled reaching

Cooper, R. M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84-107.

Cooper, 1974Cooper, 1974

While on a photographic safari in Africa, I managed to get a number of breathtaking shots of the wild terrain. These included pictures of rugged mountains and forests as well as muddy streams winding their way through big game country. One of my best shots thought was ruined by my scatterbrained dog Scotty. Just as I had slowly wormed my way on my stomach to within range of a flock….

Eye camera

Scene camera

Allopenna, Magnuson & Tanenhaus Allopenna, Magnuson & Tanenhaus (1998)(1998)

Pick up the beaker

Allopenna, P. D., Magnuson, J. S., & Tanenhaus, M. K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38, 419-439.

Allopenna et al. ResultsAllopenna et al. Results

200 ms after coarticulatory information in vowel

Thanks! Enjoy the Institute!