Understanding Action Verbs- Embodied Verbal Semantics

Approach

Pavan Kumar SrungaramM.Phil Cognitive Science (09CCHL02)

2009-10Supervisor: Prof. Bapi Raju, S.

University of Hyderabad

Language in Two Different Perspectives• Chomskyan view: • Based on structures and rules. • Syntax is autonomous free of

meaning and general cognition• Human language is an abstract

formal system

Problems

• In many cases it was observed that background knowledge, concept, meaning have come into the rules governing syntax.

Language as a part of General Cognition• Lakoff, Fauconnier – Language not to

be studied in terms of structure/ Grammar but the way it is used

• Language and thought are embodied, concepts arise from our sensorimotor experience and the neural substrates that give rise to it.

• Metaphors We Live By !

Embodiment

• Acknowledge the role of body and its sensorimotor processes can and do play in cognition.

• Understanding cognition in the context of biological function to support the activities of the body.

• Cognition as real-time situated activity, inseparable from perception and action.

Behavioural Tests

• Considered the experiments by Bergen et al. (2010) and tried to replicate three of those tests in Telugu.

• In addition to these tests, we tried to verify whether the results produced in those experiments were modality specific

Behavioural Experiment

• There are 4 tasks in this experiment• Each subject participates only in one

task• Task 1: Image-Verb Matching Task• Task 2: Verb-Image Matching Task• Task 3: Verb-Verb Matching Task• Task 4: Image-Verb (auditory)

Matching Task

Hypothesis

• If verb comprehension requires the activation of the motor areas related to the same effector with which the action is performed.

• Windhorst Hypothesis: Motor and perceptual systems display lateral inhibitions among neural structures responsible for related but incompatible functions

Materials and Methods

• Computer Based Test • Two Stimuli presented at specific

times• Subjects are to judge as quickly as

possible whether the two stimuli match

• The response times of the subjects are recorded and analyzed

Materials and Methods

• The stimuli used in the tasks are: a) Images: These contain a single

character performing some action b) Action Verbs written (or) typed both

in program compatible formats. c) Action Verbs (audio) • All these three are recorded and used

in program compatible formats.

Algorithm

• First Stimuli: Presented for 1000 milliseconds

• Inter stimulus interval: 500 milliseconds a) 450 milliseconds: Visual Mask

b) 50 milliseconds: Blank• Second Stimuli presentation: This is

until the subject makes his decision by pressing the designated key.

Presentation of Stimuli

Presentation of Stimuli

Task 1: Image-Verb Matching Task• Image is shown for a period of 1000

milliseconds

• 500 milliseconds inter stimulus interval

• This is followed by a verb in typed format

• Show images and verbs!

Results

• The observation is in accordance with the hypothesis.

• Subjects took longer time to reject the image-verb pairs of non-matching same effector condition compared to those in non-matching different effector condition.

Means Table

ConditionAverage RT in milliseconds Standard Deviation

Matching Condition 1281.34 444.4169

Non-Matching Same effector Condition 1502.599 431.3717

Non-Matching Different effector Condition 1394.467 386.6818

ANOVA: Analysis of Variance

RT

Sum of Squares df Mean Square F Sig.

Between Groups 1175122.403 2 587561.202 3.306 .040

Within Groups 2.506E7 141 177703.579

Total 2.623E7 143

Condition 1: Matching ConditionCondition 2: Non-Matching Same effector ConditionCondition 3: Non-Matching Different effector Condition

Results

• The results obtained in the test provide evidence that understanding action verbs, language users recruit neural resources that are normally used for performing an action. This also supports the claim that understanding language might require simulation of action.

Result of Image-Verb Matching Task

Test Statisticsa,b

RT

Chi-Square 7.939

df 2

Asymp. Sig. .019

a. Kruskal Wallis Test

b. Grouping Variable: Condition

Ranks

Condition N Mean Rank

RT 1.00 48 59.75

2.00 48 83.56

3.00 48 74.19

Total 144

Alternative explanation

• An alternative explanation for the observed effect is that seeing images might have led subjects to activate representations of other actions using the same effector and that the verb presented might be the actions related to those activated actions that might have resulted in delay in rejection

Task 2:Verb-Image Matching Task• Verb in typed format is shown for a

period of 1000 milliseconds

• 500 milliseconds inter stimulus interval

• This is followed by an image

Means Table

Condition

Average RT (in

milliseconds) Standard Deviation

Matching Condition 1020.568 374.1628

Non-Matching Same

Effector Condition1176.36 492.6629

Non-Matching Different

Effector Condition1111.11 529.8168

Condition 1: Matching ConditionCondition 2: Non-Matching Same effector ConditionCondition 3: Non-Matching Different effector Condition

ANOVA

RTSum of Squares df Mean Square F Sig.

Between Groups 612150.829 2 306075.415 1.384 .254

Within Groups 3.251E7 147 221140.118

Total 3.312E7 149

Result of Verb Image Matching Task

Ranks

Condition N Mean Rank

RT 1.00 50 64.60

2.00 50 87.20

3.00 50 74.70

Total 150

Test Statisticsa,b

RT

Chi-Square 6.790

df 2

Asymp. Sig. .034

a. Kruskal Wallis Test

b. Grouping Variable: Condition

Results

• As observed in task, the subjects took longer time to respond when the verb and image share the same effector.

• Thus, to some extent it is evident that the results obtained in the test could not be due to the proposed alternative explanation for test 1 since the same effect is observed even when the order is reversed.

Task 3: Verb-Verb Matching Task• Verb in typed format is shown for a

period of 1000 milliseconds

• 500 milliseconds inter stimulus interval

• This is followed by a verb in typed format

Means Table

ConditionAverage RT (in

milliseconds)Standard Deviation

Matching Condition 1353.48 351.0843

Non-Matching Same

Effector Condition1419.04 396.6966

Non-Matching Different

Effector Condition1247.92 276.2357

ANOVA

RT Sum of

Squaresdf Mean Square F Sig.

Between

Groups795980.991 2 397990.496 3.351 .038

Within Groups 1.889E7 159 118782.609

Total 1.968E7 161

Condition 1: Matching ConditionCondition 2: Non-Matching Same effector ConditionCondition 3: Non-Matching Different effector Condition

Result of Verb-Verb Matching Task

Ranks

Condition N Mean Rank

RT 1.00 54 83.48

2.00 54 91.89

3.00 54 69.13

Total 162

Test Statisticsa,b

RT

Chi-Square 6.500

df 2

Asymp. Sig. .039

a. Kruskal Wallis Test

b. Grouping Variable: Condition

Results

• When the images were replaced by verbs and the task became a lexical-lexical matching task, and still the effect, as hypothesized, was observed

Question

• In all the above tests, the presentation of stimuli was through visual modality. This might raise a question whether the motor area activation is only due to presentation of stimuli in visual modality. To verify this, a study was conducted in which the images were displayed on the screen and the verbs are presented in auditory modality

Task 4: Image-Verb (Auditory) Matching Task• Image is shown for a period of 1000

milliseconds

• 500 milliseconds inter stimulus interval

• This is followed by a verb in auditory mode.

Means Table

Condition Average RT (in

milliseconds)

Standard Deviation

Matching Condition 438.85 138.5288

Non-Matching Same 508.95 219.089

Non-Matching Different 440.08 144.3107

ANOVA

RT Sum of

Squares df Mean Square F Sig.

Between

Groups128683.860 2 64341.930 2.213 .114

Within Groups 3575614.997 123 29070.041

Total 3704298.858 125

Condition 1: Matching ConditionCondition 2: Non-Matching Same effector ConditionCondition 3: Non-Matching Different effector Condition

Result of Image-Verb (auditory) Matching Task

Ranks

Condition N Mean Rank

RT

1.00 42 59.52

2.00 42 70.75

3.00 42 60.23

Total 126

Test Statisticsa,b

RT

Chi-Square 2.491

df 2

Asymp. Sig. .288

a. Kruskal Wallis Test

b. Grouping Variable: Condition

Result

• It is evident from the table that RT in the non-matching same effector condition is greater than those in non-matching different effector condition but it is found to be statistically insignificant.

.

Summary

Condition Task 1 Task 2 Task 3 Task 4

Matching Condition 1281.34 1020.568 1353.48 438.85

Non-Matching same effector 1502.599

1176.36 1419.04 508.95

Non-Matching different effector 1394.467

1111.11 1247.92 440.08

Summary

• We observe that irrespective of statistical significance, the response times to reject pairs in non-matching same effector condition is larger than those in non-matching different effector condition.

SummaryTask TI: Picture-

Lexical Matching Task

T2: Lexical-Picture Matching Task

T3: Lexical-Lexical Matching Task

T4: Picture-Lexical (Auditory) Matching Task

Matching condition

7.45% 9.4% 11% 7.90%

Non matching same effector condition

20.90% 19.55% 18% 20.17%

Non matching different effector condition

6.12% 8.6% 7.33% 6.29%

Average error responses per Subject.

11.49% 9.71% 12.11% 11.45%

Summary

• The higher rate of errors in the non-matching same effector conditions in all the tasks is to be tested and given an explanation.

Conclusion

• The results obtained in the various tasks though may not completely support the claims of embodied semantics, can also not disprove its claims. This study shows that the notion of embodied verbal semantics is to be carefully investigated in other languages to verify the universality of the effect.

Thank you