new creating a new communication system : gesture has the upper … a new... · 2015. 11. 24. · :...

Powerpoint Templates


Creating a New Communication System

: Gesture has the Upper Hand

인지과학 협동과정

방효석

http://www.powerpointstyles.com/



Table of Contents

Backgrounds

Overview of Present Research

Experiment 1

Experiment 2

Discussion




Backgrounds




Backgrounds

Previous Research

: Creating a communication system from scratch: gesture beats vocalization

hands down (Fay et al., 2014)

<Research Question>

Does the modality of communication drives the creation of shared

inventory of sign-meaning mapping?



Backgrounds

Previous Research

: Creating a communication system from scratch: gesture beats vocalization

hands down (Fay et al., 2014)

<Hypothesis>

1. Communication success will be higher for gesture than for non-

linguistic vocalization.

2. Communication success will be higher in the combined modality

compared to gesture-alone.

3. There will be greater alignment in the gestural modality than in

the vocalization modality.



1. Language Origin

• Proto-speech account

: Rudimentary vocalization came to have communicative meaning

• Proto-sign account

: Language evolved first from manual gestures, before shifting to the

vocal modality

• Multimodal account

: The earliest forms of language were not restricted to a single modality

Basic Concepts

Backgrounds



Backgrounds

Theories of Origin of

Languages

1. Proto-sign account

2. Proto-speech account

3. Multimodal account

Experimental Design

1. Gesture-only condition

2. Non-linguistic vocalization

condition

3. Multimodal condition



2. Types of Signs(Pierece’s triadic model of sign)

1) Iconic: Signs where the signifier resembles the signified(e.g., a

portrait, a cartoon, sound effects, or a statue)

2) Indexical: Signs where the signifier is caused by the signified(e.g.,

smoke signifies fire)

3) Symbolic: The signifier is assigned arbitrarily or is accepted as

societal convention. The relationship must be learned (i.e. alphabet,

mathemetical signs, computer codes)

Basic Concepts

Backgrounds



3. Referential Communication Task

Referential communication?

: Communication occured by making reference

Ex) You can refer one of the cars on the right,

by using

• names (the Corvette)

• noun modifiers, like adjectives

(the light green car)

• prepositional phrases

(the car in the upper right-hand corner)

• relative clauses

(the car that you just put on the shelf)

Basic Concepts

Backgrounds



4. Measures

1) Effectiveness: measures how successful the signs were at identifying

their referent (communication success)

2) Alignment: : measures the similarity between the sign a participant

produced, and the sign their partner produced on the previous game

when communicating the same concept (basis of shared inventory)

Basic Concepts

Backgrounds



5. Motivated signs & Arbitrary signs

1) Motivated signs: Signs that are linked to meaning by structural

resemblance or by natural association (i.e. Iconic/indexical signs)

2) Arbitrary signs: Signs where the relation between signifier and signified

is purely conventional and culturally specific (i.e. Symbolic signs; name,

language...)

Basic Concepts

Backgrounds




Research Overview




Research Overview

Purpose of the present research

1. Limitation of previous referential task studies

: lack of the number of concepts used

2. Provide the comparison between the vocal and gestural

modalities to figure out the differences of affordances of sign

motivations of gesture and non-linguistic vocalization




Experiment 1




Experiment 1

Hypothesis

1. Sign motivation would be higher for gestured signs than for

signs produced using non-linguistic vocalization

2. Communication success would be higher for gesture than for

non-linguistic vocalization

3. Alignment would be higher in the gesture-only condition than

in the vocal-only condition



Experiment 1

Method

1)Participants:

106 undergraduate students(63 females)

Placed into unacquainted pairs

All were free from auditory, visual, speech and motor impairment

2)Materials:

18 target concepts were sampled without replacement

(Adjective, Noun, Verb)

Presented with six distractor concepts

A different set of concepts were used by each pair



Experiment 1

Method

3) Task & Procedure:

• 2 task conditions (gesture/vocal) with 6 games in each condition

• Pairs 2 participants (matcher/director)

matcher: Select the same concept more than once within the

same game

director: Produce as many gestures and vocalizations as they

wished for each concepts

• Counterbalancing of modality across participants

gesture-only: face one another

vocal-only: face away from each other(eliminate gesture effects)



Experiment 1

Result & Discussion

Three-way ANOVA (modality, game, concept)

1) Motivation

• 7-point likert scale: 0 (entirely symbolic) to 6 (highly motivated, either

iconic or indexical)

• Gesture outperformed vocalization [F(1, 52) = 700.33, p < .001]

• Increase in sign motivation across games 1-6 in both conditions [F(5,

260) = 39.89, p < .001]



Experiment 1

Result & Discussion

2) Communication Success



Experiment 1

Result & Discussion

3) Alignment

• 7-point likert scale, 0 (did not copy partner’s sign) to 6 (nearly identical

copy of partner’s sign)

• Gesture showed higher alignment than vocalization(p<.001)

• Alignment increased in both modalities across games(p<.001)



Experiment 1

Result & Discussion

4) Limitation

• Concurrent feedback effect in gesture-only condition

: Participants in gesture-only condition may have and advantage

seeing their partner’s facial expressions of confusion or

comprehension

→ Experiment 2 addresses this issue




Experiment 2




Experiment 2

Method

1)Participants:

60 undergraduate students(42 females)

2)Materials:

Same corpus of concepts used in experiment 1 was used

Fewer concepts were sampled(540)

3) Task & Procedure:

Participants communicated signs to a video camera

Only 2 games were played



Experiment 2

Result & Discussion

1) Motivation



Experiment 2

Result & Discussion

2) Communication Success

• Gestured signs were communicated more successfully than vocal

signs(p<.001)

3) Alignment

• Alignment was higher in gesture-only condition[F(1, 29)=283.41,

p<.001]



Experiment 2

Result & Discussion

• Experiment 2 replicates the pattern of results observed in experiment 1

: Indicates that the benefit of gesture over non-linguistic vocalization is

due to the modality itself, rather than concurrent feedback



Experiment 2

Result & Discussion

Based on interaction effect of 3 variables:




Discussion




Discussion

If gesture is better than non-linguisitc vocalization in terms of sign

motivation, why do we use conventional spoken language instead of

conventional sign language?

There are several studies showing evidence that people also

spontaneously produce motivated vocal language in their speech. If

both gestural and vocal language have same effect of motivation,

which of these do you think the origin of conventionalized language?


new creating a new communication system : gesture has the upper … a new... · 2015. 11. 24. · :...

Documents