from syllables to syntax: investigating staged linguistic development through computational...

From Syllables to Syntax:Investigating Staged Linguistic Development through

Computational Modelling

Kris Jack, Chris Reed, and Annalu Waller[kjack|chris|awaller]@computing.dundee.ac.uk

Applied Computing, University of Dundee,

Dundee, DD1 4HN, Scotland

Staged Language Acquisition

Pre-linguistic StageHolophrastic

StageEarly Multi-word Stage Late Multi-word Stage Abstract Stage

0 months 6 months 12 months 18 months 24 months 30 months 36 months 42 months

• Language acquisition is consistently described in stages• Lexical and syntactic acquisition strategies must operate

within a unified model• The Model

– Training Data– Initial Assumptions– Lexical Acquisition– Syntactic Acquisition– Comprehension

• Results

Lexical Acquisition

• Siskind

• Steels

• Regier Siskind (1996)

• Cross-situational analysis– Relationship between the appearance

and words and their referents

Lexical Acquisition

• Siskind

• Steels

• Regier Steels (2001)

• Language games– Social pressure to communicate

within a community of agents can lead to an emergent and shared vocabulary

Lexical Acquisition

• Siskind

• Steels

• Regier Regier (2005)

• Associative learning– Fast-mapping– Shape bias

• No mechanistic changes– Selective attention

Syntactic Acquisition

• Roy

• Elman

• Kirby Roy (2002)

• Trained a grounded robot to play a ‘show-and-tell’ game– Training data were divided into

simple and complex descriptions

Data Model

0 > t > x

Data

Data

x > t > y

y > t > z


• Roy

• Elman

• Kirby Elman (1993)

• Incremental Learning– Mechanistic changes can lead to

changes in behaviour

Data

Module

Module

Module

Model

t > 0

t > x

t > y


• Roy

• Elman

• Kirby Kirby (2002)

• Iterated Learning– Languages with increasing

complexity can emerge across generations of agents

Model Model

Data

Model

Data

Question

Can we develop a unified model that performs staged language acquisition where:

1. The learning mechanisms are constant AND

2. Exposure to training data is constant?

Bridging the Gapbetween Words and Syntax

• Jack, Reed, and Waller (2004)– Shift from holophrastic to syntactic language– The shift was unrealistic as it appeared very early

• A form of substitution was employed (similar to Harris (1966); Wolff (1988); Kirby (2002); van Zaanen (2002))

• If the model encountered A B and A C then B and C were considered substitutable for one another

– Given the two rules:» S/eats(john, cake) → johneatscake» S/eats(mary, cake) → maryeatscake

– Three rules were derived:» S/eats(x, cake) → N/x eatscake» N/john → john» N/mary → mary

• This is a reasonable, yet powerful, form of syntactic learning– The target language was unrealistically simple (two-word sentences)

Training Data

• Played the Scene Building Game– Based on the Miniature Language Acquisition

Problem (Feldman et al., 1990)

– Aim; describe a visual event so that someone else can recreate the event based on the description

→ t = 1 t = 2

→ → t = 3 t = 4

Training Data




→ t = 1 t = 2

→ → t = 3 t = 4

a red square has appeared

Training Data




→ t = 1 t = 2

→ → t = 3 t = 4

a pink cross to the upper right of the

red circle

Training Data




→ t = 1 t = 2

→ → t = 3 t = 4

a blue cross on the other side of

the red circle

Training Data




→ t = 1 t = 2

→ → t = 3 t = 4

another red circle under the pink

cross

Initial Assumptions

• Joint attention is established at around one-year-old (Tomasello, 1995)

• Receives <event, description> pairs– An event is a set of six feature tuples

– A description is a string

→

{<red, (1)>, <circle, (1)>, <pink, (2)>, <cross, (2)>, <above, (0)>, <right, (0)>}

“a pink cross to the u pper right of the red cir cle”t = 1 t = 2

Initial Assumptions

• Sensitivity to data– Children can identify objects through displacement during

motion (Kellman et al., 1987). – Children can use shape and colour to differentiate between

objects (e.g. Landau et al., 1988)

→



Initial Assumptions

• Sensitivity to data– Children show sensitivity to the relative spatial

relationships between objects, making distinctions between left and right, and above and below (Quinn, 2003)

→



Initial Assumptions

• Sensitivity to data– Children can perform analogies (Gentner and Medina,

1998)

→



Initial Assumptions

• Sensitivity to data– Children can determine transitional probabilities between

syllables (Saffran, Aslin, and Newport, 1996)

→



The Model

• Training the model– The Lexical Analysis Unit

• Discovers string-meaning associations

– The Syntactic Analysis Unit• Discovers compositional relationships

The Lexical Analysis Unit

<event, description> pairs are compared through a form of cross-situational analysis


“a pink cross to the u pper right of the red cir cle”

{<green, (1)>, <circle, (1)>, <red, (2)>, <diamond, (2)>, <even_vertical, (0)>, <right, (0)>}

“a red dia mond to the right of the green cir cle”

<event, description>#1



Feature tuple comparisons are value sensitive and object identifier insensitive. Two feature tuples, <v1, (o1)> and <v2, (o2)>, are equivalent iff v1 = v2








Co-occurring syllable sequences are found








New <feature tuple set, description> pairs are derived







<{<red, (1)>, <circle, (1)>, <right, (0)>}, “a”>

<{<red, (1)>, <circle, (1)>, <right, (0)>}, “to the”>

<{<red, (1)>, <circle, (1)>, <right, (0)>}, “right of the”>

<{<red, (1)>, <circle, (1)>, <right, (0)>}, “red”>

<{<red, (1)>, <circle, (1)>, <right, (0)>}, “cir cle”>

<{<circle, (1)>, <red, (2)>, <right, (0)>}, “a”>

<{<circle, (1)>, <red, (2)>, <right, (0)>}, “red”>

<{<circle, (1)>, <red, (2)>, <right, (0)>}, “to the”>

<{<circle, (1)>, <red, (2)>, <right, (0)>}, “right of the”>

<{<circle, (1)>, <red, (2)>, <right, (0)>}, “cir cle”>


• Cross-situational analysis can produce pairs that share the same strings (homonyms) or the same feature sets (synonyms)

• Homonyms and synonyms are removed, following a principle of mutual exclusivity (Markman and Wachtel, 1988)

• When pairs are equal, with insensitivity to object identifiers, they are merged. Merging produces a new pair, that expresses both of the relationships<{<red, (1)>}, “red”> is merged with<{<red, (2)>}, “red”> to produce

<{<red, (1, 2)>}, “red”>


• From all merged pairs, homonyms are removed by selecting the most probable feature set for each string,

where Frequency of (Sj | Fi) is the number of times that Sj has been observed with Fi and the Frequency of Sj is the number of times that Sj has been observed

• Then synonyms are removed by selecting the most probable string for each feature set, P(Sj | Fi), and erasing the remaining pair’s feature sets

• A set of lexical items are derived

j

ijji

S

FSSFP

ofFrequency

ofFrequency )|()|( =

The Syntactic Analysis Unit

• Compositional relationships are found by combining and comparing lexical items

• Lexical items are combined by set union and string concatenation

• The lexical item triple <<f1, s1>, <f2, s2>, <f3, s3>> expresses a compositional relationship iff

<f1, s1> = <f2, s2> combined with <f3, s3>

21212211 with combined ssffsfsf += ,,,


A lexical item triple can be made to express a rule by:

1. Converting lexical items into phrasal categories

2. Constructing transformations





<<{<red, (1, 2)>, <square, (1, 2)>}, “red square”>,

<{<red, (1, 2)>}, “red”>, <{<square, (1, 2)>}, “square”>>







<{<red, (1, 2)>, <square, (1, 2)>}, “red square”>

<{<red, (1, 2)>}, “red”> <{<square, (1, 2)>}, “square”>







<{<red, (1, 2)>, <square, (1, 2)>}, “red square”>

<{<red, (1, 2)>}, “red”> <{<square, (1, 2)>}, “square”>

(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)


Rules that modify object identifiers can be constructed

<{<red, (2)>}, “a red”>

<{}, “a”> <{<red, (1, 2)>}, “red”>

() (<(1, 2) → (2)>)

<{<blue, (1)>}, “the blue”>

<{}, “the”> <{<blue, (1, 2)>}, “blue”>

() (<(1, 2) → (1)>)


Rules can be merged when they share transformations <{<red, (1, 2)>, <circle, (1, 2)>}, “red cir cle”>

<{<red, (1, 2)>}, “red”> <{<circle, (1, 2)>}, “cir cle”>

(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)<{<blue, (1, 2)>, <circle, (1, 2)>}, “blue cir cle”>

<{<blue, (1, 2)>}, “blue”> <{<circle, (1, 2)>}, “cir cle”>

(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

<{<pink, (1, 2)>, <diamond, (1, 2)>}, “pink dia mond”>

<{<pink, (1, 2)>}, “pink”> <{<diamond, (1, 2)>}, “dia mond”>

(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)


Rules can be merged when they share transformations <{<red, (1, 2)>, <circle, (1, 2)>}, “red cir cle”>

<{<red, (1, 2)>}, “red”> <{<circle, (1, 2)>}, “cir cle”>

(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)<{<blue, (1, 2)>, <circle, (1, 2)>}, “blue cir cle”>

<{<blue, (1, 2)>}, “blue”> <{<circle, (1, 2)>}, “cir cle”>

(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

<{<pink, (1, 2)>, <diamond, (1, 2)>}, “pink dia mond”>

<{<pink, (1, 2)>}, “pink”> <{<diamond, (1, 2)>}, “dia mond”>

(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>){<{<red, (1, 2)>, <circle, (1, 2)>}, “red cir cle”>, <{<blue, (1, 2)>, <circle, (1, 2)>}, “blue cir cle”>, <{<pink, (1, 2)>, <diamond, (1, 2)>}, “pink dia mond”>}

{<{<red, (1, 2)>}, “red”>, <{<blue, (1, 2)>}, “blue”>, <{<pink, (1, 2)>}, “pink”>}

{<{<circle, (1, 2)>}, “cir cle”>, <{<diamond, (1, 2)>}, “dia mond”>}

(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

Comprehension

• The model is tested for evidence of language acquisition through comprehension tasks

• The model can comprehend a string by:– Finding it in a phrasal category (lexical)– Or creating it through applying a rule (syntactic)

Comprehension

• Example 1. Comprehension of “cir cle”– Find “cir cle” in a phrasal category

– Attempt to create “cir cle” by applying a rule

{<{<red, (1, 2)>, <circle, (1, 2)>}, “red cir cle”>, <{<blue, (1, 2)>, <circle, (1, 2)>}, “blue cir cle”>, <{<pink, (1, 2)>, <diamond, (1, 2)>}, “pink dia mond”>}



(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

Comprehension






(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

Meaning is {<circle, (1, 2)>}

Comprehension






(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

Comprehension


– Attempt to create “cir cle” by applying a rule• Find “cir” and “cle” in phrasal categories of a rule




(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

No meaning found

Comprehension






(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

Meaning is found lexically as {<circle, (1, 2)>}

Comprehension

• Example 2. Comprehension of “red dia mond”– Find “red dia mond” in a phrasal category– Attempt to create “red dia mond” by applying a rule




(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

Comprehension





(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

No meaning found

Comprehension





(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

Comprehension


• Find “red” and “dia mond” in phrasal categories of a rule

• Find “red dia” and “mond” in phrasal categories of a rule




(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

{<red, (1, 2)>}, transformed by (<(1, 2)> → (1, 2)>), and combined with{<diamond, (1, 2)>}, transformed by (<(1, 2)> → (1, 2)>), gives {<red, (1, 2)>, <diamond, (1, 2)>}

Comprehension


• Find “red” and “dia mond” in phrasal categories of a rule

• Find “red dia” and “mond” in phrasal categories of a rule




(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

No meaning found

Comprehension





(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

Meaning is found syntactically as {<red, (1, 2)>, <diamond, (1, 2)>}

Comprehension

• Phrasal categories are substitutable for one another if they share a subset relationship




(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)

{<{<red, (2)>}, “a red”>, <{<blue, (2)>}, “a blue”>, <{<pink, (2)>}, “a pink”>, <{<green, (2)>}, “a green”>, <{<white, (2)>}, “a white”>}

{<{}, “a”>} {<{<red, (1, 2)>}, “red”>, <{<blue, (1, 2)>}, “blue”>, <{<pink, (1, 2)>}, “pink”>, <{<green, (1, 2)>}, “green”>, <{<white, (1, 2)>}, “white”>}

() (<(1, 2) → (2)>)

{<{<pink, (1)>}, “the pink”>, <{<green, (1)>}, “the green”>, <{<white, (1)>}, “the white”>}

{<{}, “the”>} {<{<pink, (1, 2)>}, “pink”>, <{<green, (1, 2)>}, “green”>, <{<white, (1, 2)>}, “white”>}

() (<(1, 2) → (1)>)

Comprehension





(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)



() (<(1, 2) → (2)>)

Construction Islands

Comprehension




(<(1, 2) → (1, 2)>) (<(1, 2) → (1, 2)>)



() (<(1, 2) → (2)>)

{<{<red, (1, 2)>}, “red”>, <{<blue, (1, 2)>}, “blue”>, <{<pink, (1, 2)>}, “pink”>, <{<green, (1, 2)>}, “green”>, <{<white, (1, 2)>}, “white”>}

Results

• Observing the developmental shift from lexical to syntactic comprehension– Tested for comprehension of colours (10), shapes (10), and colour shape combinations

(100) during training. Results are averaged over 10 sessions.

Developmental Shift

0

5

10

15

20

25

0 3 6 9 12 15 18 21 24 27 30

No. <event, description>s entered

No

. s

trin

gs

co

mp

reh

en

de

d

8

Lexical

Syntactic

3 P

re-lin

gu

istic

3 H

olo

ph

ras

tic

3 E

arly

3 M

ulti-w

ord

Results

• Comprehension of colours and shapes compared to colour shape combinations

– Tested for lexical comprehension of colours (10), and shapes (10), and syntactic comprehension of colour shape combinations (100) during training. Results are averaged over 10 sessions.

Expressivity

0

20

40

60

80

100

0 5 10 15 20 25 30 35 40 45 50 55 60 65

No. <event, description>s entered

% s

trin

g s

et

com

pre

hen

ded

f

Lexical

Syntactic

Conclusions

The model demonstrates staged linguistic acquisition– No maturational triggers are employed– Training data are kept constant– Lexical items are required before compositions can

be derived

Conclusions

The model demonstrates staged linguistic acquisition– No maturational triggers are employed– Training data are kept constant– Lexical items are required before compositions can

be derived

Can this work be extended into further stage transitions?

Thank you

from syllables to syntax: investigating staged linguistic development through computational...

Technology

description t

syntactic language

pinkcross t

game training data

target language

artificial language

months12 months

months24 months