internal knowledge representation for conversational ai · 2018-07-26 · illustration credit to...
TRANSCRIPT
![Page 1: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/1.jpg)
Internal Knowledge Representation for Conversational AI
Zachary Brown & William MyersBrigham Young University
![Page 2: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/2.jpg)
![Page 3: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/3.jpg)
What makes a good conversation?
![Page 4: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/4.jpg)
Aspects of Natural Language
1. How do we process what the user says?
2. How can we create a response in a naturally-worded way?
3. Given several different responses, how do we pick the most relevant one?
https://cdn.mantelligence.com/wp-content/uploads/2017/11/weird-conversation-starters.png
![Page 5: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/5.jpg)
Aspects of Natural Language
1. How do we process what the user says? - Understanding
2. How can we create a response in a naturally-worded way?
3. Given several different responses, how do we pick the most relevant one?
https://cdn.mantelligence.com/wp-content/uploads/2017/11/weird-conversation-starters.png
![Page 6: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/6.jpg)
Aspects of Natural Language
1. How do we process what the user says? - Understanding
2. How can we create a response in a naturally-worded way? - Generating
3. Given several different responses, how do we pick the most relevant one?
https://cdn.mantelligence.com/wp-content/uploads/2017/11/weird-conversation-starters.png
![Page 7: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/7.jpg)
Aspects of Natural Language
1. How do we process what the user says? - Understanding
2. How can we create a response in a naturally-worded way? - Generating
3. Given several different responses, how do we pick the most relevant one? - Selecting
https://cdn.mantelligence.com/wp-content/uploads/2017/11/weird-conversation-starters.png
![Page 8: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/8.jpg)
understanding generating selecting
![Page 9: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/9.jpg)
Illustration credit to Pixar Animation Studio and their film “WALL-E”
System Architecture
![Page 10: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/10.jpg)
Illustration credit to Pixar Animation Studio and their film “WALL-E”
System Architecture - Understanding
![Page 11: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/11.jpg)
Illustration credit to Pixar Animation Studio and their film “WALL-E”
System Architecture - Generating
![Page 12: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/12.jpg)
Illustration credit to Pixar Animation Studio and their film “WALL-E”
System Architecture - Selecting
![Page 13: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/13.jpg)
understanding generating selecting
![Page 14: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/14.jpg)
understanding generating selecting
![Page 15: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/15.jpg)
Knowledge Graphs:
Using Graphs to Understand and Generate Text
● General Knowledge
● High-Level and Low-Level Topic Identification
● User and Self Modelling
● Knowledge Disambiguation and Conflict Resolution
![Page 16: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/16.jpg)
Graph Nodes and Edges
The Hobbit
J.R.R. Tolkien
Wrote
![Page 17: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/17.jpg)
Graph Nodes and Edges
The Hobbit
FictionJ.R.R. Tolkien
Wrote
Written By
Belongs to the Genre
Genre Is a Writes in the Genre
![Page 18: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/18.jpg)
J.K. Rowling The Hobbit
FictionJ.R.R. Tolkien
Wrote
Written By
Writes in the Genre
Belongs to the Genre
Genre Is a Writes in the Genre
Person
Is a
Is a
Graph Nodes and Edges
![Page 19: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/19.jpg)
General Knowledge: Understanding Text
User said “I like J.R.R. Tolkien.”:
1) Look for a node named “J.R.R. Tolkien” in the knowledge graph
2) Grab all edges and nodes connecting to the J.R.R. Tolkien node
3) Now you know:a) J.R.R. Tolkien is a personb) J.R.R. Tolkien wrote The Hobbit
4) Repeat steps 1-3 for the nodes connected to the J.R.R. Tolkien node for more information:
a) The Hobbit belongs to the genre called Fiction
The Hobbit
FictionJ.R.R. Tolkien
Wrote
Written By
Belongs to the Genre
Writes in the Genre
Person
Is a
![Page 20: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/20.jpg)
General Knowledge: Generating Text
User said “I like J.R.R. Tolkien.”:
1) You now know that:a) J.R.R. Tolkien is a personb) J.R.R. Tolkien wrote The Hobbitc) The Hobbit belongs to the genre
called Fiction
2) Given these facts and that the user ‘likes’ J.R.R. Tolkien, it’s straightforward to create templates for generating text:
a) “Since you like node1 and node1 edge node2, do you also like node2?”
3) In this case, we get:a) “Since you like J.R.R. Tolkien, and
J.R.R. Tolkien Writes in the Genre Fiction, do you also like Fiction?” Note: The node and edge names
determine the output of the template.This doesn’t sound natural!
The Hobbit
FictionJ.R.R. Tolkien
Wrote
Written By
Belongs to the Genre
Writes in the Genre
Person
Is a
![Page 21: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/21.jpg)
Topic Identification: High-Level and Low-Level
“Since you like J.R.R. Tolkien, and J.R.R. Tolkien Writes in the Genre Fiction, do you also like Fiction?”
How do you know the topic of the conversation?● Topics are hierarchical:
○ The Fiction topic has subtopics, one of which is The Hobbit
● Topics are sometimes never said:○ J.R.R. Tolkien is a topic within the
Person topic
Solution:● Count number of edges leading to nodes● Layer topics● Consider having ‘meta-topics’ determined
by combinations of edges present in the conversation
User said: “I like J.R.R. Tolkien.”You said:
The Hobbit
FictionJ.R.R. Tolkien
Wrote
Written By
Belongs to the Genre
Writes in the Genre
Person
Is a
![Page 22: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/22.jpg)
User and Self Modelling: Understanding Dialogue
User said: “I like J.R.R. Tolkien.”
User
Likes
EVE
Dislikes
Which nodes and edges should we focus on?
Our models of the user and the EVE system tell us to focus on the The Hobbit node, and not the J.K. Rowling node.
Good models lead to interesting, engaging, and informative dialogue.
The Hobbit
FictionJ.R.R. Tolkien
Wrote
Written By
Belongs to the Genre
Writes in the Genre
J.K. Rowling
Writes in the Genre
![Page 23: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/23.jpg)
User and Self Modelling: Generating Text
User said: “I like J.R.R. Tolkien.”
Text can be generated by trying to connect the EVE node to the User node. If the User likes something that connects to a node that EVE doesn’t like, we could use the following:
Template:● “But node1 edge2 node2 and node2
edge3, so why would you edge1 node1?
Map:● {“User” : “you”, “Likes” : “like”,
“Dislikes” : “is boring”, “Writes in the Genre” : “wrote”}
To obtain:● “But J.R.R. Tolkien wrote
Fiction and Fiction is boring, so why would you like J.R.R. Tolkien?”
User
Likes
EVE
Dislikes
The Hobbit
FictionJ.R.R. Tolkien
Wrote
Written By
Belongs to the Genre
Writes in the Genre
![Page 24: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/24.jpg)
Knowledge Disambiguation and Conflict Resolution
User said: “I like J.R.R. Tolkien.”
EVE said: “But J.R.R. Tolkien wrote Fiction and Fiction is boring, so why would you like J.R.R. Tolkien?”
User said: “Good point, I typically don’t like fiction.”
FictionJ.R.R. Tolkien
Writes in the Genre
User
Likes
EVE
Dislikes
Dislikes
We appear to have a conflict in our knowledge graph, so we ask for clarification; if the user likes something directly connected to something they dislike:
Template:● “How can you edge1 node1 if you
edge2 node2?”
Map:● {“User” : “you”, “Likes” : “like”,
“Writes in the Genre” : “wrote”, “Dislikes” : “dislike”}
EVE said: “How can you like J.R.R. Tolkien if you dislike Fiction?”
User said: “Because J.R.R. Tolkien was my grandfather.”
Grandparent of
Grandchild of
![Page 25: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/25.jpg)
http://colah.github.io/posts/2014-07-FFN-Graphs-Vis/
Typical Graph
![Page 26: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/26.jpg)
Pros:
● Easy to Template● Enables scalable internal
representation of vast amounts of data○ ~ 50 million items
● Straight forward to map out and add new functionality
Cons:
● Naming Conventions● Time intensive maintenance● Must try to envision every possible
template for user interaction● Not scalable for broad tasks such as
human conversation
Summary: Knowledge Graphs
![Page 27: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/27.jpg)
understanding generating selecting
![Page 28: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/28.jpg)
understanding generating selecting
![Page 29: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/29.jpg)
Text Embeddings:
Using Vector Representations of Text to Understand and Select
● What makes a sentence interesting?
● Predicting what the user would like to hear
● Semantic meaning
![Page 30: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/30.jpg)
Text Embeddings● Map complex data like English text to a numerical representation more
suitable for computers.● Position (distance and direction) captures semantic meaning in these spaces
○ Semantically similar ideas are close○ Dissimilar ideas are far apart
● Uses:○ As input for neural networks (e.g. for classification)○ Directly for NLP tasks
![Page 31: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/31.jpg)
Uses of Text Embeddings for Understanding
![Page 32: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/32.jpg)
Analogical Reasoning
swam
walkedswimming
walking
Verb Tense
building
software
programmer
architect
Arbitrary RelationshipsVector Representations of Words
https://www.tensorflow.org/tutorials/word2vec
![Page 33: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/33.jpg)
Semantic SimilarityHow old are you?
What is your age?
My phone is good.
Your cellphone looks great.
...
Embed
[0.1, 0.6, …, 0.7]
[0.3, 0.8, …, 0.5]
[0.9, 0.2, …, 0.1]
[0.4, 0.7, …, 0.2]
…
How old are you?
What is your age?
My phone is good.
Your cellphone looks great.
How
old
are
you
?
Wha
t is
your
age
?
My
phon
e is
goo
d.
You
r cel
lpho
ne
look
s gr
eat.
The Universal Sentence Encoderhttps://www.tensorflow.org/hub/modules/google/universal-sentence-encoder/1
1.0 0.8 0.6 0.4 0.2 0.0
![Page 34: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/34.jpg)
ClassificationHow old are you
What is your age
My phone is good
Your cellphone looks great
...
Embed
[0.1, 0.6, …, 0.7]
[0.3, 0.8, …, 0.5]
[0.9, 0.2, …, 0.1]
[0.4, 0.7, …, 0.2]
…
Question Classifier
96%
98%
7%
8%
...
How old are you
What is your age
My phone is good
Your cellphone looks great
The Universal Sentence Encoderhttps://www.tensorflow.org/hub/modules/google/universal-sentence-encoder/1
![Page 35: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/35.jpg)
Selection
![Page 36: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/36.jpg)
Which is the best response?Where are you from? ● New York.
● I grew up in California.● Have you ever been to California?● I am from California.● Yes.● Babies are usually born in a hospital.● Manhattan.● NY.● Cali.● I love Star Wars.● California.● I grew up in Santa Fe, but my family just moved to Salt Lake City.
![Page 37: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/37.jpg)
We will use:● Google’s Universal Sentence
Encoder ● Our Chit Chat dataset
![Page 38: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/38.jpg)
Embed user’s inputWhere are you from? [0.124, 1.34, … .9]Universal
Sentence Encoder
Where are you from?
![Page 39: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/39.jpg)
Find most similar utterances to user’s inputFind closest n points in Chit Chat dataset.
Where did you grow up?
Where are you from?
Where are you from?
Where are you from?
Where were you born?
![Page 40: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/40.jpg)
Get their responsesGet the response vectors from Chit Chat dataset for each point.
Are any of my messages sendingNOOOOOOO
I am from south carolina
Arizona:)
I am from south carolina. If money were no object for me...
Where did you grow up?
Where are you from?
Where are you from?
Where are you from?
Where were you born?
![Page 41: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/41.jpg)
Construct analogy vectorsSubtract the closest n points from their responses.
![Page 42: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/42.jpg)
Construct analogy vectorsAverage the vectors and extend it by the average norm
analogy_vec *= avg_norm / np.linalg.norm(analogy_vec)
analogy_vec
![Page 43: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/43.jpg)
Estimate ideal response vectorAdd re-lengthed average analogy vector to user’s input.
Where are you from?
![Page 44: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/44.jpg)
Rank candidate outputs by distance to ideal vectorWhere are you from? 0.789 I am from California.
0.843 California.0.845 Cali.0.855 I grew up in Santa Fe but my family just moved to Salt Lake City.0.864 I grew up in California.0.899 NY.0.925 New York.0.998 Manhattan.1.058 Yes.1.082 Have you ever been to California?1.112 I love Star Wars.1.203 Babies are usually born in a hospital.
![Page 45: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/45.jpg)
Rank candidate outputs by distance to ideal vectorWhere were you born? 1.065 California.
1.106 Cali.1.131 NY.1.132 New York.1.158 Manhattan.
1.18 I am from California.1.21 I grew up in California.
1.231 I grew up in Santa Fe but my family just moved to Salt Lake City.1.277 Have you ever been to California?1.375 Babies are usually born in a hospital.1.394 Yes.1.487 I love Star Wars.
![Page 46: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/46.jpg)
Pros:
● Captures semantics● Semantic similarity is very useful● Building block to build many different
ML models on
Cons:
● Doesn’t readily generate text: ○ Currently can’t decode from the
vector space - nearest neighbor searches are the best we have
Summary: Text Embeddings
![Page 47: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/47.jpg)
understanding generating selecting
![Page 48: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/48.jpg)
understanding generating selecting
![Page 49: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/49.jpg)
Deep Methods:Using Machine Learning for
End to End Text Understanding, Generation, and Selection
● Novel text generation
● End to End
● Robust
![Page 50: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/50.jpg)
Recurrent Neural Network (RNN)
http://colah.github.io/posts/2015-08-Understanding-LSTMs/
Sequential text generation
Pros:● Simple loss
function● Mimic good data
Cons:● Difficult to train
![Page 51: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/51.jpg)
Latent Variable Hierarchical Recurrent Encoder - Decoder (VHRED)
Sequential text generation
Pros:● Contextual
integrity improved● Mimic good data
Cons:● Difficult to train
A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues (2016)https://arxiv.org/pdf/1605.06069.pdf
![Page 52: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/52.jpg)
Convolutional Sequence to Sequence (ConvS2S)
Non-sequential text generation
Pros:● Parallelizable● Faster Training ● Mimic good data
Cons:● More complex loss
function than sequential generators
Convolutional Sequence to Sequence Learning (2017)https://arxiv.org/pdf/1705.03122.pdf
![Page 53: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/53.jpg)
ConvS2S
Non-sequential text generation
Pros:● Parallelizable● Faster Training● Mimic good data
Cons:● More complex
loss function than sequential generators
VHRED
Sequential text generation
Pros:● Contextual
integrity improved● Mimic good data
Cons:● Difficult to train
RNN
Sequential text generation
Pros:● Simple loss
function● Mimic good data
Cons:● Difficult to train
Summary: Deep Methods
CON:● Lack of good data● Can’t ‘seed’ for a
targeted result
PRO:● Potential for
self-play
![Page 54: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/54.jpg)
Unanswered:How do we produce targeted text with ML?
![Page 55: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/55.jpg)
Unanswered:How do you score a conversation?
![Page 56: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/56.jpg)
What makes a good conversation?
![Page 57: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/57.jpg)
Thank you.pcc.cs.byu.edu
![Page 58: Internal Knowledge Representation for Conversational AI · 2018-07-26 · Illustration credit to Pixar Animation Studio and their film “WALL-E ... Naming Conventions Time intensive](https://reader034.vdocuments.net/reader034/viewer/2022042623/5fb185a8faa0431e8d0d5f39/html5/thumbnails/58.jpg)
Questions?