Face Recognition Memory with NLP backend of Social Robot for HRI

8th June 2018, Friday

Nidhi Mishra

Project Officer, IMI

IMI Research Seminar

1

Contents• Background of Robots

• Social Robots

• Motivation

• Proposed Model

• Algorithm

• Conclusion

• Future Work

• References

2

3

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Industrial Service~27.5M 2

~1.7M

- https://assets.kpmg.com/content/dam/kpmg/pdf/2016/06/social-robots.pdf

4

70%

17%

9%2%2%

Robots

Cleaning work Hobby Research Lawn-mowing Others

Hobby

Non-Social Social

Research

Non-Social Social

- https://assets.kpmg.com/content/dam/kpmg/pdf/2016/06/social-robots.pdf

Robot roll-call

5

- https://assets.kpmg.com/content/dam/kpmg/pdf/2016/06/social-robots.pdf

SocialHumanoid Robot

6

Aging population

7

• The world is aging at a rapid rate and by 2030 there will be 34 nations where more than 20% of the population is over 65. This has broad implications for economic growth and immigration trends.

• With an older population that works less, support and dependency ratios get out of whack.

• Robots will perform many elder-care tasks within a decade.

- http://money.cnn.com/interactive/news/aging-countries/

8

Robot as Companion

Companion robots are robots that can make themselves useful, carry out various type of tasks to assist humans in a domestic environment. They should behave socially and interact with a socially acceptable manner with humans [Dautenhahn et al., 2005]

9

Challenges

Several research have suggested characteristics that a social robot should visibly possess [Fong et al., 2003, Dautenhahn, 2007b, Heerink, 2010]:

• Express and perceive emotions

• Communicate with high level dialogue

• Learn and recognise models of other agents

• Establish and maintain social relationships

• Use natural cues of communication

• Exhibit distinctive personality and character

• Learn and develop social competencies

10

Have a MemoryRecognize it’s CompanionUnderstand Human Language Naturally

11

12

Proposed Solution

Real Time face Recognition and Training

13

OpenFace

• Detect faces with a pre-trained models from dlib or OpenCV.

• Transform the face for the neural network. This repository uses dlib's real-time pose estimation with OpenCV's affine transformation to try to make the eyes and bottom lip appear in the same location on each image.

• Use a deep neural network to represent (or embed) the face on a 128-dimensional unit hypersphere.

• Apply classification technique to the features to complete recognition task.

- https://cmusatyalab.github.io/openface/

14

15

16

17

18

Extracting NLP features

• Part of speech (reads a sentence and assigns parts of speech to each word)

• Named entity recognition (labels sequences of words in a text which are the names of things, such as person and company names, or gene and protein names.)

• Find dependencies (representation of grammatical relations between words in a sentence)

• Coreference ( the task of finding all expressions that refer to the same entity in a text.)

-https://stanfordnlp.github.io/CoreNLP/index.html

19

Sentence as input :-

1. Nidhi is working at NTU Singapore.

2. She works on Nadine robot.- http://nlp.stanford.edu:8080/corenlp/process

20

WordNet

• WordNet is lexical database for English Language based conceptual look-up.

• Organizes lexical information in terms of word meanings rather than word form.

How wordNet can help us?One person can say a sentence in different ways using different words whose meaning are same.Example : I am going to downtown this weekend.

I am going to downtown this sunday.I am going to downtown this week.

WordNet can help me find similar words for one word. -http://www.nltk.org/howto/wordnet.html

21

22

23

https://stanfordnlp.github.io/CoreNLP/index.htmlFabrizio Sebastiani. Machine Learning in Automated Text Categorization. ACM Computing Surveys, 34(1):1–47, Mars

Conclusion

• For robots, it is crucial that they possess human-like social interaction skills. It is vital for a robot to know who it is talking with and remember facts about the human companion.

• Aims to explore the possibility of improving human-robot interaction(HRI) by exploiting natural language resources and using natural language processing (NLP) methods.

24

Future Plans

• An illustration through a real case of this model.

• Compare the model using standard database and a new database for HRI applications.

25

Thank you!

Q & A

26