AN INTELLIGENT ASSISTANT FOR NAVIGATION OF VISUALLY IMPAIRED PEOPLE

N.G. Bourbakis*# and D. Kavraki #

#AIIS Inc., Vestal, NY, email: aiis@stny.rr.com

*WSU, ITRI, Image-Video-Machine Vision Research Lab, Dayton OH.

Outline

• Introduction• Tyflos System• Human Tyflos Interaction• Visual to Audio Communication• Detection of Other Moving Objects• Navigation with L-G Landmarks• IMPLEMENTATION OF THE TYFLOS PROTOTYPE• CONCLUSIONS

Introduction

• This paper presents the navigation methodology employed by an intelligent assistant (agent) for people with disabilities partial independence.

• Tyflos– Work in a 3-D dynamic environment– System carries two vision cameras and captures images from

the surrounding 3-D environment.– These images into verbal descriptions for each image into a

verbal communication.

Introduction

• In USA– there are more than 3 millions with low vision and millions more

completely blind.

• A great challenge– robotic-like vision

• Be able to obtain 3-D representations of the surrounding world.

• provide verbal descriptions of it

Introduction

• Of particular importance– Unknown environment– High resolution– Analyze– Events based on simple queries– Vision system convert visual descriptions into verbal communication

Introduction

• Applied artificial intelligence and software technologies– QBIC

• Query By Image Content

– EUROTRA• Machine languages translation

– STARS• Newsletter

– MEL• An autonomous intelligent vehicle

– Intelligent walking robots, 3-D space maps generation, autonomous vision systems, neuromorphic vision chips, the robot with feelings

Introduction

• verbally see– an intelligent system to make visually impaired people via a

machine vision system equipped with speech and natural language understanding technologies.

• challenges and technological– the navigation methodology of an intelligent assistant for visually

impaired people's partial independence.

TYFLOS SYSTEM

• System's Main Components– The– The– The– The– The– The– The

Pair of Glasses

Vision Cameras

Laser Range Scanner

Ear Speaker

Microphone

Portable Computer

Communication System

TYFLOS SYSTEM

• The Pair of Glasses– The pair of glasses is a important part of the Tyflos system, and

several system’s parts are mounted on it.

TYFLOS SYSTEM

• The Vision Cameras– The vision camera is mounted on the top of the glasses in order

to provide a better view of the targeted scene.

• Camera– High resolution– Color– Small in size and weight.

TYFLOS SYSTEM

• The Laser Range Scanner– "scan“

• the same view with the camera.

– For determination of the distances of the existing objects– For 3-D view of a captured image

• The size of the laser scanner– resolution at least 1cm radius in a distance of 15 meters.

TYFLOS SYSTEM

• The Ear Speaker– high fidelity component

• has only one ear speaker through it the user receives the detailed description of the surrounding environment.

TYFLOS SYSTEM

• The Microphone– User communicates with computer and the

appropriate Databases.

– Filter• To eliminate the noise from the user's environment.

TYFLOS SYSTEM

• The Portable Computer– It hosts all the important software tools that make the Tyflos

system functional.

– DBs• Image• Speech• Natural Language

– Software Interfaces– Knowledge Conversion Tools– Communication Tools

TYFLOS SYSTEM

• The Communication System– the user needs to establish communication with other

resource• portable or mobile information DBs

TYFLOS SYSTEM

Source:http://tweakers.net/nieuws/31815/computer-met-ogen-begeleidt-slechtzienden.html

HUMAN TYFLOS INTERACTION

• Tyflos is able to assist the user are：– where am I ?– describe the surrounding scene.– guide me to a particular point– where is a particular object?

HUMAN TYFLOS INTERACTION

• Independent– Human assistant by describing to the user the 3-D visual

environment making him/her more independent.• find an “object” → microphone → scanning the surrounding

space → vision camera → analyzed and recognized → ear speaker

• What Object– describes to the user via the ear speaker what object made a

certain sound.

Visual to Audio Communication

• Where am I ?– GPS (Global Positioning System)

• Tyflos – uses a Speech synthesizer– provides to the user the names of the streets and / or the names

of certain buildings.– provides to the human user a verbal description of the

surrounding 3-D scene.

Visual to Audio Communication

• Describe the Surrounding Scene– SPN graph

• Generation and Description of a 3-D Surrounding Unknown Environment

• Fusion of Images & range Data for 3-D Modeling of the Free Space

Visual to Audio Communication

• Generation and Description of a 3-D Surrounding Unknown Environment– stereo speaker– vision camera – laser range data

• Dr.Albus and Dr. Bourbakis– develop 3-D images via audio signals generated by two

speakers during the motion of a sound generation equipment.

Visual to Audio Communication

• Fusion of Images & range Data for 3-D Modeling of the Free Space– Cameras Stabilization Issues

• synthesis of images– Generation & Representation of the Navigation Space

• Free Navigation Space (FNS)– orientation-angle– magnetic north– current position– inclination angle

• shape SH– ” open-navigation corridors ”

Visual to Audio Communication

• Where is a Particular object? – particular object is converted into a hierarchical SPN graph– the SPN graph is compared to the system's KB– recognized or appropriately classified– speech DB and ear speakers

Visual to Audio Communication

• Guide me in a Particular Place– Collision Free Path Planning

• the system attempts to select the First Best Choice (FBC) open navigation corridors.

– Detection of Other Moving Objects • How Tyflos detects other moving objects in the same navigation space• What is the perceived size of the moving object• What is the direction and the velocity of the moving object• How many different moving objects are in the same free space

Detection of Other Moving Objects

• How Tyflos detects other moving objects in the same navigation space– the space shape changing methodology

• shape SH & Dt

• do(Dt) = VRa*Dt → P'

• P' → SH'– IF SH" = SH' THEN there is no other moving object in the same navigation space

– ELSE there is at least one moving object in the same free navigation space.

– the differences detected from the comparison of two consecutive 2-D images.

• ”motion”

Detection of Other Moving Objects

• What is the perceived size of the moving object– When Tyflos, detects another moving object, Rb, in the same

navigation area, then it attempts to define the size (dimensions) "perceived" from its current position P.

• Rb = |dk'(t(i+1)) - dk(t(i+1))|

– Corresponding window• 2-D image• Knowledge Base

Navigation with L-G Landmarks

• L-G Landmarks– local-global graphs

• Knowledge Base (KB)– successful and flexible generation of Space Maps is the development of

a knowledge base able to store, process and access the Space Maps' forms.

• Communication and Mobile DBs Issues– to carry CDs with different DBs

– its computer to be connected with a multimedia network in order to download the appropriate DB

IMPLEMENTATION OF THE TYFLOS PROTOTYPE

• Integration of AI Methodologies– The actual implementation of the Tyflos system combines the

state-of-the-art in AI methodologies.

• The Prototype– AIIS (American Institute of Indian Studies) – Tyflos system consists of the pair of glasses, a Hitachi high

resolution, color vision camera, a portable computer Mc book-note, an ear speaker, a voice synthesizer and a microphone.

CONCLUSIONS

• It will provide the ability to the user to understand better the surrounding environment, and give more independence in his/her motion.– Vision– Speech

• However, a significant progress has been done, and new methods in both vision and speech developed by other researchers will be appropriately used.

Thank you for your attention