assisting visually impaired people with mobility through ... · pdf fileassisting visually...

Assisting Visually Impaired People

with Mobility through Technology in

the Age of ContextA SURVEY

Presented by Martin Goldberg

Committee:

• Prof. Zhigang Zhu (Advisor)

• Prof. Ioannis Stamos

• Prof. Zhanyang Zhang

• Dr. William H. Seiple

Assisting VIPs with Mobility in the Age of Context

Introduction

Mobile Devices

Sensors

Location Awareness

Social Media

Data

Integration

Conclusions


Background and Motivation

Robert Scoble & Shel Israel

Age of Context: Mobile, Sensors, Data

and the Future of Privacy. Patrick Brewster Press, 2013

Introduces the concept of the 5 Forces of the

Age of Context, called a ‘technological

storm’ – their main concern is how this affects

our privacy.

We want to use this ‘technological storm’ as

a prism to see how technology has been,

and – perhaps – can be, used to aid Visually

Impaired Persons (VIPs) with mobility issues.


Background and Motivation

https://nfb.org/blindness-statistics

Approximately

2.5%Blind People in US population.

Blind = visual acuity 20/200 or worse

VIP – can be 20/60 or worse

Most in senior years


The Five Forces of the AoC

• Mobile Devices

• Sensors

• Location Awareness

• Social Media

• Data (Big Data & the miracle of Little Data)*

* S & I consider the immense amount of Data (Big Data) and Data

Mining techniques that can find particular Data in the vast data out

there – the core of their fear for our privacy.


Definition: VIP

Visually Impaired Person

May have vision but does not always

see as clearly as required – sometimes

shapes, sometimes details; may not see

anything - although may still have light

perception.


Definition: Mobility

The area dealing with a VIP’s ability to

‘get out and about’, preferably

without needing a helper’s assistance.


Definitions: Wayfinding & Navigation

Wayfinding

Deciding a correct or optimal path to get from one

location to another.

Navigation

The process of managing the traversal decided by

wayfinding; being able to avoid hindrances or other

dangers - such as holes or edges.


Definition: Orientation

1. Defining the correct direction to

follow

2. Defining the direction the VIP is

facing.


Introduction

Mobile Devices

Sensors

Location Awareness

Social Media

Data

Integration

Conclusions


Mobile Devices

Abascal & Civit (2000)Recognize the value in portable communication as a

way to “untether” a VIP from the house. An instance of

general applicability being applied to VIP assistance.

http://bitchinlifestyle.tv/1960s-vs-2010s/

woman-talking-on-rotary-phone/ http://www.123rf.com/photo_24039158_

girl-sitting-on-a-park-bench-and-talking-

by-a-cell-phone.html


Mobile v. Portable DevicesKane (2009)

Freedom to Roam

a) Signature block

b) Water level Sensor

c) CCTV – Magnifier

d) D) Mobile Phone

S & I:

Mobile – has communication potential

Portable – lacking communications


Smartphones

Company Product OS Revealed Marketed

Google Google Now Android 2011 2012

Apple Siri iOS 2012 2013

Microsoft Cortana Windows Phone 8.1 2014 2015

Cortana Siri Google Now

Totally Blind

VIP usage:

As the iPhone

has only one

button to use,

it is favored

among blind

users.


Smartphone as Assistive TechnologyCoughlan and Manduchi (2008)

Search strategies of visually impaired

persons using a camera phone

wayfinding system.

Use of the smartphone camera has

been a major research topic, by these

authors and others.

Note the ‘awkwardness’ of how the

phone is worn on the neck.


Mobile Assistive Technologies

http://dub.washington.edu/

http://faculty.washington.edu/

wobbrock/hci.html

Studies on VIP Needs and Uses of Assistive Technology

U. Washington, Seattle - Jacob O. Wobbrock

DUB Lab

2009 – Freedom to Roam – Kane (now at Colorado)

with J. Bigham (Rochester, now CMU)

2013 - Exploring the Use of Speech Input

- Azenkot (now at Cornell)

Rochester, NY – Jeffrey Bigham - VizWiz (now at CMU)

2013 – Blind Use of Social Networks - Brady


Mobile Devices Commonly in Use

http://www.asus.com/us/Tablets/

ASUS_Transformer_Pad_TF103C/http://www.theweek.co.uk/iphone-6s/62138/

iphone-6s-and-6s-plus-smash-sales-records-

should-i-buy-the-new-iphone

Tablets

Stretch Screen TechnologySmartphones

Voice Over

Mobile devices with

cameras, stretch

screen technology

and text-to-speech

and speech

activation

technologies are very

programmable


Wearable Devices – the Future?

https://en.wikipedia.org/wiki/Google_Glass

Google Glass

https://en.wikipedia.org/wiki/Smartwatch

Smart watch

https://learn.adafruit.com/

introducing-the-raspberry-pi-2-model-b?view=all

Single-board

computers such as

the Raspberry Pi

Possibilities …


Introduction

Mobile Devices

Sensors

Location Awareness

Social Media

Data

Integration

Conclusions


Sensors

A sensor is a transducer whose purpose is to

sense (that is, to detect) some characteristic

of its environs. It detects events or changes in

quantities and provides a corresponding

output, generally as an electrical or optical

signal.- Wikipedia (Sensors)


Sensors - Sound

https://en.wikipedia.org/

wiki/Microphone


wi`ki/Loudspeaker


wiki/Headphones

Microphones – detect sound, transduce to electronic signals

1.Mid-range

2. Tweeter

3.Woofers

Transduce electronic

signals into sound

Loudspeakers Headphones


Sensors – Images /

Depth and Direction

https://en.wikipedia.org/wiki/

Digital_camera

Digital Camera (images)

transduces light capture

into digital matrices

Depth and direction can

be measured by Laser or

Ultrasound, most

commonly using time-of-

flight measurements Sekiguchi (2014) ref 74 in paper


Sensors – Force Sensors / Antennas


wiki/Computer_keyboard


wiki/Mouse_(computing)


wiki/Touchscreen https://en.wikipedia.org/wiki/Antenna_(radio)

Force Sensors transduce

pressure readings into

electronic

signals; here

button-based

keyboard and

mouse are

shown, as well as

touch-screen

technology

Antennas

transduce

electromagnetic waves

into electronic pulses


Sensors – Other Sensors

https://en.wikipedia.org/wiki/

Accelerometer

https://en.wikipedia.org/wiki/Gyroscope

https://en.wikipedia.org/wiki/Magnetometer

Accelerometers measure proper acceleration

(G-Force)

Gyroscopes are used to

measure and maintain

orientation

Magnetometers measure magnetic intensity

and can determine magnetic direction


Sensors – as used on a Smartphone

http://www.boeboer.com/page/29/ (Samsung Galaxy S III)


Body as Sensor

http://erc-assoc.org/about/erc_data/

spinoff-technology-argus-ii-retinal-prosthesis-system

http://www.news.com.au/technology/

blind-soldier-sees-with-his-tongue-using

-brainport-vision-technology/story-e6frfro0

-1225841742025

Argus II attempts to

bypass the retina,

passing signals derived

from a camera to the

optic nerve

Brainport attempts

to take a camera

input and, through

the tongue, teach

the brain to process

discernible images


Tactile Graphics

http://people.bordeaux.inria.fr/abrock/

Anke Brock (2014)

User studies on interactive

tactile maps, with ‘talking

labels’

Earlier work:

Joe Cioffi – Click ‘n Go

Noted difficulty – must be

memorized, not useful

when in motion.


Introduction

Mobile Devices

Sensors

Location Awareness

Social Media

Data

Integration

Conclusions


Global Positioning System - GPS

“A visual example of a 24 satellite GPS constellation in motion with

the earth rotating. Notice how the number of satellites in view from a

given point on the earth's surface, in this example at 45°N, changes

with time.” – Wikipedia label

https://en.wikipedia.org/wiki/Global_Positioning_System

GPS uses 24 geo-orbiting Satellites each

sending signals to points on the Earth’s

surface at a latency of 38 ms. The signal

includes time stamps generated by atomic

clocks.


GPS for VIPsProduct Software / OS Device Notes

Loadstone GPS Symbian OS Symbian (Nokia) Since 2006

LoroDux Java ME None specific Utilizes OpenStreetMaps,

under development

Mobile Geo Windows Mobile Windows enabled

devices

Seamless integration with Mobile Speak

BlindSquare iOS iPhones Based on FourSquare

combined with OpenStreetMaps

Trekker Breeze Firmware HumanWare Device not app for smartphone

previously Trekker (2003)

requires preset routes to work

BrailleNote GPS Firmware HumanWare PDA, allows note taking

and recording of points of interest


GPS use by VIPs

2005 – GPS available on Cell Phones.

3 Satellites – give a position in 2-D space.

4 Satellites – give a position in 3-D space.

VIPs will typically rely on at least two devices as location

awareness is so critical to them – Kane (2009)


When GPS is not Available

SmartPDR - Kang (2015)

Pedestrian Dead-ReckoningUses robotics location technology.

Using the IMU (Inertia Measurement

Unit) the application measures out

the pedestrian’s path, keeping track

of where the pedestrian is.

This interface may require

modification to be usable by VIPs.


Augmenting the Environment

Many solutions suggest modifying the environment, typically with

RFIDs and beacons to interact with smartphone applications.

This is typical of indoor solutions, where GPS is not available.

An outside example is suggested by

Flores & Manduchi (2014),

where they place beacons at public

transport sites informing VIPs of their

location and guiding them to the

appropriate platform.


Computer Vision Algorithms

Feng Hu (CCVCL – 2015)

Literature Review looks at various Computer Vision Algorithms

to aid with localization.

Main relevant points:

• 2D Solutions

• 3D Solutions

• 2D – 3D Solutions

• Emerging Technologies

Under consideration:

Using 2D camera pose in 3D model to orient VIP in Manhattan.


Introduction

Mobile Devices

Sensors

Location Awareness

Social Media

Data

Integration

Conclusions


Dictionary.com definition:

Websites and other online means of

communication that are used by large

groups of people to share information

and to develop social and professional

contacts.

Facebook Study (2014):

VIPs use Social Media just like everyone

else.

Social Media


Social Media for VIPs

BlindSquare etc.

https://seshagiriprabhu.wordpress.com/2013/07/21/building-an-openstreetmap-tile-server-on-ubuntu-12-04-lts/http://www.foursquare.com

http://blindsquare.com/

A combination of two

existing applications for

iPhones

Software to

locate

friends

nearby

Crowd-sourced

electronic map

generation


Social Media for VIPs

Crowdsourcing

https://itunes.apple.com/us/app/vizwiz/id439686043?mt=8

Provided by G. Olmschenk

Crowdsourcing is an interesting idea.

Ideally, volunteers log in

and help the VIP.

VizWiz – from J. Bigham’s

group had minimal

success.

Erin Brody ()2013)(same

group) found that VIPs as a rule meet this

technology with distrust. Time will tell …


Introduction

Mobile Devices

Sensors

Location Awareness

Social Media

Data

Integration

Conclusions


Big Data

The Miracle of Little Data

The Age of Context

S & I: Age of Context is a threat to our privacy.

Data today is stored in huge graph databases

which can then be mined to elicit specific data

Can we utilize this Age of Context

to aid VIPs with mobility?

on a specific subject, such as

one’s current location.


http://www.aspire-consultancy.co.uk/0_html/products/

notetakers/braillesense/sync.htm

The SyncBraille

Braille Display http://www.antilogic.co.za/technology/

plan-b-tactile-map-translator-braille-maps/

Classic Data for VIPs

Tactile

Graphics, in

particular

Braille, are

now

electronically

available for

VIPs


Image Data

https://en.wikipedia.org/wiki/Geotagging

Geotagging Li Fei-Fei (2005) – Textons for Bag-of-Words

Image understanding

Computer vision’s answers to aiding VIPs are

mostly through image data. Here we see some

image based algorithms.


Introduction

Mobile Devices

Sensors

Location Awareness

Social Media

Data

Integration

Conclusions


Integration of the Five Forces

Mobile Devices -

- Data

Social Media -

- Location Awareness

- Sensors

The Five Forces do not –

of course – exist on their

own.

Here we will consider

how some or all of the

forces have been

employed in

combination …


Beyond Tactile Maps

Anke Brock (2015)

Tactile and audio sensors on

a mobile tablet provide

location awareness. The

data is haptic.

Talking tactile maps have

labels that give information

in audio format.


Wearable Indoor Navigation System

Yicheng Bai (2014)

An example of a system

becoming more typical: the user

wears a camera and a GPS

enabled mobile device with

connectivity to the web. Data is

processed in the cloud. While

social media is not overtly

mentioned, it is suggested.

This is a concept paper.


BlindNavi : A Navigation App for VIP

On SmartphoneHsuan-Eng Chen et al. (2015)

BlindNavi Project:

Similar but applies FourSquare

into the system. Uses a

smartphone to process the data

and location based on GPS.

Smartphone = Mobile device +

sensors, in particular camera.


Talking Points 3

Rayoung Yang (2011)

We note that Talking Points 3

includes four of the five forces:

mobile devices, sensors, location

awareness, and data. Social

media could be added in such a

system for providing more relevant

and current information for VIPs.

Introduces the idea of interaction with the environment, a push-

pull approach.


Introduction

Mobile Devices

Sensors

Location Awareness

Social Media

Data

Integration

Conclusions


Summing Up the Age of Context

And VIPs

Research:

The possibilities are

there …

Evidence

suggests that

VIPs do not

readily accept

advances*


Summing Up the Age of Context

And VIPsWhite Cane

&

Guide Dog

Traditional:

The iPhone has

been widely

adopted; but, few

VIPs readily utilize

applications*Brady (2013)


Proposal:

An Interactive ApproachHere, we consider a

‘smart’ environment

that communicates

with the VIP who has a

mobile device capable

of processing image

and other data and of

communicating with

the environment.


Thank You

Any questions?

assisting visually impaired people with mobility through ... · pdf fileassisting visually...

Documents