Upload File

multimodal devices: at home, at work, on the go · multimodal devices: at home, at work, on the go...

  • Home
  • Documents
  • Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream
37
Multimodal devices: Multimodal devices: at home, at work, on the go at home, at work, on the go ETSI Workshop on Multimodal Interaction on Mobile Devices 1 18 -19 November 2008 - ETSI, Sophia Antipolis, France James Warden, Research In Motion, [email protected] Arnold Sheynman, Research In Motion, [email protected]

Upload: others

Post on 22-May-2020

2 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Multimodal devices: Multimodal devices: at home, at work, on the goat home, at work, on the go

ETSI Workshop on Multimodal Interaction on Mobile Devices

1

18 -19 November 2008 - ETSI, Sophia Antipolis, France

James Warden, Research In Motion, [email protected] Sheynman, Research In Motion, [email protected]

Page 2: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

The meaning of MultimodalThe meaning of Multimodal

2

Page 3: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Focus of this presentationFocus of this presentation

�� Applications, Features and Services that can be Applications, Features and Services that can be developed because there are multiple radios availab le:developed because there are multiple radios availab le:� In the device� External to the device

3

�� Not on unique user interface types for devicesNot on unique user interface types for devices

Page 4: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

The meaning of MultimodalityThe meaning of MultimodalityMultimodality is a concept that allows mobile device users to seamlessly switch among different mobile technologies as well as modesof interaction based on the context, certain criteria and constraints

4

Page 5: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Introduction to WirelessIntroduction to Wireless

5

Page 6: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Wireless is all around usWireless is all around us

Car remote controls

6

Wireless accessories

Cordless phones

TV remote controls

Cellular phones

Access points

Page 7: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Wireless has cut the cordsWireless has cut the cords�� Matching the multimodality of Matching the multimodality of

devices to the multimodality of our devices to the multimodality of our liveslives

7

Page 8: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Wireless technology categorizationWireless technology categorization

�� RangeRange� Relatively Long Range

� Cellular based (e.g. GSM, CDMA, etc.)

� Short Range� Bluetooth, Zigbee, NFC

8

�� Data RatesData Rates� High Speed (> 1Mbps in at least the downlink direct ion)

� Wifi, 1XEV-DO, HSPA (HSDPA/HSUPA)

� Medium Speed� GPRS, EDGE, Evolved EDGE, Bluetooth (BT)

� Slow Speed� Zigbee, NFC

Page 9: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Wireless technology categories (cont’d)Wireless technology categories (cont’d)

�� PowerPower� More power = shorter battery life� More power = longer range (in general)

�� Digital Modulation SchemesDigital Modulation Schemes

9

�� Digital Modulation SchemesDigital Modulation Schemes� PSK – finite number of phases� FSK – finite number of frequencies� ASK – finite number of amplitudes� QAM – finite number of at least two phases and at le ast two

amplitudes

Page 10: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Wireless technology tradeWireless technology trade--offsoffs

�� Data rateData rate� Higher order modulation (e.g. 64-QAM vs. 16-QAM) ge nerally

provides higher data rates but poorer noise perform ance resulting in less range

�� Battery lifeBattery lifeCertain modulation schemes generally require linear

10

� Certain modulation schemes generally require linear amplifiers (QAM, OFDM) vs. lower power class C ampl ifiers that can be used in schemes such as FSK, PSK and CD MA

�� Receiver ComplexityReceiver Complexity� Certain modulation schemes require coherent demodul ation

which requires perfect phase and frequency synchron ization with the carrier

Page 11: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Multimodal handsetMultimodal handset

�� Combined technologies used to deploy new sets of Combined technologies used to deploy new sets of features and servicesfeatures and services� High Speed Cellular Data Networks (1xEV-DO, EDGE,

Evolved EDGE, HSDPA, HSUPA)� Wifi

11

� Wifi� Bluetooth� Zigbee� GPS� Others

Page 12: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Multimodality at homeMultimodality at home

12

Page 13: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Home media networkHome media network

13

Page 14: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Home remote controlHome remote control

�� Control Home media server via HandsetControl Home media server via Handset� IP based TV sets get controlled via Wifi from cell phone

to media server� Stream content from Media Server to local TV monito rs

controlled via an app running on the Handset

14

controlled via an app running on the Handset� Music choices selected via Handset as well as volum e

control and routing etc.� Page others over home TV network� Send PiP of something interesting that is being wat ched

Page 15: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Paging mode example on home media networkPaging mode example on home media network

15

Page 16: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Current issuesCurrent issues

�� TV manufacturers are working on proprietary RF and TV manufacturers are working on proprietary RF and Control ProtocolsControl Protocols� This will force all of the components of the system to be the same

brand

16

�� They are pushing their own proprietary Remotes to c ontrol They are pushing their own proprietary Remotes to c ontrol the systemthe system

�� Industry needs to push hard for Open Standards for the Industry needs to push hard for Open Standards for the Interfaces to home media devicesInterfaces to home media devices

Page 17: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Example of home security mode Example of home security mode � Detecting the presence of a visitor using the visit or door monitor

� Communicating information received at the visitor m onitor in association with the visitor detected to a remotely located communication network gateway (home hub) over a sho rt range wireless link

� Communicating the presence of the visitor from the home hub to a remote mobile device and forwarding the correspon ding information from the home hub to a remote mobile de vice over a

17

a remote mobile device and forwarding the correspon ding information from the home hub to a remote mobile de vice over a short range wireless link

� Establishing communication between a remote mobile device and a visitor monitor over a short range wireless l ink

Source: Patent US 7,304,572

?

Page 18: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Multimodality at workMultimodality at work

18

Page 19: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Support of multiple modesSupport of multiple modesDedicated storage space and services

Internet

Conference room projector

19

Your laptop

Office printer

Office access point

Page 20: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Why is a switch necessary?Why is a switch necessary?

• User preferences based on the current task

• Context change and context awareness

• Employment of suitable User Interface technologies

20

• Minimizing the charges including the roaming charges

• Maximizing the battery life

• Minimizing the time for multiple tasks execution

Page 21: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

• The switching among the technologies and modes can The switching among the technologies and modes can be performed implicitly or explicitlybe performed implicitly or explicitly

• For implicit switching (passive modalities), the us er may For implicit switching (passive modalities), the us er may or may not be notified about the technology and/or mode or may not be notified about the technology and/or mode changechange

Multimodality and seamless switchingMultimodality and seamless switching

21

changechange

• Example: The device automatically switches from WAN to WLAN when the user enters the office

• For explicit switching (active modalities), the use r’s For explicit switching (active modalities), the use r’s intervention and input are requiredintervention and input are required

• Example: The user forces the mobile device to switc h to WLAN from WAN when the WLAN access point becomes availab le

Page 22: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Multimodal device configurationMultimodal device configuration

Out of band configuration

“Touch”

1. Configuration initialization

2. Reduction of the configuration

parameters (optional)

22

The device-inviter “touches” the device-invitee to pass its Bluetooth or/and

WLAN settings

3. Setup out of band channel

(e.g. NFC)

4. Configuration data exchange

5. Bluetooth or/and WLAN

setting selection

6. WLAN or/and Bluetooth

connection setup

Source: www.ip.com IPCOM000175673D

Page 23: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Increasing productivity: multitaskingIncreasing productivity: multitasking

• Sending an SMS from the mobile device while being o n the call

• Sending an email from the smart phone while being o n the call

23

• Browsing the Internet on the mobile device, sending an email and maintaining the phone call simultaneously

Page 24: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

• Choose the closest printer based on the beacons’ RS SI

• Use WLAN to send the document for printing

• Indoor location e.g., based on Zigbee triangulation

Example of context awareness modeExample of context awareness mode

24

Document to print

Page 25: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Multimodality on the goMultimodality on the go

25

Page 26: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Remote home automation Remote home automation –– Mode 1Mode 1

�� Security Security –– turn on lights prior to arriving at hometurn on lights prior to arriving at home

26

Page 27: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Remote home automation Remote home automation –– Mode 2Mode 2

�� Security Security –– Package DeliveryPackage Delivery� Remotely disable alarm, raise garage door, allow pa ckage delivery all

while watching remotely via security cameras

27

Page 28: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Accessing home media remotelyAccessing home media remotely

TCP/IP

HSDPA

EVDO

My media

Home network

28

�� Media server at home Media server at home � Contains users video and music content

�� On the go user selects content from home media serv erOn the go user selects content from home media serv er� Streams music to listen to in the car via high spee d cellular access to

handset which sends via Bluetooth to car stereo sys tem or by simple audio cable

� In the hotel room, via Wifi user selects video to w atch from home media server and watches via handset, laptop or even hote l TV via cables

EVDOInternet

Page 29: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Some points to take awaySome points to take away

29

Page 30: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

1. Consider Multimodality as a concept 1. Consider Multimodality as a concept that allows mobile device users to that allows mobile device users to

seamlessly switch among different mobile seamlessly switch among different mobile technologies as well as modes of technologies as well as modes of

30

technologies as well as modes of technologies as well as modes of interaction based on the context, certain interaction based on the context, certain

criteria and constraintscriteria and constraints

Page 31: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

2. Match the multimodality of devices to 2. Match the multimodality of devices to the multimodality of our livesthe multimodality of our lives

31

the multimodality of our livesthe multimodality of our lives

Page 32: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

3. The wireless technologies should be 3. The wireless technologies should be harmonized with input/output modesharmonized with input/output modes

32

harmonized with input/output modesharmonized with input/output modes

Page 33: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

3. Switch depending on the task3. Switch depending on the task

Audio Visual

WifiCellular

33

HapticsSpeech Bluetooth NFC

Other Other

Page 34: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

4. The APIs should be developed to 4. The APIs should be developed to harmonize the wireless device capabilities harmonize the wireless device capabilities

and input/output modesand input/output modes

34

and input/output modesand input/output modes

Page 35: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

5. The creativity is boundless5. The creativity is boundlessThe newest BlackBerry Storm handset uses a radically different touchscreen keyboard • The Storm uses a touchscreen

similar to the iPhone's, where you move your finger lightly on the surface to navigate around the screen

To confirm a selection or menu choice, you press harder. The slight flexing of the display triggers a tiny switch beneath it, generating distinct tactile feedback. This avoids a common problem on touchscreens: It can be difficult to distinguish between just moving

35http://www.businessweek.com/magazine/content/08_42/b4104000262138.htm?campaign_id=yhoo

• The Storm does let you edit by cutting and pasting—and you can use a two-finger stretch to select text

• You can also use a flick of a finger to scroll pages up or down, which is particularly handy for browsing quickly through a list of e-mails

around the display and initiating an action

Page 36: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

6. Explore the new opportunities6. Explore the new opportunities

Olfaction Taste

36

☺

Page 37: Multimodal devices: at home, at work, on the go · Multimodal devices: at home, at work, on the go ... IP based TV sets get controlled via Wifi from cell phone to media server Stream

Thank you!Thank you!

37


Mobile Multimodal Interaction for Older and Younger Users ... · Devices) project, funded by the EU H2020 ICT22 call for Multimodal and Computer Interaction, aims to tackle limitations

Case Report Multimodal Imaging of Acquired Vitelliform ...downloads.hindawi.com/journals/criopm/2013/461758.pdf · Multimodal Imaging of Acquired Vitelliform Lesion Diagnosed at Pseudohypopyon

Multimodal Transport 1. MTOGA Multimodal Transportation of Goods Act 1993 Multimodal Transport 2

FUSION AND COORDINATION FOR MULTIMODAL … · FUSION AND COORDINATION FOR MULTIMODAL INTERACTIVE INFORMATION PRESENTATION ... 1 Some slides are available at wahlster ... & Invisible

Transitional devices at pangatnig

Revolutionizing Prosthetics: Devices for Neural Integration€¦ · multimodal cortical and peripheral interfaces, Phase 2 consisted of selecting the devices that were best suited

Supplementary fi Multimodal epidermal devices for hydration monitoring · 2017-06-05 · Multimodal epidermal devices for hydration monitoring Siddharth Krishnan1,2,*, ... AZ5214,

Contending with Terms: “Multimodal” and “Multimedia” in the … · Available online at Computers and Composition 26 (2009) 225–239 Contending with Terms: “Multimodal”

Optimisation of Container Process at Multimodal Container Terminals …eprints.qut.edu.au/16626/1/Andy_Wong_Thesis.pdf · Optimisation of Container Process at Multimodal Container

BIOGRAPHIES WORKSHOP ON MULTIMODAL INTERACTION ON MOBILE DEVICES …€¦ · Multimodal interaction is challenging to develop especially in mobile/ubiquitous settings with novel interaction

Multimodal Sensor Front End Data Sheet …Multimodal Sensor Front End Data Sheet ADPD4000/ADPD4001 Rev. A Document Feedback Information furnished by Analog Devices is believed to be

Multimodal Research at CPK, Aalborg

Speech and Language Processing for Multimodal · devices and effectively exploit these devices to clearly and concisely express our communicative intentions. In contrast, in human-computer

MULTIMODAL FINAL REPORT PLANNING AT THE JUNE 2017 ...€¦ · Multimodal Planning at the Megaregional Scale ii • Institutional support is a barrier to addressing the megaregional

Multimodal Biometric Authentication at Feature Level Fusion · 2016-09-09 · Multimodal Biometric Authentication at Feature Level Fusion V.Sireesha ,Research Scholar, Padmavathi

Novel Devices and Material characterization at mm-wave … · Novel Devices and Material characterization at mm-wave and ... Novel Devices and Material characterization at mm-wave

Multimodal epidermal devices for hydration …rogersgroup.northwestern.edu/files/2017/mndualmode.pdfOPEN ARTICLE Multimodal epidermal devices for hydration monitoring Siddharth Krishnan1,2,*,

Multimodal Transformer for Unaligned Multimodal Language

Workshop Programme Multimodal Corpora From Multimodal ... Corpora... · "Multimodal Corpora From Multimodal Behaviour Theories to Usable Models" ... Analysis of gesture expressivity

Networking Biomedical Research Center BIOENGINEERING Multimodal Diagnosis Intelligent Devices BIOENGINEERING Multimodal Diagnosis Intelligent Devices

Alfabetización multimodal: usos y posibilidades Multimodal

Multimodal Interaction with Robotic Devices: Assistive ... · Multimodal Interaction with Robotic Devices: Assistive Technologies Applications ... People with disabilities ... Locomotion

Project Name 27/SR 60) Multimodal Mobilityswflroads.com/us27sr60/fastlanegrantapplication/... · 2016-12-15 · 27/SR 60) Multimodal Mobility ... SR 60 intersects CSX Railroad at

Intermodal and Multimodal Logistics Knowledge Paper · A multimodal transport contract is a single contract for carriage of goods by at least two different modes of transport. A multimodal

A Survey on Hardware and Software Solutions for Multimodal … · 2016-11-22 · Á. Csapó et al. A Survey on Hardware and Software Solutions for Multimodal Wearable Assistive Devices

Inter-semiotic Texture: Analyzing Cohesive Devices between Language …multimodal-analysis-lab.org/_docs/pubs03-Intersemiotic... · 2009-05-28 · Analyzing Cohesive Devices between

A4 : AT Devices

Pangatnig at transitional devices

FIATA and MULTIMODAL CORIDORS - UNECE · PDF filemode or multimodal transport means), ... restrictive to use multimodal transport operators. • Brasil multimodal ... document •

Aalborg Universitet From Annotated Multimodal Corpora to … · Abstract. Multimodal corpora prove useful at different stages of the development process of embodied conversational

Multimodal Summarization with Guidance of Multimodal Reference

Speech-Centric Multimodal Interfaces - RLE at · PDF fileFlanagan. Speech-Centric Multimodal Interfaces Moreover, accommodating natural human modes — especially those of sight, sound

Multimodal Planning at the Megaregional Scale · 2017-09-27 · page 1 Final Report MULTIMODAL PLANNING AT THE MEGAREGIONAL SCALE EXECUTIVE SUMMARY Between June 2015 and May 2017

Multimodal Biometrics at Feature Level Fusion using Texture Features

2016 Multimodal Asset Performance Report€¦ · Table of Contents & Multimodal Asset . 2016 Management Highlights. Multimodal Safety Highlights. Multimodal Asset Performance Dashboard