midas multi-device integrated dynamic activity spaces

MIDAS Multi-device Integrated Dynamic

Activity SpacesUnmil P. Karadkar

Ph.D. CandidateDept. of Computer Science

Texas A&M University

• Theory– Which aspects of information content do humans find valuable?

• Techniques– How to adapt content while optimizing for human perception? – How to enable heterogeneous device co-use?

• Frameworks– What are the attributes of multi-device systems? – How options are available to system designers for each attribute?

• System– Design a flexible architecture to embody multi-device system behaviors– Proof-of-concept implementation and analysis

• Practice– Harness device multiplicity to accomplish significant tasks

Research Questions

• Motivation• Related Work• 12C Framework• Approach• MIDAS Architecture• Content Transformation• Demonstration• Analysis• Contributions• Future Work

Outline

14” 17” 40”- 60”21-24”3.5”

Custom

13”- 17”

75”

10”

4.5” 6” 7”

– Display characteristics– Network bandwidth

– Processing power– Interaction modalities

Societal Changes

• User demographics• Nature of activity• Access location• Affordability– Free phones with contract– $150 e-readers

• Multiplicity – iPhone, iPad, MacBookPro

Heterogeneous contexts of information access

MIDAS

Multi-deviceIntegratedDynamicActivitySpaces

MIDAS

1. Deliver information elements to multiple devices simultaneously– Content– Interaction

2. Rely on widely available infrastructure– Wired and wireless computer networks– Mobile phones, e-readers, tablets, desktop computers– Sensors embedded in the environment

3. Support content transformation– Within form (image scaling, text summarization)– Across forms (text to speech, audio extraction from video)

Design Goals

4. Strive to retain content integrity from a human perspective– Select transformations that a user perceives to be

closest to the original form

5. Direct content elements to devices that can present them optimally

6. Reconfigure presentation dynamically in response to changes in device availability– New devices become available– Current devices become unavailable

Design Goals


Outline

Background

Individual devices

device-based content adaptation

Ubiquitous Computing

Infrastructure-enabled device co-use

• Service separation– Yahoo!, CNN, Weather Channel, UT Mobile

• Image distillation– Digestor (Fox 1996)

• Text and Web form summarization– Pythia (Bickmore 1997)– Power Browser (Buyukkokten 2002)– Proteus (Anderson 2001)

• Web page content extraction– Baseline SVPs + device-based PVPs (Chua 2007)

• Location-aware services– Computer for the 21st century (Weiser1991)– Java Ring (Olsen 2001)– IntelliBadgeTM (Cox 2003)

• Mobile devices in interactive spaces– Personal Server (Want 2002)– InfoStick (Kohtake 1999)– Elope (Pering 2005)

Opportunistic device co-use

Background—Device co-use• Pebbles (Myers 1997)

– Desktop and PDA co-use– Extend Windows applications to PDAs– PDAs as input and control devices

• WebSplitter (Han 2000)

– Split Web content across a user’s devices– Support for multiple views

• ANMoLE (Haneef 2004)

– Dynamic content reformulation (splitting video)– Integration of phone and computer networks

Background—Device co-use• Ubiquitous Display System (Aizawa 2002)

– Large public displays and mobile phones– Control from mobile phone

• Multibrowsing (Johanson 2001)

– Desktops, notebooks, PDAs, and large displays– Specially encoded links for content routing

• Interface Distribution Daemon (Luyten 2005)

– Web interface as well as stand-alone applications– Multiple content allocation modes– Usability analysis - Completeness and Continuity


Outline

• Concurrency– Simultaneous multi-device content presentation

• Control– Interaction support

• Comity– Permissible diversity in device characteristics

• Completeness– Presentation of relevant information elements

• Coverage– Distribution of content to optimize device usage

• Conversion– Content transformation to suit target device characteristics

12C Framework

• Composition– Relationship of components to the interface

• Coherence– Consistency in presentation of content elements

• Coordination– Reallocation of elements when devices change

• Continuity– Support for users to interpret and evaluate system state

• Constancy– Reuse of content instances that have been shown before

• Confidence– Security and trustworthiness in communication

12C Framework


Outline

Development Platform

Web or something else?

• WWW– Dominant infrastructure– World-wide audience

• Stateless server– Client-initiated communication– Content embedded within structure

• <p>, <h1>– Changing devices means restarting the activity

• Standardized browsers– Little support for custom behavior

WorldWide Web

context-aware Trellis (caT)• Petri net-based hypertext system (Furuta, Stotts - 1989)

– Formal, graph-based model– Browsing semantics (user privileges, time of day, location)

• Decouple specification from information content– Content not subsumed within structure– <img>, <a>– Separation allows devices to select suitable content

• Decouple content from presentation– Browsers present content autonomously

• Stateful server– Support for multiple clients– Easy propagation of user actions and content to multiple devices


Outline

Device 3

Browser 4Browser 3Browser 2

Device 2


Device 1

Browser 4Browser 3

Browser 2

MIDAS

Browser

ArchitectureSpecification

Information ServicecaT

Users

Device 3


ApplicationCoordinator

Device 2



Device 1

Browser 4Browser 3

Browser 2

MIDAS

Browser

Architecture

Device Manager

BrowserCoordinator

ResourceRealizer

Specification

Information Service

Users

Device 3



Device 2



Device 1

Browser 4Browser 3

Browser 2

MIDAS

Architecture

instance propertiescontent handle

user actions

content handles

Specification

Information Service

Users

instance handle, actions

status,user actions

instance handle

content

Browser

user actions content

ResourceRealizer

ResourceRepository

BrowserCoordinator

Device Manager

content handles instance

properties

(instances, devices)

Presentation onlyInput only Interactive

Device Manager

Instance properties

Device Selection Scheme

Device properties, interaction modeInstance Scorer

Content-Device Map

Content Handles, properties

Information Service

ResourceRealizer

BrowserCoordinator

Fully ReplicatedInteraction Replicated

Instance OptimalMIDAS Optimal


Outline

Content Transformation• Images– Scaling (automatic)– Color reduction (automatic)

• Text– Text extraction from formatted documents (automatic)

• HTML, Word, PDF– Summarization (semi-automatic)– Formatted data such as tables (semi-automatic)– Visual or audio rendering (automatic)

• Video– Audio extraction, down-sampling (automatic)– Video summarization (semi-automatic)

Resource Manager• Authoring support

• Device dependence for content instances

• Automatic transformation of images (ImageMagick)

• Author validation

Perception-based Image Transformation

• Images– Changes to attributes of photographs– Fastest growing information content on the Web

• Goal of user study– Understand human perception of nearness• Scaling of images• Color reduction (and gray scaling)

– Obtain actionable metrics to design rules for providing content in its most suitable form

Study Design• Two stages– Stage 1: Image classification– Stage 2: Similarity of image pairs

• Subject demographics– Five subjects per stage– Graduate students and staff– Different disciplines

Stage 1• Subjects classified 100 photographs– Taken by four photographers– Everyday and travel pictures– Prints of digital photos to make piles

• No guidelines regarding classification– Subjects created between 12 and 40 categories

• I coalesced categories based on image content– Four final categories

Image Types

People

Structures

Nature

Text

Stage 2• Subjects viewed image pairs on identical displays– No overlap or switching windows

• Each pair differed in size or color– Variables manipulated independently

• 20 image pairs for each variable

• Answered three questions– Similarity of images (9-point scale)– Suitability of automatic substitution (yes/no)– Acceptance of informed substitution (9-point scale)

Scaling

Color Reduction

Data Analysis

160x120

320x240

640x480

800x600

1024x768

2 (1-bit)

4 (2-bit)

16 (4-bit)

256 (8-bit)

16 Million (24-bit)

Size Colors1

2

3

4

Effect of Information Content Loss

Distance

Score Content Loss

0 6.89 67%1 6.03 54 %2 4.57 79 %3 4.93 90 %4 4.14 96 %

Distance Score Content Loss

1 7.12 56 %2 7.11 80 %3 6.87 93 %4 6.14 98 %

Scores on a 9 pt. Scale

320 X 240 X 24 vs. 320 X 240 X 1

1024 X 768 X 24 vs. 160 X 120 X 24

(no loss) (98 %)

(no loss) (96 %)

Scaling Color depth change

Similarity by image type

• Images of nature and structures scale well• Human faces• Legibility of textual elements• Algorithms for face and text detection in images exist

Distance nature structures people text1 6.57 7.67 7.17 6.782 7.33 6.78 7.09 7.253 7.75 7.00 7.00 6.294 7.00 7.50 5.71 5.00

Scaling

Distance Automatic substitution

Acceptability Informed Substitution

Acceptability

1 44% 7.66 56% 6.05

2 49% 7.42 51% 6.00

3 50% 7.13 50% 6.50

4 21% 7.67 79% 4.18

Scaling– Image Substitution

• No clear trends in terms of distance• Subjects allowed automatic substitution

– Corresponded with higher replacement suitability

Distance Automatic substitution

Acceptability Informed Substitution

Acceptability

0 44% 7.50 56% 5.10

1 36% 7.43 64% 4.44

2 10% 8.00 90% 3.89

3 29% 8.75 71% 4.00

4 29% 8.00 71% 3.80

Color Reduction – Image Substitution

• Acceptance of automatic substitution is lower than that for scaled images

• The suitability ratings are correspondingly higher

Perception-informed Rules • Principles

– Prefer scaling over color reduction– Avoid scaling of images that contain people and text– Gray scaling is the best option when used by itself

• Normalized score: Original image = 0 (best score)

+ Wcolor

(9 - Scoredist, color )

9x

Image Rating = Wsize

(9 - Scoredist, size)9

x Wtypex0.4

0.6

Wtogray = 1.0 (grayscale)Wtogray = 0.0 (otherwise)

Wtype

+ Wtogray

(9 - Scoredist, color )

9x

Wtype (people, text) = 0.53Wtype (nature, structures) = 0.47


Outline

1. Deliver information elements to multiple devices simultaneously– Content– Interaction

2. Rely on widely available infrastructure– Wired and wireless computer networks– Mobile phones, e-readers, tablets, desktop computers– Sensors embedded in the environment

3. Support content transformation– Within form (image scaling, text summarization)– Across forms (text to speech, audio extraction from video)

Analysis - Design Goals

✔

✔

✔

4. Strive to retain content integrity from a human perspective– Select transformations that a user perceives to be

closest to the original form

5. Direct content elements to devices that can present them optimally

6. Reconfigure presentation dynamically in response to changes in device availability– New devices become available– Current devices become unavailable

Analysis - Design Goals

✔✔

✔

Analysis - 12C FrameworkConcurrency ComityControl

ConversionCoverageCompleteness

CoordinationCoherenceCompositionConfidenceConstancy Continuity

Present information elements on multiple devices

• Inclusion of devices in MIDAS is optional• WebSplitter, ANMoLE

• A user can choose the interaction mode for each device

• Inclusion of public devices is not supported• Ubiquitous Display System

• Content is directed to devices automatically• Multibrowsing, Ubiquitous Display System

12C FrameworkConcurrency ComityControl



Interaction support

• Flexible – dependent on device choices• Most architectures support centralized or distributed

• It is possible to have no input devices at all

• May change during a session




Permissible diversity in device characteristics

• MIDAS places few restrictions on devices• Pebbles, Ubiquitous display system

• Device must run the Browser Coordinator

• No requirement of proximity to a user• Remote devices may be included - printers




Presentation of relevant information elements

• Ensures completeness by presenting all resource ids

• Resource Manager aids content authors• Support includes textual description as fallback

• IDD assertion is based on fixed content instances• ANMoLE and MIDAS convert content




Distribution of content to optimize device usage

• Flexible – dependent on Device Manager mode• Fully replicated - redundancy• Interaction replicated • Instance optimal - coverage for individual instances• MIDAS optimal – system-wide coverage

• Coverage may not apply to some systems• Ubiquitous Display System, Multibrowsing




Content transformation to suit target device characteristics

• Resource Manager• Authoring-time syntactic conversion with user validation

• Device• Presentation-time implicit conversion




Relationship of components to the interface

• Generative approach• Instances and actions together form the interface

• Many Web-based systems take a degenerative approach• Content filtering




Consistency in the presentation of content elements

• Content presented on all devices reflects the current state • Device Manager mode dependent

• Replication modes may present inconsistent information if the content is not vetted by the author




Reallocation of elements when devices change

• In response to • Device changes• User actions• Environmental properties changes (inherited from caT)

• Potential for optimization• Not all events threaten the integrity of a presentation




Reuse of instances that have been shown before• Not supported (yet)

• MIDAS always picks the most optimal instance

• Systems that use fixed instances support constancy better• Web-based presentations




Support for users to interpret and evaluate system state

• Supported with user-initiated state changes• Device changes• User actions

• Presentation changes due to environmental properties• Causes may be subtle – off-peak hours in another time zone




Security and trustworthiness in communication

• Reliance on the underlying communication infrastructure• Connection-oriented sockets

• Security or encryption is not supported


Outline

Contributions• Theory– 12C Framework for characterizing multi-device systems– Human perception of nearness in image adaptation

• Nature of content loss is important• Not all images scale equally gracefully• Gray scaled images are perceived to be the closest

• Techniques– Perception-informed metrics for image adaptation

• System– MIDAS architecture and prototype

• Flexible infrastructure • Minimal requirements for including diverse devices• Expression of various server- and client-side behaviors• Coordination in response to users, environment, and devices


Outline

Future Work• Infrastructure

– WebSockets API (HTML5), CSS3 (device-specific templates)– Clients for Android, iOS, and Windows– Inclusion of large-screen displays — GoogleTV(?)

• Interaction and Usability– Realistic setting, contemporary devices– Attention, cognition, privacy– Controlled studies, data from long-term use– Exploration of collaborative use

• Content transformation– Other image adaptations– Suitability of lessons learned from presentation for disabled

populations– Cross-form adaptation– Normalization of scores across different content forms

The most profound technologies are those that disappear.

- Weiser 1991.

midas multi-device integrated dynamic activity spaces

Documents

device availabilitynew

content integrity

split web content

heterogeneous device

mobile devices

dynamic content reformulation

availablecurrent devices

information elements