© 2003, carla schlatter ellis display management: sensing user intention and context (hotos 2003)
TRANSCRIPT
2© 2003, Carla Schlatter Ellis
Outline
• Motivation and Research Objective• FaceOff Architecture and Prototype• Evaluation
– Best Case Feasibility Study– Responsiveness Study
• Future Work• Related Work
– Dark Windows
3© 2003, Carla Schlatter Ellis
Motivation
• Current energy management techniques tied to process execution
• Can we use low power sensors to match I/O behavior more directly to user behavior and reduce system energy consumption?
Sensing User Intention and Context
for Energy Management
4© 2003, Carla Schlatter Ellis
Case Study: FaceOff
• Displays:– Typically responsible for large power drain– Power State can be controlled by software– State transition strategies naïve
A display is only necessary if someone is looking at it.
6© 2003, Carla Schlatter Ellis
Prototype
• IBM ThinkPad T21 running RedHat Linux– Base Power Consumption = 9.6 Watts
– Max CPU = 8.5 Watts over Base
– Display = 7.6 Watts
• Logitech QuickCam Web Cam– Power Consumption = 1.5 Watts
• Software components:– Image capture, face detection, display power state
control
7© 2003, Carla Schlatter Ellis
Face Detection
• Skin detection used for prototype
• Real time proprietary methods exist
8© 2003, Carla Schlatter Ellis
Outline
• Motivation and Research Objective• FaceOff Architecture and Prototype• Evaluation
– Best Case Feasibility Study– Responsiveness Study
• Future Work• Related Work
– Dark Windows
9© 2003, Carla Schlatter Ellis
Best Case Feasibility Study
• What is the potential for energy savings?– Assume perfect accuracy – Best case user behavior – start it and leave.
• Tradeoff of energy costs:– CPU/Camera vs. Display
• Effect on System Performance– Network file transfer (113 MB)– CPU intensive process (Linux kernel compile)– MP3 Song (no display necessary)
12© 2003, Carla Schlatter Ellis
Energy and Time Comparisons
Energy (J) Default With FaceOff % SavingsCompile 12506.85 11023.07 11.86Transfer 6795.42 4791.19 29.49
Time (s) Default With FaceOff % OverheadCompile 575 603.5 4.96Transfer 348.6 351.3 0.77
13© 2003, Carla Schlatter Ellis
MP3 Application
• Playing an MP3– Display not necessary– Song completes before default timeout turns off
display
• Energy comparison– 3,403 J with FaceOff vs. 4,714 J with Default– 28% energy savings
• No noticeable effect on playback
14© 2003, Carla Schlatter Ellis
Responsiveness Study
• Use full prototype including skin detection
• Establish baseline timing
• Examine Responsiveness– varying system load– varying polling rate
15© 2003, Carla Schlatter Ellis
Responsiveness Timing
Face arrives (or departs)
pollinglatency
detectionlatency
Image acquired detection completedisplay signaled
Total responsiveness latency
16© 2003, Carla Schlatter Ellis
Baseline Detection Latency
• Measured over a period of one hour with no programs other than background processes running
• Latency increased over time– Started at ~110ms– Increased to ~160ms– Why?
• Appears to be an effect of Linux scheduler reducing priority of long running jobs
18© 2003, Carla Schlatter Ellis
Detection Latency Under Load
Workload Average(99% Confidence)
Maximum Minimum
Network Transfer
175±7ms 305ms 116ms
Kernel Compile
230±5ms 669ms 51ms
MP3 Song 154±3ms 229ms 84ms
19© 2003, Carla Schlatter Ellis
Outline
• Motivation and Research Objective• FaceOff Architecture and Prototype• Evaluation
– Best Case Feasibility Study– Responsiveness Study
• Future Work• Related Work
– Dark Windows
20© 2003, Carla Schlatter Ellis
Varying Polling Rate
• Reduce overhead by reducing polling rate– Increases responsiveness latency
• Adaptive polling rate– Eliminate polling in presence of UI events– Begin polling as duration without UI events
increases and face is detected– Reduce polling when no face present
• Similar problem with latency increase upon return
21© 2003, Carla Schlatter Ellis
Optimization with Motion Sensor
• Combine adaptive polling & motion sensing
• Meet responsiveness requirements with minimal FaceOff system overhead
• Eliminate image polling when no motion
• Switch display state on immediately when motion detected and restart image polling
22© 2003, Carla Schlatter Ellis
Implementation
• Prototype using X10 ActiveHome Wireless Motion Sensor and Receiver– Receiver connects to serial port– Reading port blocks until sensor triggers– Takes up to 10 seconds to recharge
• Promising addition to FaceOff system
23© 2003, Carla Schlatter Ellis
More Roles for Sensors
• Touch Sensor– Detect picking up of a PDA
• Light, Sound sensors– Adjust display brightness (Compaq iPAQ)– Adjust speaker volume
• 802.11 Signal Strength sensor– Determine possibility of offloading
computation
24© 2003, Carla Schlatter Ellis
Enhanced Sensors
• “Active Camera”– Perform some or all of the face detection
• Color filtering– Preprocessing skin color segmentation
• Low Power processor for external sensor control, computation
26© 2003, Carla Schlatter Ellis
Future Work
• Continue work on optimizing responsiveness• Comprehensive user study
– Survey of usability– Characterization of usage patterns
• End-to-end experiment
• Implementation with available very low power camera/motion sensor and prototype for small device (handheld)
27© 2003, Carla Schlatter Ellis
Conclusions
• Context information offers promising method of energy management
• FaceOff illustrates feasibility of approach
• Available very low power sensors as well as optimization techniques would improve upon the FaceOff energy savings
28© 2003, Carla Schlatter Ellis
Outline
• Motivation and Research Objective• FaceOff Architecture and Prototype• Evaluation
– Best Case Feasibility Study– Responsiveness Study
• Future Work• Related Work
– Dark Windows
29© 2003, Carla Schlatter Ellis
Related Work• Display Power Management
– Industry Specifications• APM, ACPI, DPMS
– Zoned Backlighting– Energy-Adaptive Display System Design
• Attentive/Perceptual UIs– Smart Kiosk System: Gesture analysis– CAMSHIFT: Game control– IBM PupilCam: Head gesture recognition
30© 2003, Carla Schlatter Ellis
ACPIAdvanced Configuration and Power
InitiativeBrought to you by Intel, Microsoft, and Toshiba
and designed to enable OS Directed Power Management (OSPM).
• Goal is to be able to move power management into software for more sophisticated policies
• Abstract OS-HW interface• Replaces APM interface
31© 2003, Carla Schlatter Ellis
What ACPI Offers• Standardization industry-wide
(Vendors to support ACPI in products instead of building their own power mgt)
• System and device power states• Thermal model
– Thermal zones, indicators, cooling methods
• BIOS interfaces– Motherboard configuration tables – Interpreted control methods
• Plug-and-play• Complexity moved into OS
32© 2003, Carla Schlatter Ellis
What ACPI Offers• System
– Mechanisms for putting computer as a whole in sleep/wake states
• Devices– ACPI tables describe motherboard devices
• Power states• Controls for managing states
• Processor – Detecting idle state and swapping to low power
• Batteries– Querying and controlling battery behavior
33© 2003, Carla Schlatter Ellis
Power States
G: global states apply to entire system and are visible to user
D: states of individual devices
S: sleeping states within the G1 state
C: CPU states G2-S5Soft off
LegacyG0-S0
working
G3mech off
G1Sleep
S1S2 S3
S4
Dxmodem x DxHDD xDxcdrom x Cxcpu
wakeup
34© 2003, Carla Schlatter Ellis
ACPI Internal Structure
AC PI T ab lesAC PI B IO SAC PI R eg isters
K ernel
D eviceD river
ACPIRegisterInterface
ACPI T ableInterface
ACPI BIO SInterface
Platform H ardw are
Existingindustrystandardregister
interfaces to:CM O S, PIC,
PIT s, ...
AC PI D river/AM L Interpreter
ApplicationsO S
D ependentA pplication
A PIs
O S Specifictechnolog ies,
in terfaces, and code.
O SIndependenttechnologies,
interfaces,code, andhardw are.
BIO S
O SPM System Code
- Hardw are/Platform- Provided by ACPI CA
- ACPI Spec Covers this area.- O S specific technology
37© 2003, Carla Schlatter Ellis
OSDM: OnNow
Applications
OS
HW
OnNow WIN32 ext
ACPI Spec
SetSystemPowerState– initiate sleep state, query apps(?)
SetThreadExecutionState– specifies level of support needed
(e.g. display required)
WM_POWERBROADCAST– a message notifying of power state
changes to which applications can respond
SetWaitableTimer– ensure PC is awake at scheduled
time
RequestDeviceWakeupRequestWakeupLatency - to specify latency requirements
GetSystemPowerStatus and GetDevicePowerState
38© 2003, Carla Schlatter Ellis
Outline
• Motivation and Research Objective• FaceOff Architecture and Prototype• Evaluation
– Best Case Feasibility Study– Responsiveness Study
• Future Work• Related Work
– Dark Windows
© 2003, Carla Schlatter Ellis
Energy-Adaptive Display System Designs for Future
Mobile EnvironmentsS. Iyer, L. Luo, R. Mayo, P.
Ranganathan, MOBISYS 2003
40© 2003, Carla Schlatter Ellis
Opportunity• New technology:
OLEDs – Organic Light Emitting Diodes– Energy consumption on a per-pixel basis
by determining each pixel’s brightness and color
– Energy consumption of different regions of screen to be changed independently
– No separate backlight– In development by Kodak, Sanyo, Sony,…
41© 2003, Carla Schlatter Ellis
Dark Windows
• Software support – modifications to windowing system to ensure energy is spent mostly on window-of-focus (capturing user’s area of visual interest)
• Non-active screen area is changed (dimmed or re-colored for energy optimization)
42© 2003, Carla Schlatter Ellis
Justification• Usage study – what are the user’s needs and
how well do they match display characteristics?• Characterization of display usage in Microsoft
Windows by 17 “typical” users• Application logger – recorded, for up to 14
days, when user was active.– Window of focus (the one accepting keyboard input)
– its size, location, title– Size of total screen area used (all non-minimized
windows
43© 2003, Carla Schlatter Ellis
Screen Usage Results• On average
only 59% of area used by window-of-focus,additional 17% bybackground windows
• High variability among users
• Large fraction of smaller windows have very low content (notifications, alerts) – don’t need full color characteristics of display to convey it.
44© 2003, Carla Schlatter Ellis
Dark Windows Design• Prototyped on X Windows under Linux• Used VNC – Virtual Network Computing
Server – provides a virtual representation of display – virtual framebuffer where pixels can be manipulated between application and display
• Track window-of-focus, apply modifications to pixels outside of it
45© 2003, Carla Schlatter Ellis
Modifications
• Half Dimmed – areas outside window-of-focus are dimmed to 50% brightness
• Fully Dimmed – areas outside are turned off• Gray Scale – areas outside are changed to
gray by setting rgb to average value.• Green Scale – areas outside are changed to
green which is lowest power color for OLEDs. Dims to 67%
47© 2003, Carla Schlatter Ellis
Evaluation• Energy benefits measured by generating synthetic
trace from usage study and playing trace on prototype.
• 15 inch OLED displays were not available so they used a software power model to calculate power consumption– Controller power set to 0.5W– Driver power – 1W– 1024x768 pixels individually consume -- red power 4.3W,
green power 2.2W, blue power 4.3W.
• User experience – users want to see background but willing to use dark windows
48© 2003, Carla Schlatter Ellis
Results based on default Teal Background*
* Benefits depend on original Choiceof backgroundand windowcolors.
49© 2003, Carla Schlatter Ellis
Conclusions
• While benefits depend on usage scenario, significant energy savings can be achieved with these optimizations
• Further opportunities for application specific adaptivity– More meaningful notions of area of focus can be
defined (e.g. most recent email message, most recently changed region of screen)
– Better match to (low) content – e.g., notifications could be audible signal instead of popup window