ice ss2013
TRANSCRIPT
ICE Summer School, July 2013
Visual Sensor Networks
Bernhard Rinner
Institut für Vernetzte und Eingebettete Systeme
B. Rinner. 2
Agenda
Sensor Networks
Smart Cameras
Visual Sensor Networks
B. Rinner 3 3 3
Introduction to Sensor Networks
B. Rinner 4
Wireless Sensor Networks (WSNs)
• Networks of typically small, battery-powered, wireless devices, (“sensor nodes”, “motes”) – On-board processing,
– Communication, and
– Sensing capabilities.
Sensors
Processor
Radio
Storage
P O W E R
Sensor node schematics [© Oracle Labs]
B. Rinner 5
Sensor Node Platforms
• From research prototypes to commercial products
The Vision „Smart Dust“ UC Berkeley late 1990‘s
Commercial Products „Mote-on-a Chip“ Dust Networks, 2010
Research Prototypes „Mica-2“ Crossbow 2004
B. Rinner 6
Some Applications of Sensor Networks
• Health
• Structural Monitoring
• Agriculture
• Environmental
[(c) University of Ghent] [Kim et al. ACM SenSys, 2006
[AgriNet] [M. Welsh, Harvard 2007]
B. Rinner 7
Communication is Key
• Wireless communication is an enabling technology – Eases deployment
– Enables mobility
– Increases flexibility
– Reduces costs
• Communicate on demand (ad hoc, spontaneous) with dynamic infrastructure – Nodes organize themselves into network
– Data is transferred via multiple hops
source destination
B. Rinner 8
But also …
• Advances in sensor technology – Micro electro-mechanical system
(MEMS) revolutionized sensing
– Integration of mechanical and electrical components on single chip
– Example: 3D accelerometer (in your cell phone)
• Embedded processors and integration – Moore’s Law still valid
– Trade-off between processing performance and power consumption
[© SensorDynamics]
[© ARM]
B. Rinner 9 9 9
The “Energy Problem”
• Major sources of energy consumption – Sensing, computing, communication
– High temporal variation
• Energy is the scarce resource for WSN. Several challenges – What energy reservoirs to exploit?
Constraints: availability, max. power, size, …
– How to distribute power over the network? Energy provider and consumer might be dislocated.
– How to control the distribution? “The proper amount of energy in the right place at the right time”
• Sensor networks have always been a “green” technology!
B. Rinner 10
Alternative Energy Reservoirs
• Maybe micro-heat engines – Exploit MEMS technology
to build „internal combustion engines“
– Expected power: 10-20 W
– Still in early research/development phase
• Or harvest energy from environment – Organic semiconductors for exploiting indoor ambient light
– Thin film batteries for storing energy
– EnHANTs : energy harvesting networked active tags
[Handbook of Sensor Networks, Wiley]
[Columbia University, CLUE]
B. Rinner. 11 11 11
Smart Cameras
B. Rinner. 12
Principle of Smart Cameras
• Smart cameras combine – sensing,
– processing and
– communication
in a single embedded device
• perform image and video analysis in real-time closely located at the sensor and transfer only the results
• collaborate with other cameras in the network
TrustEYE.M4 prototype on top of RaspberryPI
B. Rinner. 13
Differences to traditional Cameras
Traditional Camera – Optics and sensor
– Electronics
– Interfaces
delivers data in form of (encoded) images and videos, respectively
Smart Camera – Optics and sensor
– Onboard computer
– Interfaces
delivers abstracted image data and is configurable and programmable
Sensor
Electronics
Image enhancement/ Compression
Image Video
Sensor
Embedded Computer
Image analysis
„Events“
Programming Configuration
Light Light
B. Rinner. 14
SmartCams look for important things
• Examples for abstracted image data – compressed images and videos
– features
– detected events
© CMU
B. Rinner 15
Be aware of scarce Resources
• Major resource limitations – Processing power
– Communication bandwidth
– Onboard memory
– Energy
• Various Prototypes (with decreasing performance)
Sony XCISX100C/XP x86 VIA Eden ULV @ 1 GHz
TrustEYE.M4 ARM Cortex@ 168MHz
SLR Engineering Atom Z530@ 1.6 GHz
CITRIC PXA 270@ 13-640MHz
[Rinner et al. The Evolution from Single to Pervasive Smart Cameras. Proc. ICDSC 2008]
B. Rinner 16 16 16
Characteristics of Visual Sensor Networks
B. Rinner 17
Video Surveillance Network
• 3rd generation – all-digital systems
• 3+ generation – smart cameras
– surveillance tasks run on-site on smart cameras, e.g., • video compression traffic statistics
• accident detection wrong-way drivers
• stationary vehicles (tunnels) vehicle tracking
• 1st and 2nd generation – primarily analog frontends
– backend systems are digital
[Regazzoni, Ramesh, Foresti. Special Issue on Video Communications, Processing and Understanding for Third Generation Surveillance Systems. Proceedings of the IEEE. October 2001]
B. Rinner 18
Video Surveillance Network (2)
• Even third generation networks rely on “heavy” infrastructure. – Camera nodes: sensor, onboard processing (encryption)
– Network: hierarchically structured, wired, large bandwidth
– Energy: dedicated supply
• Surveillance networks typically consist of large number of cameras
• Processing in network is fixed; (compressed) data is streamed to control center
B. Rinner 19
Characteristics of VSN
• Visual sensor networks lie somewhere in between wireless sensor networks (WSN) and multi-camera/surveillance networks.
• VSN have unique characteristics (wrt. traditional WSN)
• Resource limitations – Need to process and transfer large amounts of data
– Energy and bandwidth
• On-board processing (cp. Smart cameras) – Challenging vision algorithms
– Adaptive behavior
[Soro et al. A Survey of Visual Sensor Networks. Advances in Multimedia 2009]
B. Rinner 20
Characteristics of VSN (2)
• Real-time operation – Most applications require real-time analysis (camera to user)
• Location and orientation information (spatial calibration) – Absolute or relative coordinates and orientations
– (Multi-)camera calibration
• Time Synchronization (temporal calibration)
• Data Storage – Access to historic data necessary, eg., frame buffer, detected events
– Stored data may be discarded over time
• Autonomous Camera Collaboration – cp. Distributed smart cameras (DSCs)
B. Rinner. 21
(Selected) VSN Problems
• Sensor Placement – Eg., dynamic setting of PTZ parameters
• Clustering, cluster head election – Eg., what cameras should “work together”, who is the “leader”
• Synchronization and calibration – Eg., establish temporal and spatial correlation
• Data (and energy) distribution – Eg., when and what data to exchange
• …
B. Rinner. 22 22 22
Coordinating Resources in
Visual Sensor Networks
B. Rinner 23
Configuring Smart Camera Networks
• Smart camera networks process data onboard can modify their functionality/execute actions during runtime to reflect changes – to the state of the environment
– to the user criteria
• A configuration describes what is processed/executed where; specified by – Description of camera network (including the available actions/tasks)
– Specification of the objective
• We study configuration methods to use scarce resources in these networks more efficiently
B. Rinner 24
Configuration Problem (example)
• Configuring a camera network – Select a set of cameras to monitor an area of interest
– Set the sensor (frame rate, resolution, PTZ) to achieve QoS
– Assign monitoring functions to cameras
– Optimize wrt. multiple criteria
– Dependent on dynamics of environment
s1
s2 s3
s4
s5
t1
t2
t3
p1, p2
p3
p4, p5
B. Rinner 25
Configuration Design Space
Design space for configuration methods is given by:
• Dynamics of environment (static vs. mobile observation points)
• Configuration algorithm (centralized vs. distributed)
• Tasks and sensors (homogeneous/heterogeneous; static/mobile cameras)
• A priori knowledge (complete vs. no knowledge of environment/VSN)
• Various alternatives for solving this optimization problem, eg. – Centralized configuration algorithms
– Distributed configuration algorithms
focus on resource-aware approaches
B. Rinner 26
Centralized Configuration with EA
• Approximation with evolutionary algorithm satisfying all requirements along multiple criteria (eg., energy, data, QoS)
• Smart Camera Network – Set of cameras at known position with fixed FoV
– Sensor configurations (frame rate, resolution)
• Observation Area – Static set of observation points with monitoring activity a
at required QoS (pot, fps)
• Monitoring tasks – Assign procedures for achieving
– Required resources for
},...{ 1 nSSS =
[Dieber, Micheloni, Rinner. Resource-Aware Coverage and Task Assignment in Visual Sensor Networks IEEE Transactions on Circuits and Systems for Video Technology, Aug 2011]
},...,{ 1 ki ddD =
},...,{ 1 mttT =
},...,{; 1 apaa ppPAa =∈),,(),( iiiii emcdPr →
B. Rinner 27
Self-aware Configuration
• Adopted from proprioceptive computing systems – use proprioceptive sensors to monitor “one self”
(concept from psychology, robotics/prosthetics, …, fiction)
– reason about their behavior (self-awareness)
– effectively and autonomously adapt their behavior to changing conditions (self-expression)
• Demonstrate autonomous multi-object tracking in camera network – Exploit single camera object detector & tracker
– Perform camera handover
– Learn camera topology
B. Rinner 28
Bid C4
Virtual Market-based Handover
• Initialize auctions for exchanging tracking responsibilities – Cameras act as self-interested agents, i.e., maximize their own utility
– Selling camera (where object is leaving FOV) opens the auction
– Other cameras return bids with price corresponding to “tracking” confidence
– Camera with highest bid continues tracking; trading based on Vickrey auction
Camera 1 Camera 2
Camera 3
Camera 4
Init auction
Bid C3
Fully distributed approach no a-priori topology knowledge required
B. Rinner 29
Market-based Tracking Handover
• Utility function (each camera)
rpjvcOUiOj
ijjii +−Φ⋅⋅= ∑∈
)]([)(
tracking decision visibility confidence payments made payments received
Simulation green: tracking yellow: shared FOV red: trading (handover)
B. Rinner 30
Tracking Performance
• Tradeoff between utility and communication effort
Scenario 1 (5 cameras, few objects) Scenario 2 (15 cameras, many objects)
• Emerging Pareto front [Esterle et al. Socio-Economic Vision Graph Generation and Handover in Distributed Smart Camera Networks. ACM Trans. Sensor Networks. 2013]
B. Rinner 31
Learn Neighborhood Relationships
• Gaining knowledge about the network topology (vision graph) by exploiting the trading activities
• Temporal evolution of the vision graph
B. Rinner 32
Learning Heterogeneous Strategies
• Heterogeneous strategies at cameras may improve Pareto front
• Adapt camera behaviour by online learning using bandit solvers
Homogeneous vs. heterogeneous handover strategies (offline)
Online learning strategies with different bandit solvers
[Lewis et al. Learning to be different: Heterogeneity and Efficiency in Distributed Smart Camera Networks. In Proc. IEEE SASO. 2013]
B. Rinner 33 33 33
Applications
B. Rinner 34 34 34
#1 Trustworthy Cameras
• Smart cameras – Highly capable embedded systems (on-board video analysis)
– Large software stacks
– Networked devices using closed (CCTV) and public networks
• Applications no longer only in public but also in private areas (assisted living, home monitoring, …)
• Protection of sensitive image data – Protection against manipulation (e.g., enforcement applications;
evidence at court)
– Privacy of monitored people
B. Rinner 35 35 35
Goals and Assumptions
• We present a system level approach that addresses the following security issues: – Integrity: detect manipulation of image and video data
– Authenticity: provide evidence about the origin of image and videos
– Confidentiality: make sure that privacy sensitive image data cannot be accessed by an unauthorized party
– Multi-level Access Control: support different abstraction levels and enforce access control for confidential data
• Security and privacy protection as inherent features of the camera
• Considered attack types: only software attacks [Winkler, Rinner. Securing Embedded Smart Cameras with Trusted Computing.
EURASIP Journal on Wireless Communications and Networking, 2011 ]
Approach
• Bringing of Trusted Computing concepts into cameras • Trusted Platform Modules (TPMs) are well defined, readily
available and cheap
• TC is an open industry standard • TPMs are available from many manufacturers
B. Rinner 36
Hardware Security Anchor
37
• Trusted Platform Module (TPM) at a glance – Secure storage for cryptographic keys – Data encryption, digital signatures – System status monitoring and reporting (measurement + attestation) – Unique platform ID
Security Chip (TPM) Image Sensor CPU RAM
Bootloader
Operating System (e.g., Embedded Linux)
Software Libraries and Middleware
Image Processing and Analysis Communication …
Software
Hardware
B. Rinner
Implemented Security Features
38
• Trusted boot where camera software stack is “measured” and the status is securely reported to operator
• Integrity and authenticity guarantees using non-migratable, TPM-protected RSA keys
• Freshness/timestamping for outgoing images via TPM-protected tick (counter) sessions
B. Rinner
B. Rinner 39 39 39
Hardware Prototype
• TI OMAP 3530 CPU: ARM @ 480MHz and
DSP @ 430MHz
• 256MB RAM, SD-Card as mass storage
• VGA color image sensor
• wireless: 802.11b/g WiFi
and 802.15.4 (XBee)
• LAN via USB (primarily used for debugging)
• Atmel hardware TPM
on I2C bus
Privacy Protection Approaches
40
• Protection as an inherent feature of the camera
• Object-based protection: Identification of sensitive data (e.g., human faces)
• Data abstraction and obfuscation
• Global protection techniques: Uniform protection of entire
frames (insensitive to misdetections of computer vision)
B. Rinner
Multi-Level Protection
41
• Video stream contains sub streams • Every sub stream is encrypted
– Hardware-bound cryptographic keys
• Recovery of identities only via four eyes principle
Video Stream Smart Camera Sub Streams
B. Rinner
High-Level Processing Flow
42 B. Rinner
B. Rinner 43 43 43
Privacy-aware Camera Networks
• What about users (i.e., monitored people)? • users usually do not care much about integrity, authenticity of time
stamping
• users (hopefully!) care about confidentiality and privacy!
• Question 1: How can we increase privacy awareness?
• Question 2: How can we demonstrate that (our) cameras protect the privacy of users?
B. Rinner 44 44 44
Raising Privacy Awareness
• Let users know if there are cameras in their environment
• Use user's handheld (e.g., smart phone) for location-based notifications
User Feedback
• Goal: Trustworthy feedback to monitored persons about camera’s privacy protection
• Visual communication for authentication – Direct line of sight – Intuitive way to select intended camera
• Operator discloses applications to TrustCenter
T. Winkler and B. Rinner, “User Centric Privacy Awareness in Video Surveillance,” Multimedia Systems Journal, vol. 18, no. 2, pp. 99–121, 2012.
B. Rinner
Attestation Report
46 B. Rinner B. Rinner
B. Rinner 47
#2 Aerial Cameras for Disaster Mgm.
• Develop autonomous multi-UAV system for aerial reconnaissance
• Up-to-date aerial overview images are helpful in many situations: “Google Earth with up-to-date images in high resolution”
• Small-scale quadcopter platform with onboard sensors and computation
• GPS receiver for autonomous waypoint flights
• Generic framework not bound to specific UAV
B. Rinner 48
Key Challenges
• Increase autonomy – Control and coordination of multiple UAVs
– High-level interaction with user
• Provide prompt response to user – Provide preliminary results fast and improve over time
• Deal with strong resource limitations – Flight time, payload, computation and communication
– Limited sensing capabilities
B. Rinner 49
Autonomous UAV Operation
Mission Planning Flight
Real-World
Simulator
Image Analysis
scenario specification
waypoints
Single/multiple UAV
captured image/video
stiching, detection
user interface
B. Rinner 50
Key Questions
• How to generate and update movement routes for the UAVs? – Achieve multiple optimization goals
– Deal with changes in the environment
• How to setup a wireless UAV network? – Provide networking coverage
• How to generate the mosaic image? – Apply incremental image stiching
– Combine RGB and thermal images
• System integration and demonstration
51
Demonstration Video
B. Rinner
B. Rinner. SCVSN Tutorial (Chapter 3) 52 52 52
Research Directions of Visual Sensor Networks
B. Rinner. 53
#1: Architecture
• Low-power (high performance) camera nodes – Dedicated platforms: vision processors, PCBs, systems
– Many examples: CITRIC, NXP
• Visual/Multimedia Sensor Networks – Topology and (multi-tier) architecture
– Multi-radio communication
• Dynamic Power Management – For sensing, processing and communication
How to design resource-aware nodes and networks
B. Rinner. 54
#2: Networking
• Ad hoc, p2p communication over wireless channels – Providing RT and QoS
– Eventing and/or streaming
• Dynamic resource management – (local) computation, compression, communication, etc.
– Degree of autonomy: dynamic, adaptive, self-organizing
– Fault tolerance, scalability
– Network-level software, middleware
How to process and transfer data in the network
B. Rinner 55
#3: Deployment, Operation, Maintenance
• Development support for applications – Model/simulate the application (function, resources, QoS)
– Reuse/exchange of software/libraries
– Software updates, debugging etc.
• Autonomous calibration and scene adaption – Avoid manual procedures
– Adapt to different scenes and settings
• Network configuration
Consider the entire life cycle of the camera network
B. Rinner. 56
#4: Distributed Sensing & Processing
• Sensor placement, calibration & selection – Optimization problem
– Distributed approaches eg., consensus, game theory, multi-agent systems
• Compressive Sensing
• Collaborative data analysis – Multi-view, multi-temporal, multi-modal
– Sensor fusion
• Online/real-time processing – Can not effort to store large amounts of data
Where to place sensors and analyze the data
B. Rinner 57
#5: Mobility
• Mobile cameras are ubiquitous – PTZ, vehicles, robotics etc.
– Mobile phones
• Advanced vision algorithms – Ego motion, online calibration
– Closed-loop control, active vision
How to exploit networks of mobile cameras
B. Rinner 58
#6: Usability
• Ease of deployment, maintenance – Self-* functionality
– “Smart cameras for dumb people”
• Privacy and Security – Trust of the user
– Control the privacy setting
• Interaction with the camera network
How to provide useful services to people
B. Rinner 59
#7: Applications
• Demonstrations – Large scale networks eg., for surveillance
– Small scale networks eg., for entertainment, home environments
– Only single camera application?
• Market opportunities
• Killer Application
What applications can (only) be solved by DSC
B. Rinner. SCVSN Tutorial (Chapter 3) 60
Summary
• VSNs exploit various advantages of distributed camera sensors such as increased coverage, redundancy and 3D information.
• Distributed cameras impose various challenges such as huge amount of data, required infrastructure and (network) topology.
• VSN have unique characteristics (wireless sensor networks vs. surveillance camera networks)
• Current research addresses signal processing, communications, architecture and middleware issues.
B. Rinner 61
Acknowledgements & Further Info
Pervasive Computing @ AAU UAV Research http://pervasive.aau.at http://uav.aau.at
• Tutorial site Most recent course material is available at
http://pervasive.aau.at/S5-tutorial