intel labs self localizing sensors and actuators on distributed computing platforms vikas raykar...

35
Intel Labs Intel Labs Self Localizing Self Localizing sensors and actuators sensors and actuators on on Distributed Computing Distributed Computing Platforms Platforms Vikas Raykar Vikas Raykar Igor Igor Kozintsev Kozintsev Rainer Rainer Lienhart Lienhart

Upload: rosamund-jacobs

Post on 12-Jan-2016

228 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Intel LabsIntel Labs

Self Localizing sensors Self Localizing sensors and actuators on and actuators on Distributed Computing Distributed Computing PlatformsPlatforms

Vikas RaykarVikas Raykar

Igor KozintsevIgor Kozintsev

Rainer LienhartRainer Lienhart

Page 2: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

MotivationMotivation Many multimedia applications are emerging which use multiple audio/video sensors and actuators.

Microphones

Cameras

Speakers

Displays

Dis

trib

ute

d

Cap

ture

Dis

trib

ute

d

Ren

der

ing

Other Applications

Number Crunching

Page 3: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

ApplicationsApplications

Audio/Video Surveillance

Hands free voice communication

MultiChannel Speech Enhancement

Smart ConferenceRooms

Audio/Image Based Rendering

Object LocalizationAnd tracking

Meeting Recording

Distributed AudioVideo Capture

Interactive Audio Visual Interfaces

MultiChannel EchoCancellation

Speech Recognition

Source separation andDeverberation

Page 4: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Additional MotivationAdditional Motivation Current work has focused on setting up all the sensors and actuators on a single dedicated computing platform.

Dedicated infrastructure required in terms of the sensors, multi-channel interface cards and computing power.

Computing devices such as laptops, PDAs, tablets, cellular phones, camcorders have become pervasive.

Audio/video sensors on different laptops can be used to form a distributed network of sensors.

On the other hand…

Page 5: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Problem formulationProblem formulationPut all the distributed audio-visual I/O capabilities into a common time and space.

In this paper:Focus on providing a common space by means of actively estimating the 3D positions of the sensors (microphones) and actuators (speakers).

Account for the errors due to lack of temporal synchronization among various sensors and actuators (A/Ds and D/As) on distributed general purpose computing platforms.

Page 6: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Our View of Distributed Our View of Distributed Sensor NetworkSensor Network

X

Y

Z

Page 7: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Localization with known Localization with known positions of speakerspositions of speakers

Distances are not exact

There are more speakers

Page 8: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

If positions of speakers are If positions of speakers are unknown…unknown…

Consider M Microphones and S speakers.What can we measure?

Distance between each speaker and all microphones (Time Of Flight)

MxS TOF matrix

Assume TOF corrupted by AWGN: can derive the ML estimate.

Calibration signal

Page 9: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Nonlinear Least SquaresNonlinear Least Squares

Find the coordinates which minimizes this

Page 10: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Reference Coordinate SystemReference Coordinate System

X axis

Positive Y axis

OriginSimilarly in 3D

1.Fix origin (0,0,0)

2.Fix X axis

(x1,0,0)

3.Fix Y axis

(x2,y2,0)

4.Fix positive Z axis

x1,x2,y2>0

Which to choose? Later…

Page 11: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Intel LabsIntel Labs

On a synchronized platform all is On a synchronized platform all is well..well..

Page 12: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

However on a Distributed However on a Distributed system..system..

Page 13: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Intel LabsIntel Labs

PC platform overviewPC platform overview

PCI SlotsPCI Slots

CPUCPU

AG

PA

GP MCH

ICH

ATAATA

LAN LAN

USBUSB

AC97AC97ICH, hub,

PCI, LAN, etc.

CPU, MCH, FSB, memory

Operating system

Multimedia/multistream applications

Audio/video I/O devices

I/O bus

Page 14: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

t

t

jtsSignal Emitted by source j

Signal Received by microphone i

ijFOT ˆ

itmijTOF

Capture Started

Playback Started

Time Origin

Timing on distributed systemTiming on distributed system

Page 15: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Speaker Emission Start Times

S

Microphone Capture Start Times

M -1Assume tm_1=0

Microphone and speakerCoordinates

DM+DS - [ D(D+1)/2 ]

MS TOF Measurements

Joint EstimationJoint Estimation

Page 16: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Formulation same as above but less number of parameters.

Time Difference of Arrival (TDOA)Time Difference of Arrival (TDOA)

Page 17: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Levenberg Marquadrat method

Multidimensional function.

Unless we have a good initial guess may not convergeto the global minima.

Approximate initial guess required.

Nonlinear least squaresNonlinear least squares

Page 18: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

dot product matrixSymmetric positive definiterank 3

Given B can you get X ?....Singular Value Decomposition

Multi Dimensional ScalingMulti Dimensional Scaling

Page 19: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Clustering approximationClustering approximation

Page 20: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

i i

j i

j j

i j

Clustering approximationClustering approximation

Page 21: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

k

ijd

kjd

kid

i

j

How to get dot product from the How to get dot product from the pair wise distance matrixpair wise distance matrix

Page 22: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Later shift it to our

orignal reference

Slightly perturb each location of GPCinto two to get the initial guess for the microphone and speaker coordinates

Centroid as the originCentroid as the origin

Page 23: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Sample result in 2DSample result in 2D

Page 24: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

ApproxDistance matrixbetween GPCs

Approxts

Approx tm

Clustering

Dot product matrix

Dimension and coordinate system

MDS to get approx GPC locations

perturb

TOF matrix

Approx. microphone and speaker

locations

TDOA basedNonlinear

minimization

Microphone and speakerlocations tm

AlgorithmAlgorithm

Page 25: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Gives the lower bound on the variance of any unbiased estimator.

Does not depends on the estimator. Just the data and the noise model.

Basically tells us to what extent the noise limits our performance i.e. you cannot get a variance lesser than the CR bound.

Jacobian

Rank deficit: remove theknown parameters

Cramer-Rao boundCramer-Rao bound

Page 26: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Performance comparisonPerformance comparison

Page 27: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Dependence on number of nodesDependence on number of nodes

Page 28: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Dependence on number of nodesDependence on number of nodes

Page 29: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Geometry mattersGeometry matters

Page 30: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Geometry mattersGeometry matters

Page 31: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Mic 3

Mic 1

Mic 2

Mic 4

Speaker 1

Sp

eake

r 4S

pea

ker

2

Speaker 3

X

Z

Roo

m L

engt

h =

4.2

2 m

Room Width = 2.55 m

Room Height = 2.03 m

Experimental setup: bias 0.08 cm Experimental setup: bias 0.08 cm sigma 3.8 cmsigma 3.8 cm

Page 32: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Intel LabsIntel Labs

SummarySummary General purpose computers can be used for General purpose computers can be used for

distributed array processingdistributed array processing It is possible to define common time and space for a It is possible to define common time and space for a

network of distributed sensors and actuators.network of distributed sensors and actuators. For more information please see our two papers in For more information please see our two papers in

ACM MM in November or contact ACM MM in November or contact [email protected] [email protected] [email protected]@intel.com

Let us know if you will be interested in testing/using Let us know if you will be interested in testing/using out time and space synchronization software for out time and space synchronization software for developing distributed algorithms on GPC (available developing distributed algorithms on GPC (available in November)in November)

Page 33: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Intel LabsIntel Labs

BackupBackup

Page 34: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Calibration signalCalibration signal

Page 35: Intel Labs Self Localizing sensors and actuators on Distributed Computing Platforms Vikas Raykar Igor Kozintsev Igor Kozintsev Rainer Lienhart

Results (contd.)Results (contd.)