intel labs self localizing sensors and actuators on distributed computing platforms vikas raykar...

Intel LabsIntel Labs

Self Localizing sensors Self Localizing sensors and actuators on and actuators on Distributed Computing Distributed Computing PlatformsPlatforms

Vikas RaykarVikas Raykar

Igor KozintsevIgor Kozintsev

Rainer LienhartRainer Lienhart

MotivationMotivation Many multimedia applications are emerging which use multiple audio/video sensors and actuators.

Microphones

Cameras

Speakers

Displays

Dis

trib

ute

d

Cap

ture

Dis

trib

ute

d

Ren

der

ing

Other Applications

Number Crunching

ApplicationsApplications

Audio/Video Surveillance

Hands free voice communication

MultiChannel Speech Enhancement

Smart ConferenceRooms

Audio/Image Based Rendering

Object LocalizationAnd tracking

Meeting Recording

Distributed AudioVideo Capture

Interactive Audio Visual Interfaces

MultiChannel EchoCancellation

Speech Recognition

Source separation andDeverberation

Additional MotivationAdditional Motivation Current work has focused on setting up all the sensors and actuators on a single dedicated computing platform.

Dedicated infrastructure required in terms of the sensors, multi-channel interface cards and computing power.

Computing devices such as laptops, PDAs, tablets, cellular phones, camcorders have become pervasive.

Audio/video sensors on different laptops can be used to form a distributed network of sensors.

On the other hand…

Problem formulationProblem formulationPut all the distributed audio-visual I/O capabilities into a common time and space.

In this paper:Focus on providing a common space by means of actively estimating the 3D positions of the sensors (microphones) and actuators (speakers).

Account for the errors due to lack of temporal synchronization among various sensors and actuators (A/Ds and D/As) on distributed general purpose computing platforms.

Our View of Distributed Our View of Distributed Sensor NetworkSensor Network

X

Y

Z

Localization with known Localization with known positions of speakerspositions of speakers

Distances are not exact

There are more speakers

If positions of speakers are If positions of speakers are unknown…unknown…

Consider M Microphones and S speakers.What can we measure?

Distance between each speaker and all microphones (Time Of Flight)

MxS TOF matrix

Assume TOF corrupted by AWGN: can derive the ML estimate.

Calibration signal

Nonlinear Least SquaresNonlinear Least Squares

Find the coordinates which minimizes this

Reference Coordinate SystemReference Coordinate System

X axis

Positive Y axis

OriginSimilarly in 3D

1.Fix origin (0,0,0)

2.Fix X axis

(x1,0,0)

3.Fix Y axis

(x2,y2,0)

4.Fix positive Z axis

x1,x2,y2>0

Which to choose? Later…

Intel LabsIntel Labs

On a synchronized platform all is On a synchronized platform all is well..well..

However on a Distributed However on a Distributed system..system..

Intel LabsIntel Labs

PC platform overviewPC platform overview

PCI SlotsPCI Slots

CPUCPU

AG

PA

GP MCH

ICH

ATAATA

LAN LAN

USBUSB

AC97AC97ICH, hub,

PCI, LAN, etc.

CPU, MCH, FSB, memory

Operating system

Multimedia/multistream applications

Audio/video I/O devices

I/O bus

t

t

jtsSignal Emitted by source j

Signal Received by microphone i

ijFOT ˆ

itmijTOF

Capture Started

Playback Started

Time Origin

Timing on distributed systemTiming on distributed system

Speaker Emission Start Times

S

Microphone Capture Start Times

M -1Assume tm_1=0

Microphone and speakerCoordinates

DM+DS - [ D(D+1)/2 ]

MS TOF Measurements

Joint EstimationJoint Estimation

Formulation same as above but less number of parameters.

Time Difference of Arrival (TDOA)Time Difference of Arrival (TDOA)

Levenberg Marquadrat method

Multidimensional function.

Unless we have a good initial guess may not convergeto the global minima.

Approximate initial guess required.

Nonlinear least squaresNonlinear least squares

dot product matrixSymmetric positive definiterank 3

Given B can you get X ?....Singular Value Decomposition

Multi Dimensional ScalingMulti Dimensional Scaling

Clustering approximationClustering approximation

i i

j i

j j

i j

Clustering approximationClustering approximation

k

ijd

kjd

kid

i

j

How to get dot product from the How to get dot product from the pair wise distance matrixpair wise distance matrix

Later shift it to our

orignal reference

Slightly perturb each location of GPCinto two to get the initial guess for the microphone and speaker coordinates

Centroid as the originCentroid as the origin

Sample result in 2DSample result in 2D

ApproxDistance matrixbetween GPCs

Approxts

Approx tm

Clustering

Dot product matrix

Dimension and coordinate system

MDS to get approx GPC locations

perturb

TOF matrix

Approx. microphone and speaker

locations

TDOA basedNonlinear

minimization

Microphone and speakerlocations tm

AlgorithmAlgorithm

Gives the lower bound on the variance of any unbiased estimator.

Does not depends on the estimator. Just the data and the noise model.

Basically tells us to what extent the noise limits our performance i.e. you cannot get a variance lesser than the CR bound.

Jacobian

Rank deficit: remove theknown parameters

Cramer-Rao boundCramer-Rao bound

Performance comparisonPerformance comparison

Dependence on number of nodesDependence on number of nodes

Dependence on number of nodesDependence on number of nodes

Geometry mattersGeometry matters

Geometry mattersGeometry matters

Mic 3

Mic 1

Mic 2

Mic 4

Speaker 1

Sp

eake

r 4S

pea

ker

2

Speaker 3

X

Z

Roo

m L

engt

h =

4.2

2 m

Room Width = 2.55 m

Room Height = 2.03 m

Experimental setup: bias 0.08 cm Experimental setup: bias 0.08 cm sigma 3.8 cmsigma 3.8 cm

Intel LabsIntel Labs

SummarySummary General purpose computers can be used for General purpose computers can be used for

distributed array processingdistributed array processing It is possible to define common time and space for a It is possible to define common time and space for a

network of distributed sensors and actuators.network of distributed sensors and actuators. For more information please see our two papers in For more information please see our two papers in

ACM MM in November or contact ACM MM in November or contact igor.v.kozintsev@intel.com igor.v.kozintsev@intel.com rainer.lienhart@intel.comrainer.lienhart@intel.com

Let us know if you will be interested in testing/using Let us know if you will be interested in testing/using out time and space synchronization software for out time and space synchronization software for developing distributed algorithms on GPC (available developing distributed algorithms on GPC (available in November)in November)

Intel LabsIntel Labs

BackupBackup

Calibration signalCalibration signal

Results (contd.)Results (contd.)