sensor-level privacy for thermal cameras - focus...

Sensor-level Privacy for Thermal Cameras

Francesco Pittaluga

Aleksandar Zivkovic

Sanjeev J. Koppal

University of Florida

Imaging and Tracking People

Surveillance Military

Gaming IoT Mobile

2Intro Digitization Noise Exposure Conclusion

Balancing Privacy and Utility

Hospitals Schools

Retirement homes Workplaces


Conventional Privacy Processing

Scene

Camera

ComputerStorageSecure

Data

4

Post-capture processing Lock data with cryptography

Edit images computationally

Boyle 2000, Sweeney 2002, Johnson et al. (IETF) 2003,

Gross et al. 2009, Agrawal and Narayanan 2011, …

Intro Digitization Noise Exposure Conclusion

Conventional Privacy Processing

Scene

Camera

ComputerStorageSecure

Data

Post-capture strategy has an inherent vulnerability


Privacy Preserving Computational Cameras

Scene

Only capture light-field samples that are needed

Computational Camera

Secure

Data


Computational Camera: Three Approaches

SceneComputational Camera

Secure

DataComputerStorage

X Pattern recognition

in every frameX Special

OpticsX New sensor

7

Chattopadhyay and Boult 2007, Winkler and Renner 2010, Narayanan and Mrityunjay 2011

Nelson et al. 2005,Winkler et al. 2014, Fernandez-Berni et al. 2014

Three Problems Limited Adoption

Zhang et al, 2014, Pittaluga and Koppal 2015


Our Idea: Sensor-level Privacy for Thermal Cameras

Scene

Our idea: use a thermal sensor

(Wavelengths > 3µm)

Computational Camera

Secure

DataComputerStorage

8

Reliable Physics

Based Processing Off the Shelf

Optics

Thermal

sensor


Thermal Cameras are Coming Soon!

Thermal cameras are not exotic anymore

Thermal face recognition works

FLIR One 80x60

~$250

Melaxis 16x4

~$75FLIR A6751SC


Three Sensor Level Approaches

SceneComputational Camera

Secure

Data

Thermal Sensor



Thermal Sensor

Digitization Noise Exposure



Thermal Sensor



Humans are broadband

13

Spectral Power Response vs Wavelength

8

11

7 14

Human Spectral Response

r(λ)

(μm))

𝑊

𝑚2 𝑠𝑟 𝜇𝑚


Facial skin temperature reaches an equilibrium

37 Degrees

(Celsius)

Outside temperature known

= T Celsius

Facial skin temperature F (T, 37)


This mapping is known

Olesen and Parsons 2002

15

Skin

Tem

pera

ture

°C

Ambient Temperature °C


Removing pixels in this range during digitization

ASIC Modification

16

Pixel

Readout

Output

Voltage

A-to-D

Masking measurements

based on temp. range

16-bit

decoder

Lower

Voltage

Upper

VoltageUpper Bound

Comparator

Lower Bound

Comparator

AND

AND

NOT

Seq. Counter

Random-Access

Memory

Address (18.0)

Data(15.0)


Digitization Result



Thermal Sensor



Adding noise to the bolometer

20

GFID

GSK

VCC

Blind Bolometer

Active Bolometer

Vbus

Vout

Cint

Tunable Bias

Voltages


The effect of bias voltages

21

GSK

GFID

5 V

5 V

0 V

0 V

GSK

GFID

5 V

5 V

0 V

0 V

GSK

GFID

5 V

5 V

0 V

0 V

GSK

GFID

5 V

5 V

0 V

0 V

Exposure 5 Exposure 15

Exposure 25 Exposure 75


Calibrating for noise and privacy

22

GSK

GFID

5 V

5 V

0 V

0 V

9000

0

0 120Grayscale values

Histogram of values for the highest standard deviation

Occurrences

2𝜎 = 14 graylevels

Bias voltages with exposure set to 5

For a flat lambertian plane


Noise Result: Head Tracking



Thermal Sensor



Temperature and radiant power

25


8

11

7 14

s(λ)

Camera sensitivity

(μm)

𝑊


Pixel Number vs Radiant Power

Φ𝑐𝑜𝑙𝑑 Φℎ𝑜𝑡

𝐼max~

𝐼min~

𝐼max

Φ = λ𝑡

λℎ

𝑠 λ 𝑟 λ 𝑑λ


r(λ)

Φℎ𝑢𝑚𝑎𝑛

r(λ)r(λ)


No capture region

26


8

11

7 14

Φ𝑚𝑖𝑛 = λ𝑡

λℎ

𝑠 λ [𝑟 λ −∆ 𝜆

2] 𝑑λ

Φ𝑚𝑎𝑥 = λ𝑡

λℎ

𝑠 λ [𝑟 λ +∆ 𝜆

2] 𝑑λ

(μm)

𝑊



Φ𝑚𝑖𝑛 Φ𝑚𝑎𝑥

𝐼max

“No

Ca

ptu

re”

s(λ)

Camera sensitivity

Δ(𝜆)Human Spectral Response

r(λ)


Exposures that remove no capture region

27


8

11

7 14

s(λ)

Camera sensitivity

(μm)

𝑊


Underexposure

Overexposure


Φ𝑚𝑖𝑛 Φ𝑚𝑎𝑥

𝐼max~

𝐼min~

𝐼max

“No

Ca

ptu

re”

𝑡 ≥𝑔 𝐼𝑚𝑎𝑥(𝑔)

Φ𝑚𝑖𝑛

𝑡 ≤𝑔 𝐼𝑚𝑖𝑛(𝑔)

Φ𝑚𝑖𝑛

Human Overexposed Human Underexposed


r(λ)


Optimal algorithm to obtain exposures

28

Grossberg and Nayar 2003

𝝽(𝑛, 𝑇) = Γ𝑚𝑖𝑛

Φ𝑚𝑖𝑛

ℎ𝑑𝑒𝑠′ − ℎ′ 𝑝𝜔𝑑Φ +

Φ𝑚𝑎𝑥

Γ𝑚𝑎𝑥

ℎ𝑑𝑒𝑠′ − ℎ′ 𝑝𝜔𝑑Φ 𝜔 =

0 ℎ𝑑𝑒𝑠′ (Φ) < ℎ′(Φ)

1 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒,

…

Error Function Binary Weights

1. 𝑇𝑖 > 0

2. ∀ 𝑇𝑖 𝑇𝑖 ≥ 𝐼𝑚𝑎𝑥

Φ𝑚𝑖𝑛⊕ 𝑇𝑖 ≤

𝐼𝑚𝑖𝑛

Φ𝑚𝑎𝑥

𝑎𝑟𝑔𝑚𝑖𝑛 𝑛,𝑇 𝝽(𝑛, 𝑇) 𝑠. 𝑡.


Optimal algorithm to obtain exposures

Algorithm assumes a single no capture region

Brute force search is therefore tractable

29

Objective function score

Optimal solution for

112 exposures

Grid search index

HDR Image


HDR Results

Over Under Fusion


HDR Results


Method Comparison


Comparison


• Low noise

• Good image

quality

• Real-time

• Hardware and

firmware upgrades

• Low noise

• No hardware

modification

• Good image quality

• Multiple Images and

more capture time

• Real-time

• No hardware

modification

• Low image quality

• Noisy


Future Work

Pilot deployment program of private sensors at

UF Health Shands Hospital.

Generate database of private face images to for

privacy challenge.

Generate database of private videos for activity

recognition in a hospital setting.


DHS

Sanjeev Koppal Andreas Enqvist

This material is based upon work supported by theU.S. Department of Homeland Security under Grant Award

Number, 2014-DN-077-ARI083-01. The views and conclusionscontained in this document are those of the authors

and should not be interpreted as necessarily representing theofficial policies, either expressed or implied, of the U.S. Department

of Homeland Security.

Acknowledgements


Summary: Three Sensor Level Approaches



sensor-level privacy for thermal cameras - focus...

Documents