Thermal-Depth Fusion for Occluded Body Skeletal Posture Estimation

Shane Transue, Phuc Nguyen, Tam Vu, and Min-Hyung Choi

University of Colorado

IEEE Conference on Connected Health: Applications,

Systems and Engineering Technologies

NON-CONTACT RESPIRATORY MONITORING

Sleep-based Supervised Respiration

Rate, tidal volume, apnea, COPD

Automated Radar Solutions

Orthogonal radar monitoring

Region-based chest movements

Automated Camera Solutions

Monitors chest surface deformations

Computes changes in volume/behavior

Automated Solutions Require Posture

Patient chest orientation

Occlusion detection

[P. Nguyen et al, IEEE INFOCOM’16]

[S. Transue et al, IEEE/ACM CHASE’16]

Camera-based Tidal-volume Estimation

Radar-based Tidal-volume Estimation

2/20

DEPTH AND THERMAL POSTURE ESTIMATION

Occluded Skeletal Tracking

Problem: Identifying occluded joints Blankets, clothing, hide joint positions

Depth-image is ambiguous

No ground-truth training/labeling/scoring

Prior: Depth-based Skeletal Tracking

Prior: Thermal Posture Imaging (low)

3/20

Blanket induced occluded body (example)

[F. Achilles et al., MICCAI’16]

THERMAL DEPTH FUSION IMAGING

Core Idea: Fuse Depth + Thermal

Depth for 3D surface reconstruction

Thermal for patient heat tracking

Monitoring Devices:

Microsoft Kinect2 (512x424@30[fps])

FLIR C2 (80x60@15[fps])

Alignment Bracket Prototype Device and Experimental Design

4/20

MODELING THERMAL VOLUME POSTURE

Occlusion Implications Ambiguous Depth + Thermal data

Partial skeletal data (c)

Disconnected skeletal components (c)

Model Assumptions

Predefined Skeletal Structure (b)

Enclosed volume (patient on surface)

Thermal volume reconstruction (a)

5/20 IEEE/ACM Chase 2017

THERMAL POSTURE GROUND-TRUTH

Occluded Skeletal Tracking Visual markers are occluded

Skeletal structure may be incomplete

Require a method for capturing thermal markers

Solution: Thermal Motion Tracking Borrows from traditional motion-capture

Markers are defined by thermal spheres

Interchangeable Thermal Suit

Thermal Posture Tracking

(training only)

6/20

Heated markers

Markers are detachable

Flexible Design

Fixed joint count

DEPTH + THERMAL POSTURE MODELING (1)

7/20

Depth + Thermal

Fusion

Proposed thermal posture estimation: Thermal + Depth to CNN Classification

DEPTH + THERMAL POSTURE MODELING (2)

8/20

Depth + Thermal

Fusion

TEGI Heat

Propagation

Proposed thermal posture estimation: Thermal + Depth to CNN Classification

DEPTH + THERMAL POSTURE MODELING (3)

9/20

Depth + Thermal

Fusion

Thermal Volume

Reconstruction

TEGI Heat

Propagation

Proposed thermal posture estimation: Thermal + Depth to CNN Classification

DEPTH + THERMAL POSTURE MODELING (4)

10/20

Depth + Thermal

Fusion

Thermal Volume

Reconstruction

Occluded Estimate

Posture

TEGI Heat

Propagation

Proposed thermal posture estimation: Thermal + Depth to CNN Classification

THERMAL VOLUME RECONSTRUCTION

Posture Volume Assumptions Posture is enclosed

Human Body is a Connected-component

Propagate from known location (head)

Generate internal structure (enclosed volume)

Solution: Thermal Sphere Hierarchy

Sphere-packing

Boundary Conditions(1) Surface Boundary

(2) Thermal threshold

11/20

Infrared Image of Posture

(enclosed volume under surface)

2D Thermal Sphere Packing

2D-3D INVERSE THERMAL PROPAGATION

How can we map 2D surface thermal information to 3D voxels?

Solution: Thermal Extended Gaussian Images

Maps 2D thermal data to 3D volumes

Parametrized by distance, heat, etc.

Computed by spherical projection

12/20

TEGI Point-to-volume Mapping

3D THERMAL VOLUME RESULT

13/20

Volume Reconstruction Pipeline

(1) Surface thermal-cloud

(2) Volume enclosure

(3) Sphere-packing and heat propagation

(4) Voxel-grid representation

Depth + Thermal Enclosed Volume Heat Propagation Thermal Volume

3D THERMAL MONITORING (VIDEO)

14/20

POSTURE MONITORING APPLICATION

15/20

LABELING, TRAINING, AND CLASSIFICATION

Training: Correlate skeletal posture to volumetric thermal data

Volumetric data provided as 3D image to CNN

Classification based on 3D distribution

Coarse-grain posture from classifications

16/20

Training Components (training-data + labeling) Runtime data

POSTURE CLASSIFICATION RESULTS

17/20

Note: Classification labels correspond to confusion matrix results

CLASSIFICATION RESULTS

Standard Posture Confusion Matrices

18/20

Patient-specific training/classification Cross-patient training/classification

CONCLUSION AND FUTURE WORK

Conclusion

Fuse Depth + Fusion for occluded patient tracking 3D Patient heat distribution

Occluded skeletal estimation

Sleep-study patient tracking/posture Automated respiratory monitoring

Long-term studies

Future Work Feature-based Training (RDF)

Improve image resolution

Address challenges/ambiguity

Other Depth + Thermal applications

19/20

THANK YOU