non-intrusive apnoea / hypopnoea detection...
TRANSCRIPT
Non-intrusive Apnoea / Hypopnoea Detection System
via MS Kinect captured Depth Video Analysis
Cheng Yang#, Yu Mao∗, Gene Cheung∗, Vladimir Stankovic#, Kevin Chan%
#Department of Electronic and Electrical Engineering, University of Strathclyde, Glasgow, UK
∗The Graduate University for Advanced Studies, National Institute of Informatics, Tokyo, Japan
%School of Medicine, University of Western Sydney, Camden and Campbelltown Hospitals, Sydney, Australia
Simon Fraser University, Nov 13, 2014
1
Motivation Sleep-disordered breathing is common in the general population.
Repeated episodes of apnoea / hypopnoea can significantly disturb a person’s sleep.
Existing sleep monitoring systems:
1. Vibration-sensing wrist-bands (Fitbit, Jawbone UP) – Minimally intrusive; Mostly record sleep time only.
2. Full multi-sensing monitoring device (Philips Alice PDx) – Accurate in detecting vital signs; expensive and intrusive.
http://static1.businessinsider.com/image/51a8b6246bb3f7df3c000009-1200/the-fitbit-
has-a-better-sleep-tracker.jpg
http://cdn0.vox-cdn.com/entry_photo_images/7275017/DSC_3047-hero_verge_medium_landscape.jpg
http://www.newscenter.philips.com/pwc_nc/main/standard/resources/corporate/press/2010/siesta/Siesta_3_small.jpg
2
Related Work
A previous non-intrusive sleep monitoring system w/ depth video:
M.-C. Yu et al., “Breath and position monitoring during sleeping with a depth camera,” in Int’l
Conference on Health Informatics, Vilamoura, Portugal, Feb 2012.
1. Torso movement detection - cannot distinguish chest/abdomen movements individually.
2. Not robust: mount a MS Kinect on the ceiling and measure the distance to the body – will not work if
the patient sleeps sideway.
We propose a dual-ellipse model (to be discussed) to detect chest/abdomen movements individually –
possible to track the breathing cycle even if the patient is sleeping in sideway.
3
Our Sleep Monitoring System
GOAL: a non-intrusive apnoea / hypopnoea detection system using
MS Kinect depth video.
System site: Bondi Junction Private Sleep Laboratory in Sydney,
Australia.
Targeted subjects: patients admitted for diagnostic sleep studies.
Operate in a completely dark room (active infrared sensing).
Unobstructed view of the patient’s upper body.
3 components:
1. 11-to-8-bit depth video coding.
2. Temporal depth video denoising.
3. Sleep-event classification.
pillow
patient
Kinect depth video
4C. Yang, G. Cheung, K. Chan, V. Stankovic, “Sleep Monitoring via Depth Video Recording & Analysis,” 5th IEEE
International Workshop on Hot Topics in 3D (Hot3D), Chengdu, China, July, 2014.
graph signal processing
Depth Video Coding
Given 11-bit depth images captured by MS Kinect 1.0, alternately encode 8 LSBs / MSBs as 8-bit images via H.264 video.
At receiver, recover missing 3 MSBs in each block using block-based motion estimation (overlapped bit matching / MV smoothness criteria).
MSB frame LSB frame
5C. Yang, G. Cheung, K. Chan, V. Stankovic, “Sleep Monitoring via Depth Video Recording & Analysis,” 5th IEEE
International Workshop on Hot Topics in 3D (Hot3D), Chengdu, China, July, 2014.
Microsoft Kinect
Temporal denoising & event classificationTwo tasks leveraging on recent advances in graph signal processing (GSP) [1]:
1. Graph-based temporal denoising:
Reduce temporal flicker in depth video via a graph-signal smoothness prior.
2. Graph-based classification:
Robustly classify apnoea / hypopnoea vs. normal breathing via graph-signal interpolation.
[1] D. I. Shuman, et al., “The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains,” IEEE Signal Processing Magazine, vol. 30, no.3, May 2013, pp. 83–98.
C. Yang, Y. Mao, G. Cheung, V. Stankovic, K. Chan, "Graph-based Depth Video Denoising and Event Detection for Sleep Monitoring," IEEE International Workshop on Multimedia Signal Processing, Jakarta, Indonesia, September, 2014.
6
Common thread: a piecewise smooth signal x has small graph Laplacainregularization term xx LT
Related Work (cont’d)
Spatial denoising based on Graph Fourier Transform (GFT), but not temporal:
W. Hu et al., “Depth map denoising using graph-based transform and group sparsity,” in IEEE MMSP, Italy, Oct 2013.
Data classification using GSP tools:
A. Sandryhaila and J. Moura, “Classification via regularization on graphs,” in IEEE GlobalSIP, Austin, TX, Dec 2013.
We propose a more intuitive and less complex graph-based classification for event detection.
7
Problem: temporal flicker across Kinect depth frames in time.
Proposal: Graph-based temporal denoising
Piecewise smoothness (PWS) of graph-signal x can be measured using , where L is graph Laplacian.
Signal prior: Motion vector (MV) field is PWS!
Draw graph for MVs of pixel blocks in denoised frame t-1 and noisy frame t.
Use in denoising objective as regularization term.
Temporal Depth Video Denoising
Frame t - 1 Frame t
Target blocksPredictor blocks
𝑤𝑖,𝑗 = exp −𝐯𝑖 − 𝐯𝑗 2
2
𝜎𝑣2
MV: 𝐯𝑖 = 𝑥𝑖 , 𝑦𝑖
MV field: 𝐯 = 𝐯1, … , 𝐯𝑁
𝐋 = 𝐃 𝐀−
xLxT
xLxT
ji,
2
ji,
T wxLx ji xx
8
Temporal Depth Video Denoising (cont’d)Objective: Find the optimal Motion Vector (MV) field and denoise blocks in frame t given:
i) denoised frame t-1 and ii) noisy frame t.
Graph Construction:
Draw an edge between two spatial neighbouring blocks of same frame (intra-frame edge).
Draw an edge between target and predictor blocks of different frames (inter-frame edge).
Assign edge weights based on:
Intra-frame edge: MV differences, spatial distance.
Inter-frame edge: MV difference.
Frame t - 1 Frame t
Target blocksPredictor blocks
MV field: 𝐮 MV field: 𝐯
9
Temporal Depth Video Denoising (cont’d)
Two nodes representing… Edge weight 𝑤 between two nodes
𝑏𝐩𝑖𝑡 and its neighboring block 𝑏𝐩𝑗
𝑡 . exp −𝐯𝑖 − 𝐯𝑗 2
2
𝜎𝑣2
𝑏𝐩𝑖𝑡 and corresponding predictor block
𝑏𝐩𝑖+𝐯𝑖𝑡 − 1 .
exp −𝐯𝑖 − 𝐮𝑖 2
2
𝜎𝑣2
two predictor blocks at locations 𝐩 and 𝐪in Frame t-1, if 𝐩 − 𝐪 2
2 ≤ 𝛿. exp −𝐮𝐩 − 𝐮𝐪 2
2
𝜎𝑣2 × exp −
𝐩 − 𝐪 22
𝜎𝑔2
Frame t - 1 Frame t
Target blocksPredictor blocks
MV field: 𝐮 MV field: 𝐯
10
𝐌𝐕𝐒
Temporal Depth Video Denoising (cont’d)
Frame t - 1 Frame t
Target blocksPredictor blocks
min𝐯,𝑏(𝑡)
𝑖
𝑏𝐩𝑖+𝐯𝑖𝑡 − 1 − 𝑏𝐩𝑖
𝑡2
2+ 𝜆 𝜻𝑇𝐋𝜻 2
2 + 𝜇
𝑖
𝑏𝐩𝑖𝑡 − 𝑏𝐩𝑖
𝑜 𝑡2
2
Combined MV field 𝜻: 𝜻 =𝐮𝐯
1. Initialize MV field 𝐯 fix 𝐯 find optimal 𝑏𝐩𝑖𝑡 .
min𝑏(𝑡)
𝐌𝐄𝐄 + 𝐒𝐅
2. Fix 𝑏𝐩𝑖𝑡 find optimal 𝐯. min
𝐯𝐌𝐄𝐄 + 𝐌𝐕𝐒
Optimal 𝑏𝐩𝑖𝑡 = 𝜖𝑏𝐩𝑖+𝐯𝑖
𝑡 − 1 + 1 − 𝜖 𝑏𝐩𝑖𝑜 𝑡 .
2. Fix 𝑏𝐩𝑖𝑡 find optimal 𝐯∗ for 𝐌𝐕𝐒 find optimal 𝐯.
𝐌𝐄𝐄 𝐒𝐅
𝜖∗ =1
1 + 𝜇.
MV field: 𝐮 MV field: 𝐯
Motion estimation error (target blocks and
predictor blocks) term (𝐌𝐄𝐄).Motion vector smoothness term (𝐌𝐕𝐒)
expressed in graph-signal domain
Signal fidelity term (𝐒𝐅) (difference between
observed blocks and reconstructed blocks).
graph Laplacian:𝐋 = D - A
11
In each depth image, find two best-fitting ellipses that model cross-section of patient’s chest / abdomen given depth measurements.
Tracking ellipse major / minor axes across time reveals breathing cycles.
Ellipse Model of Chest / Abdomen
pillow
patient
Kinect depth video
Side view of a sleeping patient
abdomen
Dual-ellipse
Ellipse model
chest
C. Yang, G. Cheung, K. Chan, V. Stankovic, “Sleep Monitoring via Depth Video Recording & Analysis,” 5th IEEE
International Workshop on Hot Topics in 3D (Hot3D), Chengdu, China, July, 2014.12
Sleep Event Detection – Graph Smoothness (GS)
Problem:
Identify if a new data sample 𝑧 is normal or abnormal breathing.
Solution:
1. Split the sequence into 10-second windows.
2. Compute the variances of ellipse parameters (features):
𝑎1, 𝑏1 for chest ellipse;
𝑎2, 𝑏2 for abdomen ellipse.
3. Training data: 𝑦𝑖𝑜 = 𝑎1
𝑖 , 𝑏1𝑖 , 𝑎2
𝑖 , 𝑏2𝑖 , 𝐶 𝑦𝑖
𝑜 ∈ 1,−1 .
4. Concatenated sample vector 𝜉 = 𝐲𝑜 𝑇 𝑧 𝑇.
5. Graph construction: draw an edge between any two samples (nodes) with edge weight
6. Find optimal 𝑧∗: min𝑧
𝜉𝑇𝐋𝜉
hypopnoea / apnoea
Example graph construction.
A linear SVM classifier is also shown.
Graph-signal smoothness term
𝑧∗ = 𝐋𝑧𝑧# − 𝐲𝑜 𝑇𝐋𝐲𝑜𝑧
𝑇. 𝜉𝑇𝐋𝜉
exp −
𝑘=1
4 𝑑𝑘 𝑥𝑘𝑖 − 𝑥𝑘
𝑗
2
2
𝜎𝑐2
𝑥𝑘𝑖 : 4 ellipse parameters.
𝑑𝑘: Parameter weight.
Graph-signal
13
-1
-1 -1
-11
1
1
1
Sleep Event Detection – Robust GS (RGS)
Problem:
Find the optimal concatenated sample vector 𝜉 = 𝐲𝑜 𝑇 𝑧 𝑇
when training data 𝐲𝑜 is noisy; randomly mis-label subset of training data 𝐲𝑜 using a uniform distribution.
Solution:
1. Construct same neighbourhood graph, with edge weight in feature space Euclidean distance.
2. Find optimal 𝜉: min𝜉
𝐲 − 𝐲𝑜22 + 𝛾𝜉𝑇𝐋𝜉.
Graph variation termFidelity term
14
Example graph construction.
A linear SVM classifier is also shown.
-1
-1 -1
-11
1
1
1
Results – Temporal Depth Video Denoising (1)Performance on flickering reduction:
Energy of the difference between two consecutive frames.
Schemes: bilateral filter; temporal median filter (TMF); proposed.
Proposed effectively reduce frame-difference energy – flickering effects.
15
Results – Temporal Depth Video Denoising (2)
Reducing the flickering effect without over-smoothing and preserving sharp edges well.
Original Denoised
A sample depth frame on flickering reduction using proposed.
16
Results – Sleep Event Detection (Accuracy)Schemes: Graph smoothness (GS), Robust graph smoothness (RGS);
linear SVM (SVM-l), SVM with radial basis function kernel (SVM-rbf).
50 quadruples were used as test dataset.
All schemes achieved perfect classification for 50 training quadruples.
SVM-l had a 4% error rate for 30 training quadruples.
50 training quadruples 30 training quadruples
17
Results – Sleep Event Detection (Robustness)Randomly mis-classify a subset of training data using a uniform distribution.
Repeat the mis-classification test procedure 2500 times and then compute the average.
GS and RGS are more robust under noise in the training data than SVM-l and SVM-rbf.
RGS is more noise-robust than GS.
50 training quadruples 30 training quadruples
18
Conclusion
Non-intrusive apnoea / hypopnoea detection system
1. 11-bit to 8-bit depth video coding using H.264 at encoder, recovery of 3 MSBs at decoder.
2. Graph-based temporal denoising: Reduce temporal flickering via graph-smoothness prior.
3. Graph-based classification: Robustly detect apnoea/hypopnoea via graph-signal interpolation.
Outperform known techniques in denoising and classification literature respectively.
On-going work:
Test more patients in different poses.
Add audio features.
19
Thank you!Cheng Yang: [email protected]
Yu Mao: [email protected]
Gene Cheung: [email protected]
Vladimir Stankovic: [email protected]
Kevin Chan: [email protected]
20