Welcome to LTS

NewBornCare clinical monitoring based on vision and spectroscopy

Prof. Pierre Vandergheynst EPFLDr Jean-Marc Vesin EPFL Prof. Martin Wolf USZProf. MD Fauchre USZDr Mathieu Lemay CSEMDr Amina Chebira CSEMDamien Ferrario CSEMFabian Braun CSEMSibylle Fallet EPFLLionel Martin EPFLVirginie Moser CSEMDaniel Ostojic USZYann Schoenenberger EPFL1Dear fellow scientists, the project is called NewBornCare and it is about NewbornMonitoring based on Vision and Spectroscopy Sensors.And yet Why should you bother? First things come first, so let's start withthe motivations.Motivations2

False alarmsCurrent monitoring systems

Premature births9% of the infants in Switzerland are born prematurely Brain leasions lead to long-term disabilities in ~25% infantsCrucial to continuously monitor vital signalsHeart and respiratory rates, arterial oxygen saturation (SpO2) Limitations of current monitoring systems: Current monitoring systems prone to frequent motion artifactsInacceptable high rate of false alarms of 87.5% leading toDiscomfort, stress and cardiorespiratory instability of the neonatesDesensitized caregivers, also stressedDangerously long response times Slow application of sensors, while speed is important during resuscitationBrain, most sensitive organ, is NOT yet monitoredLack of accurate contactless technologyNegative impact onAlmost 10 percent of children born in Switzerland are prematurely born.In first hours/days and sometimes even weeks, their vital signs need to be monitored, in order to react in timely matter and prevent disability or even death. Current monitoring devices have their limits though, such as a lot of false alarms (the beeps and the flashes of light), that stresses out the staff and cause the discomfort for the neonates, mainly because of the unnecessary intervention by caregivers. The caregivers get desentisized and provide a less timely response, because they start to believe that "it is yet another false alarm", which is obviously detreamental for the neonate's health status. Yet additional limits are sensors that are not adapted for the purposes of neonatology in size and flexibility, that do not monitor brain and finally most of the time are attached to the extremeties, which is problematic because of the movement artefacts created by the leg or arm movements.2ObjectivesHigher quality in vital sign monitoringSubstantial reduction of false alarms Absence of sensors on the chest and on the limbs

3

NewbornCare concept

Multisensor componentCentral componentCommunication componentArterial and cerebralHeart and respiration rateO2 saturation

It's all about better quality of the vital sign monitoring, meaning improved specificity of the alarms, while at the same time maintaining the system's sensitivity. Substantial reduction of false alarms is to be achieved by omission of sensors from the chest and the limbs regions of the newborn, because these are the regions mostly affected by the motion artefacts. The central component of our NBC, That is NewBornCare system is incorporating a vision sensor that with the help of image processing algorithms extracts the heart and the respiratory rate, the two vital parameters in neonatal NICU monitoring. Thus the vision sensor works fully contactless with respect to the body of the neonate. An additional sensor mounted on the head of the neonate monitors the arterial oxygenation, which is yet another important vital parameter, while at the same time acquiring information on tissue oxygenation, which is a paramater, that is a very promissing candidate to become the next vital sign, since it holds the key information, which when combined with the already mentioned HR/RR and SaO2 in the sense of agreement or non-agreement can help in getting more robust decision to raise or not to raise the alarm.3MethodologyNewbornCare multi-optical sensor component

Aim: Monitor arterial & brain tissue oxygen saturation bySpO2 dedicated to neonatology sensorStO2 near-infrared spectroscopy (NIRS)Miniature multi-sensor devices held in place by headbandUSZ: NIRS design, validation (task leader)CSEM: SpO2 design, system integration4

NIRS prototype

SpO2

Methodology, Let's discuss the components of the system one after the other. A crucial component of every vital signal monitoring is measurement of the arterial oxygenation. The commercial sensors may well be developed for the purposes of pediatry, still lack the suitability for much smaller category of preterm infants, sometimes weighing not more than 0.6 kg. One of main disadvantages of commercial sensors is, that they are being attached to the arms or the legs, which make them prone to movement artefacts. This is why we chose to develop a sensor of our own for measurements at the head of the neonate, which moves far less than the rest of the body. In addition to arterial oxygenation, we are creating a device to measure the second important parameter, the tissue oxygenation.4MethodologyNewbornCare central component

Aim: Monitor cardiac & respiratory activities byVideo-based estimation of blood diffusionVideo-based tracking of thoracic motionWith visible & NIR camera using algorithms & real-time communicationEPFL: data processing (task leader) CSEM: system integration5

Respiratory rate

Heart rate

In order to capture the heart and respiratory rate, we are going to be using a camera based system, that extracts changes of texture and coloring in the image and uses them to extract the data on heart and respiratory rate.5Current project statusT1:Camera set-up & system specifications

T2:Skin area tracking & segmentation

T3:Heart rate estimation

T4:Respiratory rate estimation

T5:Arterial & brain tissue oxygen saturation

T6:Communication module (skipped due to budget decrease)

T7:Clinical investigation6In order to organise work and distribute the workpackages, we have defined the 7 tasks, the stages of completion will now be discussed one after the other.6T1: Camera set-up & system specifications7Selected camsNIR night vision (UI-3240CP-NIR-GL)RGB day vision (UI-3240CP-C-HQ)

High sensitivity cams requiredMinimum 10 bits of pixel dynamics

Frame rateMinimum 20 frames/sec for 10 bits dynamics

No auto-exposure timeHeart rate extraction is based on light changes

Selected illumination for night visionabus TV6700 (12 LEDs 850nm) with eye safety certificationT1: Completed

the round white piece of the equipment above the two cameras is the IR-illumination The black and white image is the image taken in the dark with the NIR illumination with the NIR camera, the color image is the one taken with artificial light with the RBG camera.We need both cameras because we have a better signal in the visible (3 channels), but we need a solution for the night, when there is only a scarce illumination. On the other hand, in the dark, we will have controlled illumination (NIR) which will improve the signal.

Auto-exposure means that the camera itself controls the time of the "image integration". The longer the time, the more light you have. If there is too much light the image looks over illuminated. So if the camera itself controls the exposure time, we will have different exposure times for each frame, which will make impossible to estimate the heart rate. So if we need to change the exposure time (lighting changes), we need to do it in accordance with the algorithms. Most of sensors have 8 bits of dynamics, so each pixel can have a value between 0 and 255. 10 bits of dynamics means that each pixel can have a value between 0 and 1024 which means a better sensitivity of the signal. This value has been determined by a pre-study at CSEM with different sensors.7T2: Skin area segmentation & tracking 8AchievementsAlgorithms segments and tracks ROIIntegration to heart/respiration rate pipelineSupport for multi-ROI trackingFeature highlights:Very simple (i.e. fast) algorithm relying on histogram backprojectionSkin color histogram can be built offline given sample dataNext steps:Automatic initialization (segmentation)Use spatial information to refine segmentation maskCompare result with histograms in different color spaces

What is that the sketches of a baby with a cap and two openings have in common with the image of the adult person? Well they both carry a cap and indeed it is about the opening in the cap being often the only place where the skin shines through and can be looked at by the camera in neonatology set-up, which is why we have to automatically find the opening and once found, track it, for it is the only patch of skin on our disposal to extract the heart- and respiratory rate from. Here are some of the features of the algorithm either already implemented or yet to be implemented:

-working hand in hand with HR/RR algos-supporting for multi-ROI

Smartness in terms of performance, such smart drops of the frames to reduce the search space. Next step: this is about automatic init, so the algo can find the ROI, without the input from human intelligence.8T3: Heart rate estimation9Question: Which region of face and wavelengths to use? Methods: Partitioning of face into 260 small regions of interestPower spectral density analysis on sliding-windowPercentage of the power at the local true heart rate (from ECG)Results:Forehead & cheekbones are best regionsThe green channel is the best channelNext steps:Use this information to improve alreadrunning tracking

Achievement: Optimal detection of heart rate in face determined

Power around the true heart rate9T3: Heart rate estimation10

Achievement: Real-time estimation of heart rate from video in adult subjects in different experimental conditions

VisibleMean abs error [CI 95%] [bpm]Pearson correlation3.14 [-7.80 9.65]0.93DarkAverage absolute error [bpm]Pearson correlation4.21 [-10.66 13.77]0.86Method:Signals are derived from a region of interest on the subjects forehead. Particularities of the database:Video-sequences recorded in visible light (RGB camera) or darkness ( NIR camera)Important heart rate changes induced by handgrip exercise or by respiration modulationAdaptive frequency tracking for heart rate estimationPerformance assessment: comparison with ground-truth derived from ECG Results:Average absolute error of 3.14 / 4.21 bpm for visible light / darkness Heart rate fluctuations were correctly trackedNext steps:Increase robustness against motion artifactsValidation of this processing scheme with neonates 10T4: Respiratory rate estimation11Achievement: Real-time estimation of respiratory signal from video

Block diagram of the respiration detection algorithmMethod:Approximation of vertical motion for regions (blocks) of imageClassification of each block by its likelihood to contain a true respiratory signalResults:Close resemblance between ground-truth (respiratory belt) and estimated respiratory signalsEstimation Error: 0.0 2.4 bpmFor 18 recordings of 4 minutes (9 subjects both in darkness and artificial illumination)Next steps:Performance assessment on larger databaseImprove robustness to non-respiratory movements

11T4: respiratory rate estimation12

Achievement: Real-time estimation of respiratory signal from video

Example of estimated respiratory signal (red) and ground-truth (blue)Method:Approximation of vertical motion for regions (blocks) of imageClassification of each block by its likelihood to contain a true respiratory signalResults:Close resemblance between ground-truth (respiratory belt) and estimated respiratory signalsEstimation Error: 0.0 2.4 bpmFor 18 recordings of 4 minutes (9 subjects both in darkness and artificial illumination)Next steps:Performance assessment on larger databaseImprove robustness to non-respiratory movements

12T3 & T4: Heart & respiration rate estimation (live)13

This video demonstrates the ability of the system and algorithms to estimate the heart- as well as the respiratory rate ...The heart-rate vision estimation complies well to the values simultaneously obtained by the ECG ... as for the respiratory rate, we have not been so far able to compare them with the gold reference, but the values obtained look realistic and we hope to get that confirmed soon.13T5: Arterial & brain tissue oxygen saturation14

Datalogger

Pulse oximetry sensorUSB electrically isolated (IEC 60601 compliant)

Casing protected against water dropLED current limitation (fuses)Flexible PCBSensor and cables biocompatibleHere is the system overview for the tissue oxygenation sensor, that will be placed on the forehead of the neonate and will staying inside of the isolette the place, where the infants feel safe and warm and where they get enough oxygen. The data is being constantly sent to the host PC, where it is being processed and visualized in real-time.14T5: Arterial & brain tissue oxygen saturation15Near-infrared spectroscopy cerebral tissue O2 saturation (StO2)Re-useable sensorPower supply box

Real-time graphical user interface

StO2, HHb, O2HbAcceleration data

Event marksSignal strength

Here we see the graphical user interface, that provides means of communication with the sensor e.g. setting the intensities of the light sources, while at the same time visualizing all of the important measured parameters coming from the sensor, such as StO2, HHb, O2Hb, acceleration data & signal quality in addition to providing the user with the capability to setting the markers for events.15StO2

T5: Arterial & brain tissue oxygen saturation16Timely coincidence of StO2 & SpO2On the second sight however, we realize that the heavy drops in StO2 levels at certain time points timely coincide with the measurements obtained by the commercial grade SpO2 meters, which is a clear indicator of the physiological origin of the spikes. The delay between the two signals, because there should be some even for the neonates (40mL of blood, shorter distances), is yet to be determined. The time-points are 4 seconds each, overall time is 800 sec = 12 min.16

T5: Arterial & brain tissue oxygen saturation17Validation in liquid phantom

We have performed a considerable amount of measurements in liquid phantoms that mimic the behaviour of the living tissue having similar absorption and scattering coefficients. For different concentrations of human blood we obtain similar results as are measured with commercially available devices, which asserts us in the belief, that the StO2 sensor made by us is a good one.17Test of new technology in vital- & could-become-vital parameters monitoring under realistic conditions

T7: Clinical investigation18State: Clinical study protocol in preparation for SwissEthics and Swissmedic

LAB CONDITIONSREALITYIn the centre of our activities now is the definition of the documents to be compiled for Swissethics, one of them being the clinical study protocol.Given the fact, that our multiple-sensor system needs a realistic environment first to be tested for usability, we are going to have an Exploratory Study on Ability of Dedicated Novel Sensor Technologies to Reduce False Alarms in Vital Sign Monitoring of the Neonates18

T7: Clinical investigation19

StO2

Primary objective:

Quality of HR/RR videobased Secondary objective:

Correlation SpO2 / StO2 Within the clinical study we will aim to estimate the quality of HR/RR videobased contactless estimation by comparisson to the gold standard, which is the ECG. As secondary objective we aim at finding more about the coupling of events of interest between the arterial and tissue oxygenation.19Conclusions20Technical advances as plannedAims achieved as plannedTiming as planned20