Newborn Monitoring based on Multiple Vision Sensors Prof. Pierre Vandergheynst – EPFL Dr Jean-Marc Vesin – EPFL Prof. Martin Wolf – USZ Prof. MD Fauchère – USZ Dr Mathieu Lemay – CSEM Dr Amina Chebira – CSEM Mr Damien Ferrario – CSEM
NewbornCare Project
Motivations Premature births – 9% of the infants in Switzerland are born prematurely – Crucial to continuously monitor vital signs: heart and respiratory rates and arterial oxygen saturation (SpO2) – Brain lesions lead to long-term disabilities in ~25% of infants Limitations of current monitoring systems: – Prone to frequent body motion artifacts – Unacceptable rate of false alarms (87.5%) leading to: – Discomfort, stress and cardiorespiratory instability of the neonates – Desensitized and stressed caregivers – Dangerously long response times – Slow application of sensors, while speed is important during resuscitation – Most sensitive organ, the brain is not monitored yet – Lack of accurate contactless technology
NewbornCare Project
Objectives • Improvement of the quality of the vital sign monitoring • Drastic reduction of the false alarm rate
– By avoiding the physical presence of sensors on the chest and on the limbs of newborns
Multi-optical sensor components (SpO2 and brain tissue oxygen saturation)
NewbornCare Project
Central component (including computer vision algorithms for heart and respiratory rates)
Dedicated smartphone application (selection of neonates + video and data streaming)
NIRS and SpO2 sensors Multi-optical sensor component – Goal: monitor arterial & brain tissue oxygen saturation – Modalities: – Near-infrared spectroscopy (NIRS) – SpO2 dedicated system – Implementation: miniature multi-sensor device integrated into a cap or headband
SpO2
NewbornCare Project
NIRS prototype
NIRS sensor: introduction • Optical sensor for vital parameters monitoring: arerial and brain tissue oxygenation transmission measurement
NewbornCare SpO2 technology
reflective measurement
Photo-diode
Multiple optical channels
source: D.Ferrario
NewbornCare Project
NIRS sensor: introduction • NIRS (near-infrared spectroscopy) for monitoring of vital parameters: arterial and brain tissue oxygenation
source: Dr. T. MĂźhlemann, D.Ostojic
NewbornCare Project
NIRS sensor: aim • NIRS works in setting of NICU (neonatal intensive care unit) • It supplements vision sensor data (reliability by „four-eyes principle“)
vision
senso
r
source: S. Kleiser
NewbornCare Project
NIRS sensor: aspects of WIP Novel sensor shape • power PCB (BT, Battery) & cable • 20cm flexible connection for optics • optical part Expected benefits: • easier handling • optical part fits skull‘s curvature
NewbornCare Project
NIRS sensor: aspects of WIP sensor spot Headband LoFi Prototype • holds sensor in place • provides opening for vision sensor Properties • biocompatible and disinfectable • seamless to avoid pressure-sore • easy-to-fixate • friendly and warm colors
source: Manuel Bühler
NewbornCare Project
camera spot
Accomodating all the sensors • Camera spot needs certain minimal size.
source: Manuel Bühler
• Average minimal size diameter = 2.8 cm NewbornCare Project
Heart and respiratory rates estimation Goal: monitor of cardiac & respiratory activities, wireless central node Modalities: ● Video-based estimation of blood perfusion ● Video-based tracking of thoracic motion
●
Implementation: ● visible light + NIR camera embedded system for algorithms and communication
Heart rate
NewbornCare Project
Respiratory rate
Heart and respiratory rates estimation: camera set-up Assessment of illumination conditions in Neonatal Intensive Care Unit:
[h]
• Illumination range : from 0.5 to 18’000 lux
NewbornCare Project
Heart and respiratory rates estimation: camera set-up 2 cameras were selected: uEye Camera, Color Sensor, 1280x1024 Pixel • For day monitoring uEye Camera, Monochrome Sensor, 1280x1024 Pixel, improved NIR sensitivity • With NIR illumination • For night monitoring
NewbornCare Project
Heart and respiratory rates estimation: validation on adults subjects Database acquisition on adults Color and NIR video acquisition is synchronized with BIOPAC for groundtruth values for ECG, resperation, SpO2 and accelerometer 40 people monitored: ● With natural night ● In the dark ● Using a hand grip ● Doing small movements ● Breathing irregularly
NewbornCare Project
Heart and respiratory rates estimation: tracking and segmentation After getting input video stream from cameras: • Tracking algorithm based on state of the art tracking-by-detection • Improved to be able to track several regions at multiple scales • The tracked region(s): input to the skin segmentation algorithm and then to the heart and respiratory rate estimation algorithms
Advantages of the tracker: • Can be implemented in real-time • Can be initialized with a single frame • Accuracy and robustness can be improved when using custom tracking features (learned from a large set of sample videos) • Very robust to drift (useful for accurate tracking over long periods of time) NewbornCare Project
Heart and respiratory rates estimation: data processing After the tracking of the region of interest: • • • •
Separation of the 3 RGB channels Averaging of the pixels in the ROI for each frame and each color Processing of the time series for heart rate extraction Proposed approach: adaptive frequency tracking
Advantages of adaptive frequency tracking: • Can be implemented in real-time • Accuracy can be improved when using multivariate data (for example: waveforms obtained from the video processing of different skin regions) • Can be used for heart and respiratory rates extraction NewbornCare Project
Thank you !
Any questions ?
NewbornCare Project