presentation cbms 2014, mount sinai, ny

The Medical Cyber-Physical Systems Activity at EIT: A Look under the Hood

CBMS 2014 NEW YORK

29/5/2014Daniel Sonntag(DFKI), Sonja Zillner (Siemens), Samarjit Chakraborty (TUM),

Andras Lorincz (ELTE), Esko Strommer (VTT), and Luciano Serafini (FBK)

Wednesday, May 28, 14

Vision• In the future, clinical environments will develop into

medical cyber-physical systems of their own. Patients will get direct treatment according to a direct data acquisition and interpretation workflow. In the end, the doctor’s decision support will be provided according to the data the cyber- physical system (CPS) collects from the individual patients. This approach will be scalable, extend patient monitoring to data collections at home by using portable sensors providing information about a patient’s recovery status, and influence healthcare of the future.

2


UPEN’S DEFINITION

Medical Cyber-Physical Systems (MCPS) are complex, safety-critical, intelligent systems of interconnected medical devices. An MCPS brings together monitoring devices, such as heart-rate monitors, and delivery devices, such as medication infusion pumps.

3


4

Networked Closed-loop Medical Cyber-Physical SystemsClosed-Loop Decision Support?


MEDICAL CPS ENVIRONMENTS

Objective, Workscope, Outcomes


• EIT ICT Labs is the Knowledge and Innovation Com- munity of EIT with a focus on future Information and Communication Society. In this framework, we have a specific innovation objective within cyber-physical systems, namely to combine active and passive user input modes in clinical environments for knowledge acquisition and decision support.

6


The EIT ICT Labs' investment model, the so-called "catalyst-carrier model"

7

http://www.dfki.de/MedicalCPS/

New co-operation of European institutions in a future European healthcare sector. Rich capture of clinical data, including symptoms and signs rather than just a single diagnosis, are among the objectives of parallel European efforts (e.g., http://www.transformproject.eu/).






EIT Activity: Specific Objectives

• We will combine active and passive user input modes in clinical environments for knowledge acquisition and decision support.

• Active input modes include digital pens, smartphones, and automatic handwriting recognition for a direct digitalisation of patient data.

• Passive input modes include sensors of the clinical environment and/or mobile smartphones.

• This combination for knowledge acquisition and decision support (while using machine learning techniques) has not yet been explored in clinical environments and is of specific interest because it combines previously unconnected information sources.

• The innovative aspect is a holistic view on individual patients whereby individual active and passive patient data can be taken into account for clinical decision support.


9

In our first running testbed system, sensor data aggregate clinical databases automatically whereupon scalable patient data can be recorded on a daily basis in a clinical database.

Difference to IBM WATSON: Realtime aspect, direct digitisation, holistic view


10

• We hope that the reference architecture will provide new insights into the interplay of active and passive sensor modes in the medical domain which we will additionally exploit in the technology transfer into the European healthcare sector.

• three main steps: (1) digitisation of patient data, (2) usage of environmental (medical) sensor technology, (3) distributed data access and real-time clinical decision support.


11

Milestones• knowledge acquisition by intelligent user interfaces (active modes); • networked embedded systems development and sensors development

in presence, posture, and activity monitoring of humans by non-intrusive sensors, thereby including a functional sensor architecture (passive modes);

• data mining: segmentation of behavioural patterns, finding analogies between behaviours, and predicting using these analogies;

• the knowledge integration (including a semantic model for clinical information) according to the data intelligence models in combination with clinical patient data towards clinical decision support.


Overall rationale• The overall rationale is to address the knowledge acquisition

bottleneck: future clinical care relies on digitised patient information; this information can be collected manually (active mode) by a doctor or a patient, or automatically (passive mode) by using suitable environmental sensors, e.g., of a portable device the patient is carrying.

• Suitable data mining models should enable us to combine those independent information sources for a combined decision support model. For example, a patient’s recovery from a back injury can be monitored by the movement data while climbing steps and combined with the digitised patient disease record at the hospital’s side.

12


COMPONENTS


15

Input/Output Real-Time Dialogue Server


Real-time communication• The communication between activity sensor server and system server

is realized using XML-RPC.

16


DFKI Active User Input


18


19Technische Universität München Philipp Kindt <[email protected]> 5/5

Topic Suggestion: ECG + activity monitoring at home

● Heart data is collected and surveilled

continuously

● Compressed form of data is sent to

hospital through a wireless link to the

smartphone or a base station

● Seamless integration into hospital's

information processing system

● Long-time data will become available to

doctor

● Motion artifacts should be filtered by

accellerometer data


Mobile Sensor Architecture / Data Flow

20

1) The activity sensor measures acceleration data in three dimensions and transmits them via a Bluetooth Low Energy (BLE)-link to the activity sensor server.

2) The activity sensor server receives the streaming data. It converts the raw acceleration signals from the sensors to physical units. The signal is then processed, transformed to the frequency domain and activity detection performed.


21


Eö

Fa

cu

lty

of

Info

rmati

cs

Neu

ral I

nf P

roce

ssin

g G

roup

?

?Hint…

Research:

Pattern completion spatio-temporalSparse methods dictionary learning structure discovery NLP

Applications:

Face action-unitsEducational games recommender systemsCPS pattern discovery and forecasting

ELTE


02/19/1323

Another pilot use case: a sofa/bed with capacitive sensors

11

9

8

7

51

3

0 4

Applications:• Sleep disorder monitoring• Supporting independent living of elderly


05/28/1424

Pilot sensor installation in a bed bottom plate

Unfolded bed bottom plate Bed bottom plate folded for transportation

Leg end instrumentation of the bed bottom plate: • four sensor elements, • a sensor board,• a communication board• USB connection

Size: • unfolded: 80 cm x 200 cm • folded: 80 cm x 50 cmMaterial: foam PVC


05/28/1425

User interfaceUser is laying twistedUser is laying flatUser is sitting

High activityMedium activityLow activity


05/28/1426

Functionality of the bed installationSelf calibration• Sensor platform powers up and calibrates itself when the USB connection is

established• Persons are not allowed on the bed at this time, but all the bed clothes (mattress, bed-

sheet, etc.) must be on place• After a few seconds the sensor system is calibrated and operational

Measurement• Sensor Board A measures the capacitance of 5 sensor electrodes, and

communicates with the Communication Board by using SPI bus• Sensor Board B operates the same but with 4 sensor electrodes• Communication Board combines the results from both of the Sensor boards and

sends the measurement results to the PC by using USB connection

Data processing• The data is processed on a PC or in a server• The system detects if the bed is empty or if there is a person on the bed

• It also detects the posture of the person: sitting on the side of the bed, laying flat on one’s back, laying twisted in fetal position

• Also the activity of the person is detected: whether the person has low, medium, high or shocking activity


• The motivation for additional environmental sensors lies in tracking the behaviour of patients or care home residents and detecting abnormal living patterns.

• In addition, the medical CPS perspective includes the monitoring of clinical care of bedridden patients.

• We take an approach for such an (eHealth) monitoring detector by implementing an intelligent furniture network (here, a bedplate).

• Human behaviour in the form of postures and activity levels is monitored by using a set of very low cost low-intrusive capacitive proximity sensors.

27


SEMANTIC MODEL FOR CLINCAL INFORATION

28


LOGO 29

Data Integration Steps

Knowledge Lifecyle in Multimodal Medical CPS

“Retrieve patient record”

“Add captured input data”


• We developed a generic clinical data access strategy within this medical CPS activity that allows us to seamlessly and flexibly align patient data from various data sources, including sensor data.

• First, an integrated clinical information model that establishes the foundation to integrate and structure clinical data by providing concepts covering meta-information and interpretations of clinical patient data;

• Second, an natural language processing (NLP) information extraction pipeline that provides the basis algorithms needed to extract the relevant information from unstructured clinical textual documents.

• Goal: include sensory data towards real-time clinical decision support• (Currently, we are in the process of aligning the semantically

described clinical data with the patient sensor data.)• Increased data availability should make individualised treatments

possible; the problem we, however, face is that these data are not semantically integrated.

30


• Standardised representation requires the use of established ontologies, vocabularies or coding systems like, e.g., the ICD10, LOINC1 or SNOMED CT2.

• We propose a semantic Model for Clinical Information (MCI) based on the Ontology of General Medical Science (OGMS).

• The resulting semantic model allows us to monitor for example the sleeping status which can be associated with health information; or the duration a patient is bedridden can be associated with administrative data.

31


Model for Clincial Information

32

Link to terminologies

Storage of applicationdata

SNOMED CT—Systematized Nomenclature of Medicine Clinical Terms; ICD—International Classification of Diseases; LOINC—Logical Observation Identifier Names and Codes: ATC—Anatomical Therapeutic Chemical Classification System; RadLex—Radiological Lexicon; DOID— Disease Ontology; FMA—Foundational Model of Anatomy; OPS— Operationen- und Prozedurenschluessel (German coding system for pro- cedures); SYMP—Symptom Ontology; LinkedCT—Linked Clinical Trials; DrugBank—Open Drug Data; SIDER—Side Effect Resource.

The current version of MCI has links to ICD-10, OPS10, ATC11, LOINC, the FMA, and RadLex. By using federated SPARQL queries, it becomes possible to combine data of MCI with knowledge contained in other ontologies

RDF

RDF

RDFOWL


Integrating processes and ontologies

33

Roles / Organization Documents

Semantic Annotations

Actions


34

In Medical CPS, we modelled the ACR guidelines: http://www.acr.org/Quality-Safety/Standards-Guidelines/ Practice-Guidelines-by-Modality/Breast-Imaging.

(1) support for the construction of integrated domain and process models; (2) easy editing of a wiki page by means of forms accessible to the medical experts; (3) automatic import and export in OWL and BPMN; easy import of lists of elements organised according to predefined semantic structures (taxonomy or partonomy); (4) term extraction functionalities; (5) graphical browsing/editing of the domain and process models; and (6) fully-integrated model evaluation functionalities (model checklist and quality indicators)


http://www.acr.org/Quality-Safety/Standards-Guidelines/

http://www.acr.org/Quality-Safety/Standards-Guidelines/

Forthcoming • http://www.kuenstliche-intelligenz.de/

index.php?id=7801

• EIT action line health and wellbeing• “Clinical Data Intelligence” (BMWi)• “Kognit” (BMBF)• Augmented Reality

35


http://www.kuenstliche-intelligenz.de/index.php?id=7801




CONCLUSIONS• We described how we combined active and passive user input modes in clinical

environments for knowledge discovery and knowledge acquisition towards real-time decision support in clinical environments.

• The main contributions are, first, technical components for aggregating digitised patient information, combining manual data acquisition and sensory data together with a dialogue server, and second, the integration of data acquisition technologies into a clinical testbed environment based on semantic models for clinical information.

• The expected outcomes within EIT include enabling technologies for clinical decision support; and a medical CPS reference architecture that can contribute to the definition of a European Healthcare Infrastructure in Horizon 2020, see http://ec.europa.eu/programmes/horizon2020.

• In the next project phase (2015), we will focus on CPS related programming questions, i.e., physiological close loop control with human in the loop extension (proactively request users for real-time expert input).

36


http://ec.europa.eu/programmes/horizon2020




Thank you!

37


38


NEW DIRECTIONS DFKI / Siemens / ELTE


40

Augmented Reality in Medicine

Video: https://dl.dropboxusercontent.com/u/48051165/ERmed_CHI2013video.mp4


https://dl.dropboxusercontent.com/u/48051165/ERmed_CHI2013video.mp4

https://dl.dropboxusercontent.com/u/48051165/ERmed_CHI2013video.mp4

41


CPS PROGRAMMING OBJECTIVES 2014-2015

DFKI / Siemens / KTH / ELTE


43

Decision support systems: professionals and patients, as well as patient guidance services, which build on multimodal data fusion, data and pattern analysis, and modeling and predictive algorithms of patient health status


CPSAction

CPSAction

CPSAction


45

Server/Client infrastructure security & resilience (DFKI)“the capacity to recover quickly from difficulties”

Quality assurance of Medical CPS programs (ELTE)

Clinical workflow standardisation (SIEMENS)

CPSAction

CPSAction

CPSAction

Task Structure


Valorisation potential and expected revenues to be generated

• Our experts predict a change of the medical healthcare sector in Europe (according to the US developments) with three main steps:

• (1) digitalisation of patient data, • (2) usage of environmental (medical) sensor technology, • (3) distributed data access and real-time clinical decision support.

This activity addresses all three steps in very specific but related tasks.

• The European healthcare market profits directly by the provided reference architecture, and indirectly by a new co-operation of European institutions in a future European healthcare sector.


presentation cbms 2014, mount sinai, ny

Technology