Upload File

representation and reinforcement learning for personalized...

  • Home
  • Documents
  • Representation and Reinforcement Learning for Personalized ...ckbjimmy.github.io/docs/weng2017representation_p.pdf · • Current clinical practice follows the guidelines suggested
1
Representation and Reinforcement Learning for Personalized Glycemic Control in Septic Patients Wei-Hung Weng 1 , Mingwu Gao 2 , Ze He 3 , Susu Yan 4 , Peter Szolovits 1 1 CSAIL, MIT | 2 Philips Connected Sensing Venture | 3 Philips Research North America | 4 Massachusetts General Hospital Background Motivation •Critically ill patients have the issue of poor glucose control, which in- cludes the presence of dysglycemia and high glycemic variability. •Current clinical practice follows the guidelines suggested by the NICE- Sugar trial to control the blood sugar level for cricital care. •However, there are overwhelming variations in clinical conditions and physiological states among patients under critical care. This limits cli - nicians’ ability to perform appropriate glycemic control. In addition, clinicians sometimes may not be able to consider the issue of glycemic control. •To help clinicians better address the challenge of managing patients’ glucose level, we need a personalized glycemic control strategy that can take into account the variations in patients’ physiological and pathological states. Reinforcement Learning (RL) in Clinical Domain •RL is a potential approach for the scenario of sequential decision mak- ing with delayed reward or outcome. •RL also has the ability to generate optimal strategies based on non-op- timized training data. •RL has been used for treatment of schizophrenia [Shortreed 2011]; hep- arin dosing problem [Nemati 2016]; mechanical ventilation administra- tion and weaning [Prasad 2016]; and sepsis treatment [Raghu 2017]. •Related to glycemic control, some studies utilize RL and inverse RL to design clinical trials and adjust clinical treatments [Bothe 2014]. •Fewer studies have utilized the RL approach to learn a better target laboratory values as references for clinical decision making. Proposed Approach and Objectives •Learn optimal policy to simulate personalized optimal glycemic trajec- tories, which are sequences of appropriate glycemic targets. •The simulated trajectories are intended as a reference for clinicians to decide their glycemic control strategy, and to achieve better clinical outcomes. •We hypothesized that the patient states, glycemic values, and patient outcomes can be modeled as a Markov decision process (MDP). •‘Action‘ = the glycemic value that lead to real clinical action. •We explored RL approach to learn the policy for learning personalized optimal glycemic trajectories, and compared the prognosis of the tra- jectories simulated by the optimal policy to the real trajectories. •The learned policy is intended as references for clinicians to adapt and optimize their care strategy, and to achieve better clinical outcomes. Abstract Glycemic control is essential for critical care. However, it is a challenging task since there has been no study on personalized optimal strategy for glycemic control. This work aims to learn personalized optimal glycemic trajectories for severely ill septic patients via learning data-driven poli - cies of deciding optimal targeted blood glucose level as clinicians’ refer- ence. We encoded patient states using a sparse autoencoder and adopted a reinforcement learning paradigm using policy iteration to learn the op- timal policy from data. We also estimated the expected return following the policy learned from the recorded glycemic trajectories, which yielded a function indicating the relationship between real blood glucose values and 90-day mortality rates. This suggests that the learned optimal policy could reduce the patients’ estimated 90-day mortality rate by 6.3%, from 31% to 24.7%. The result demonstrates that the reinforcement learning with appropriate patient state encoding can potentially provide optimal glycemic trajectories and allow clinicians to design a personalized strat- egy for glycemic control in septic patients. Methods Experiment Reference • Bellman. Dynamic Programming. 1957. • Bothe et al. The use of reinforcement learning algorithms to meet the challenges of an artificial pancreas. Expert Re- view of Medical Devices, 2014. • Howard. Dynamic Programming and Markov Processes. 1960. • Nemati et al. Optimal medication dosing from suboptimal clinical examples: A deep reinforcement learning approach. In IEEE Engineering in Medicine and Biology Society, 2016. • Ng. Sparse autoencoder. 2011. https://web.stanford.edu/class/archive/cs/cs294a/cs294a.1104/sparseAutoencoder.pdf. • Prasad et al. A reinforcement learning approach to weaning of mechanical ventilation in intensive care units. arXiv 2017. • Raghu et al. Continuous state-space models for optimal sepsis treatment - a deep reinforcement learning approach. arXiv 2017. • Shortreed et al. Murphy. Informing sequential clinical decision-making through reinforcement learning: an empirical study. Machine Learning 2010. • Silver. Reinforcement Learning Lecture 3: Planning by Dynamic Programming. 2015. Data Source and Study Cohort •5,565 septic patients in MIMIC-III version 1.4. •Sepsis-3 criteria to identify patients with sepsis. •Exclusion: age < 18, SOFA < 2, not first ICU admission. •Diabetes: (1) ICD-9, (2) pre-admission HbA1c > 7.0%, (3) admission medication, and (4) history of diabetes in the free text •Data were collected at one hour interval. •Missing values: linear and piecewise constant interpolation RL Settings •Reward: 90-day mortality (+100 / -100) •Action: discretized glucose levels (11 bins) as the proxy of real actions •State: Total 46 normalized variables (patient level variables, blood glu- cose related variables, periodic vital signs) Patient State Encoding •Raw features vs. sparse autoencoder-encoded features [Ng 2011]. •500 state clusters by k-means clustering. Policy Evaluation / Iteration •Learn optimal policy & evaluate on real trajectories. •90-day mortality rate = f (expected return) •Compute and compare the estimated mortality rate of real and opti - mal glucose trajectories obtained by RL-learned policy. [Figure courtesy: David Silver] Conclusion We utilized the RL algorithm with representation learning to learn the personalized optimal policy for better predicting glycemic targets from retrospective data. The method may reduce the mortality rate of septic patients, and potentially assist clinicians to optimize the real-time treat- ment strategy at dynamic patient state levels with a more accurate treat- ment goal, and lead to optimal clinical decisions. Future works include applying a continuous state approach, different evaluation methods, and applying the method to different clinical decision making problems. Results •The data distribution of learned expected re- turn, which is the rescaled Q-value, is negative- ly correlated with mortality rate with high cor- relation value. •The learned expected return reflects the real patient status well. •The optimal policy learned by the policy itera- tion algorithm can potentially reduce around 6.3% of estimated mortality rate if we chose the appropriate patient state representations. Raw feature representation 32-dimension sparse autoencoder representation

Upload: others

Post on 28-Sep-2020

8 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Representation and Reinforcement Learning for Personalized ...ckbjimmy.github.io/docs/weng2017representation_p.pdf · • Current clinical practice follows the guidelines suggested

Representation and Reinforcement Learning for Personalized Glycemic Control in Septic PatientsWei-Hung Weng1, Mingwu Gao2, Ze He3, Susu Yan4, Peter Szolovits1

1 CSAIL, MIT | 2 Philips Connected Sensing Venture | 3 Philips Research North America | 4 Massachusetts General Hospital

BackgroundMotivation•Criticallyillpatientshavetheissueofpoorglucosecontrol,whichin-cludesthepresenceofdysglycemiaandhighglycemicvariability.•CurrentclinicalpracticefollowstheguidelinessuggestedbytheNICE-Sugartrialtocontrolthebloodsugarlevelforcricitalcare.•However,thereareoverwhelmingvariationsinclinicalconditionsandphysiologicalstatesamongpatientsundercriticalcare.Thislimitscli-nicians’abilitytoperformappropriateglycemiccontrol.Inaddition,clinicianssometimesmaynotbeabletoconsidertheissueofglycemiccontrol.•Tohelpcliniciansbetteraddressthechallengeofmanagingpatients’glucoselevel,weneedapersonalizedglycemiccontrolstrategythatcantakeintoaccountthevariationsinpatients’physiologicalandpathologicalstates.

Reinforcement Learning (RL) in Clinical Domain•RLisapotentialapproachforthescenarioofsequentialdecisionmak-ingwithdelayedrewardoroutcome.•RLalsohastheabilitytogenerateoptimalstrategiesbasedonnon-op-timizedtrainingdata.•RLhasbeenusedfortreatmentofschizophrenia[Shortreed2011];hep-arindosingproblem[Nemati2016];mechanicalventilationadministra-tionandweaning[Prasad2016];andsepsistreatment[Raghu2017].•Relatedtoglycemiccontrol,somestudiesutilizeRLandinverseRLtodesignclinicaltrialsandadjustclinicaltreatments[Bothe2014].•FewerstudieshaveutilizedtheRLapproachtolearnabettertargetlaboratoryvaluesasreferencesforclinicaldecisionmaking.

Proposed Approach and Objectives•Learnoptimalpolicytosimulatepersonalizedoptimalglycemictrajec-tories,whicharesequencesofappropriateglycemictargets.•Thesimulatedtrajectoriesareintendedasareferenceforclinicianstodecidetheirglycemiccontrolstrategy,andtoachievebetterclinicaloutcomes.•Wehypothesizedthatthepatientstates,glycemicvalues,andpatientoutcomescanbemodeledasaMarkovdecisionprocess(MDP).•‘Action‘=theglycemicvaluethatleadtorealclinicalaction.•WeexploredRLapproachtolearnthepolicyforlearningpersonalizedoptimalglycemictrajectories,andcomparedtheprognosisofthetra-jectoriessimulatedbytheoptimalpolicytotherealtrajectories.•Thelearnedpolicyisintendedasreferencesforclinicianstoadaptandoptimizetheircarestrategy,andtoachievebetterclinicaloutcomes.

Abstract

Glycemiccontrolisessentialforcriticalcare.However,itisachallengingtasksincetherehasbeennostudyonpersonalizedoptimalstrategyforglycemiccontrol.Thisworkaimstolearnpersonalizedoptimalglycemictrajectoriesforseverelyillsepticpatientsvialearningdata-drivenpoli-ciesofdecidingoptimaltargetedbloodglucoselevelasclinicians’refer-ence.Weencodedpatientstatesusingasparseautoencoderandadoptedareinforcementlearningparadigmusingpolicyiterationtolearntheop-timalpolicyfromdata.Wealsoestimatedtheexpectedreturnfollowingthepolicylearnedfromtherecordedglycemictrajectories,whichyieldedafunctionindicatingtherelationshipbetweenrealbloodglucosevaluesand90-daymortalityrates.Thissuggeststhatthelearnedoptimalpolicycouldreducethepatients’estimated90-daymortalityrateby6.3%,from31%to24.7%.Theresultdemonstratesthatthereinforcementlearningwithappropriatepatientstateencodingcanpotentiallyprovideoptimalglycemictrajectoriesandallowclinicianstodesignapersonalizedstrat-egyforglycemiccontrolinsepticpatients.

Methods

Experiment

Reference•Bellman.DynamicProgramming.1957.•Botheetal.Theuseofreinforcementlearningalgorithmstomeetthechallengesofanartificialpancreas.ExpertRe-viewofMedicalDevices,2014.

•Howard.DynamicProgrammingandMarkovProcesses.1960.•Nematietal.Optimalmedicationdosingfromsuboptimalclinicalexamples:Adeepreinforcementlearningapproach.InIEEEEngineeringinMedicineandBiologySociety,2016.

•Ng.Sparseautoencoder.2011.https://web.stanford.edu/class/archive/cs/cs294a/cs294a.1104/sparseAutoencoder.pdf.•Prasadetal.Areinforcementlearningapproachtoweaningofmechanicalventilationinintensivecareunits.arXiv2017.

•Raghuetal.Continuousstate-spacemodelsforoptimalsepsistreatment-adeepreinforcementlearningapproach.arXiv2017.

•Shortreedetal.Murphy.Informingsequentialclinicaldecision-makingthroughreinforcementlearning:anempiricalstudy.MachineLearning2010.

•Silver.ReinforcementLearningLecture3:PlanningbyDynamicProgramming.2015.

Data Source and Study Cohort•5,565septicpatientsinMIMIC-IIIversion1.4.•Sepsis-3criteriatoidentifypatientswithsepsis.•Exclusion:age<18,SOFA<2,notfirstICUadmission.•Diabetes:(1)ICD-9,(2)pre-admissionHbA1c>7.0%,(3)admissionmedication,and(4)historyofdiabetesinthefreetext•Datawerecollectedatonehourinterval.•Missingvalues:linearandpiecewiseconstantinterpolation

RL Settings•Reward:90-daymortality(+100/-100)•Action:discretizedglucoselevels(11bins)astheproxyofrealactions•State:Total46normalizedvariables(patientlevelvariables,bloodglu-coserelatedvariables,periodicvitalsigns)

Patient State Encoding•Rawfeaturesvs.sparseautoencoder-encodedfeatures[Ng2011].•500stateclustersbyk-meansclustering.

Policy Evaluation / Iteration•Learnoptimalpolicy&evaluateonrealtrajectories.•90-daymortalityrate=f (expectedreturn)•Computeandcomparetheestimatedmortalityrateofrealandopti-malglucosetrajectoriesobtainedbyRL-learnedpolicy.[Figurecourtesy:DavidSilver]

Conclusion

WeutilizedtheRLalgorithmwithrepresentationlearningtolearnthepersonalizedoptimalpolicyforbetterpredictingglycemictargetsfromretrospectivedata.Themethodmayreducethemortalityrateofsepticpatients,andpotentiallyassistclinicianstooptimizethereal-timetreat-mentstrategyatdynamicpatientstatelevelswithamoreaccuratetreat-mentgoal,andleadtooptimalclinicaldecisions.Futureworksincludeapplyingacontinuousstateapproach,differentevaluationmethods,andapplyingthemethodtodifferentclinicaldecisionmakingproblems.

Results•Thedatadistributionoflearnedexpectedre-turn,whichistherescaledQ-value,isnegative-lycorrelatedwithmortalityratewithhighcor-relationvalue.•Thelearnedexpectedreturnreflectstherealpatientstatuswell.•Theoptimalpolicylearnedbythepolicyitera-tionalgorithmcanpotentiallyreducearound6.3%ofestimatedmortalityrateifwechosetheappropriatepatientstaterepresentations.

Rawfeaturerepresentation

32-dimensionsparseautoencoderrepresentation

Informational Interviews · Add contact on LinkedIn with personalized message Reply to original email Report back after following up with suggested contacts Send them articles of

TOWARDS THE REINFORCEMENT OF STUDENTS GLOBAL … · national schools that will use Malay as the instructional medium) and the 1951 Fenn-Wu Report (which suggested the preservation

Personalized medicine, personalized intelligence

DOCUMENT RESUME ED 095 016 Capasso, Ronald L.; Lachat ... · Based upon diagnostic testing, skill analysis, personalized practice, and immediate reinforcement, APT seeks to develop

Temperament and exercise dependence Running title ... · 3 1 Introduction 2 ... 3 Prior research suggested a link between specific personality and temperament ... Reinforcement Sensitivity

Reinforcement Learning Lecture Inverse Reinforcement Learningipvs.informatik.uni-stuttgart.de/mlr/wp-content/uploads/2017/07/09... · Reinforcement Learning Inverse Reinforcement

Reinforcement Learning - 4. Model-free reinforcement Learning

Schedules of reinforcement. Schedules of Reinforcement Continuous reinforcement refers to reinforcement being administered to each instance of a response

SUGGESTED TECHNIQUES SUGGESTED ACTIVITIES PRACTICE

Reinforcement Learning and Deep Reinforcement Learningcse.ucdenver.edu/.../Class-22-Reinforcement-learning-DL.pdf · 2018. 11. 28. · Outlines 1 Principles of Reinforcement Learning

Personalized Intelligent Tutoring System using Reinforcement …ravi/papers/Ankit_FLAIRS_11.pdf · Ankit Malpani Microsoft Corporation, India B. Ravindran and Hema Murthy Department

Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

A Romantic Experience - Four Seasons · A Romantic Experience T ... Dedicated waiter for the evening Half bottle of Champagne Personalized theme SUGGESTED ENHANCEMENTS Fully decorated

Causal Inference & Estimands - Universitätsmedizin MainzStatistical Methods for Dynamic Treatment Regimes: Reinforcement Learning, Causal Inference and Personalized Medicine. Springer

Personalized Dynamics Models for Adaptive Assistive Navigation … · 2018. 10. 9. · Keywords: Indoor Navigation, Model-Based Reinforcement Learning, Human-Robot Interaction, Assistive

Eick: Reinforcement Learning. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning

Representation and Reinforcement Learning for Personalized ... · Many critically ill patients have poor glycemic control, such as dysglycemia and high glycemic variability. Several

for high REINFORCEMENT SYSTEMS - nevoga.com€¦ · 2 REINFORCEMENT SYSTEMS REINFORCEMENT SYSTEMS REINFORCEMENT SYSTEM PLEXUS®, PYRAPLEX®, FTW The reinforcement system PLEXUS®,

Suggested Suggested textbook:

Reinforcement Learning - Multi-Agent Reinforcement

DURABILITY OF GLASS FRP COMPOSITE BARS … REINFORCEMENT UNDER TENSILE SUSTAINED LOAD ... suggested by ACI 440.1R-01, ... than the specified design strength as recommended by ACI 440

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Social, Personalized Lifelong Learning Personalized - CiteSeer

Continuous hoops for transverse reinforcement of ... · Continuous hoops for transverse reinforcement of ... transverse reinforcement details for the ... Fig. 1 Transverse reinforcement

Helly Hansen | Skiing, Sailing, & Outdoor Apparel | HH US · 2018. 6. 20. · personalized fit Extra reinforcement in critical areas for durability New Hi-Vis pant in our Alna collection

Meyer from personalized med to personalized research-april 2013

Reinforcement Learning Das Reinforcement Learning-Problem Alexander Schmid

Social-Personalized versus Computer-Personalized Methods ... · Social-Personalized versus Computer-Personalized ... Social-Personalized versus Computer-Personalized Methods to

Helly Hansen | Skiing, Sailing, & Outdoor Apparel | HH US · personalized fit YKK zipper Extra reinforcement in critical areas for durability New Hi-Vis pant in our Alna collection

INSPIRE: An INtelligent System for Personalized ... · where Honey and Mumford, based on Kolb's theory of experiential learning [9], suggested four types of learners: Activists, Pragmatists,

Social-Personalized versus Computer-Personalized Methods

Inverse Reinforcement Learning CS885 Reinforcement

MULTI-ARMED BANDITS WITH COVARIATES: THEORY AND ... · personalized medicine, recommender system, reinforcement learning. 1. Introduction and Background The k-armed bandit problem

Personalized Medicine and Human Genetic Diversityperspectivesinmedicine.cshlp.org/content/4/9/a008581.full.pdf · between “racial” groups (Lewontin1972). Lew-ontin’s data suggested

Interview Fair TOMORROW!!! Folders Must have… –Cover Letter (personalized) –Resume (personalized) –References –Application (personalized)