Extractive Summarization of EHR Discharge Notes

Emily Alsentzer emilya@mit.eduAnne Kim anneykim@mit.edu

Abstract

Patient summarization is essential for cliniciansto provide coordinated care and practice effec-tive communication. Automated summariza-tion has the potential to save time, standard-ize notes, aid clinical decision making, and re-duce medical errors. Here we provide an up-per bound on extractive summarization of dis-charge notes and develop an LSTM model tosequentially label topics of history of presentillness notes. We achieve an F1 score of 0.876,which indicates that this model can be em-ployed to create a dataset for evaluation of ex-tractive summarization methods.

1. Introduction

Summarization of patient information is essential to thepractice of medicine. Clinicians must synthesize infor-mation from diverse data sources to communicate withcolleagues and provide coordinated care. Examples ofclinical summarization are abundant in practice; pa-tient handoff summaries facilitate provider shift change,progress notes provide a daily status update for a pa-tient, oral case presentations enable transfer of infor-mation from overnight admission to the care team andattending, and discharge summaries provide informa-tion about a patient’s hospital visit to their primarycare physician and other outpatient providers (Feblowitzet al., 2011).

Informal, unstructured, or poor quality summaries canlead to communication failures and even medical errors,yet clinical instruction on how to formulate clinical sum-maries is ad hoc and informal. Non-existent or limitedsearch functionality, fragmented data sources, and lim-ited visualizations in electronic health records (EHRs)make summarization challenging for providers (Chris-tensen & Grimsmo, 2008; Singh et al., 2013; Natarajanet al., 2010). Furthermore, while dictation of EHR notes

Submitted for MIT’s 6.872 class project,Copyright 2017 by the authors.

allows clinicians to more efficiently document informa-tion at the point of care, the stream of consciousness-like writing can hinder the readability of notes. Kri-palani et al. show that discharge summaries are oftenlacking key information, including treatment progressionand follow-up protocols, which can hinder communica-tion between hospital and community based clinicians(Kripalani et al., 2007). Recently, St. Thomas Hospi-tal in Nashville, TN stipulated that discharge notes bewritten within 48 hours of discharge following incidenceswhere improper care was given to readmitted patientsbecause the discharge summary for the previous admis-sion was not completed(Alsentzer).

Automated summary generation has the potential tosave clinician time, avoid medical errors, and aid clin-ical decision making. By organizing and synthesizinga patient’s salient medical history, algorithms for pa-tient summarization can enable better communicationand care, particularly for chronically ill patients, whosemedical records often contain hundreds of notes. In thiswork, we explore the automatic summarization of dis-charge summary notes, which are critical to ensuringcontinuity of care after hospitalization. We (1) providean upper bound on extractive summarization by assess-ing how much information in the discharge note can befound in the rest of the patient’s EHR notes and (2)develop a classifier for labeling the topics of history ofpresent illness notes, a narrative section in the dischargesummary that describes the patient’s prior history andcurrent symptoms. Such a classifier can be used to createtopic specific evaluation sets for methods that performextractive summarization. These aims are critical stepsin ultimately developing methods that can automate dis-charge summary creation.

2. Related Work

In the broader field of summarization, automization wasmeant to standardize output while also saving time andeffort. Pioneering strategies in summarization startedby extracting ”significant” sentences in the whole corpusto build an abstract where ”significant” sentences weredefined by the number of frequently occurring words(Luhn, 1958). These initial methods did not consider

arX

iv:1

810.

1208

5v1

[cs

.IR

] 2

6 O

ct 2

018

Extractive Summarization of Electronic Health Record Discharge Notes

word meaning or syntax at either the sentence or para-graph level, which made them crude at best. More ad-vanced extractive heuristics like topic modeling (Allah-yari & Kochut, 2016), cue word dictionary approaches(Edmundson, 1969), and title methods (Ferreira et al.,2013) for scoring content in a sentence followed soonafter. For example, topic modeling extends initial fre-quency methods by assigning topics scores by frequencyof topic signatures, clustering sentences with similar top-ics, and finally extracting the centroid sentence, which isconsidered the most representative sentence (Allahyariet al., 2017). Recently, abstractive summarization ap-proaches using sequence-to-sequence methods have beendeveloped to generate new text that synthesizes origi-nal text(Paulus et al., 2017; Nallapati et al., 2016; Liu& Pan, 2016; Rush et al., 2015); however, the field ofabstractive summarization is quite young.

Existing approaches within the field of electronic healthrecord summarization have largely been extractive andindicative, meaning that summaries point to importantpieces in the original text rather than replacing the orig-inal text altogether. Few approaches have been deployedin practice, and even fewer have demonstrated impact onquality of care and outcomes (Pivovarov, 2016). Summa-rization strategies have ranged from extraction of rele-vant sentences from the original text to form the sum-mary (Moen et al., 2016), topic modeling of EHR notesusing Latent Dirichlet allocation (LDA) or bayesian net-works (Pivovarov, 2016), and knowledge based heuristicsystems (Goldstein & Shahar, 2016). To our knowledge,there is no literature to date on extractive or abstractiveEHR summarization using neural networks.

3. Methods

3.1. Data

MIMIC-III is a freely available, deidentified databasecontaining electronic health records of patients admittedto an Intensive Care Unit (ICU) at Beth Israel DeaconessMedical Center between 2001 and 2012. The databasecontains all of the notes associated with each patient’stime spent in the ICU as well as 55,177 discharge reportsand 4,475 discharge addendums for 41,127 distinct pa-tients. Only the original discharge reports were includedin our analyses. Each discharge summary was dividedinto sections (Date of Birth, Sex, Chief Complaint, Ma-jor Surgical or Invasive Procedure, History of PresentIllness, etc.) using a regular expression.

3.2. Upper Bound on Summarization

Extractive summarization of discharge summaries relieson the assumption that the information in the discharge

summary is documented elsewhere in the rest of the pa-tient’s notes. However, sometimes clinicians will docu-ment information in the discharge summary that mayhave been discussed throughout the hospital visit, butwas never documented in the EHR. Thus, our first aimwas to determine the upper bound of extractive summa-rization.

For each patient, we compared the text of the dischargesummary to the remaining notes for the patient’s currentadmission as well as their entire medical record. Con-cept Unique Identifiers (CUIs) from the Unified Med-ical Language System (UMLS) were compared in or-der to assess whether clinically relevant concepts in thedischarge summary could be located in the remainingnotes (NLM, 2009). CUIs were extracted using ApachecTAKES (Savova et al., 2010) and filtered by removingthe CUIs that are already subsumed by a longer span-ning CUI. For example, CUIs for ”head” and ”ache” wereremoved if a CUI existed for ”head ache” in order to ex-tract the most clinically relevant CUIs.

In order to understand which sections of the dischargesummaries would be the easiest or most difficult to sum-marize, we performed the same CUI overlap comparisonfor the chief complaint, major surgical or invasive proce-dure, discharge medication, and history of present illnesssections of the discharge note separately. We calculatedwhich fraction of the CUIs in each section were locatedin the rest of the patient’s note for a specific hospitalstay. We also calculated what percent of the gendersrecorded in the discharge summary were also recordedin the structured data for the patient.

3.3. Labeling History of Present Illness Notes

3.3.1. Annotation

We developed a classifier to label topics in the history ofpresent illness (HPI) notes, including demographics, di-agnosis history, and symptoms/signs, among others. Arandom sample of 515 history of present illness notes wastaken, and each of the notes was manually annotatedby one of eight annotators using the software Multi-document Annotation Environment (MAE) (Rim, 2016).MAE provides an interactive GUI for annotators and ex-ports the results of each annotation as an XML file withtext spans and their associated labels for additional pro-cessing. 40% of the HPI notes were labeled by cliniciansand 60% by non-clinicians. Table 1 shows the instruc-tions given to the annotators for each of the 10 labels.The entire HPI note was labeled with one of the labels,and instructions were given to label each clause in a sen-tence with the same label when possible.

Extractive Summarization of Electronic Health Record Discharge Notes

Table 1. HPI Categories and Annotation Instructions

Demographics Age, gender, race/ethnicity,language, social history etc.

DiagnosisHistory Diagnoses prior to currentsymptoms

MedicationHistory Medications or treatmentprior to current symptoms

ProcedureHistory Procedures prior to currentsymptoms

Symptoms/Signs Current Signs/Symptoms;chief complaint

Vitals/Labs Any vitals or labs related tocurrent care episode

Procedures/Results Procedures including CT,CXR, etc. and their results

Meds/Treatments Medication or treatment ad-ministered in current careepisode

Movement Description of patient admit-ted/transferred between hos-pitals or care units

Other Does not fit into any othercategory

3.3.2. Model

Our LSTM model was adopted from prior work by Der-noncourt et al (Dernoncourt et al., 2016). Whereas theDernoncourt model jointly classified each sentence in amedical abstract, here we jointly classify each word inthe HPI summary. Our model consists of four layers:a token embedding layer, a word contextual represen-tation layer, a label scoring layer, and a label sequenceoptimization layer (Figure 1).

In the following descriptions, lowercase italics is used todenote scalars, lowercase bold is used to denote vectors,and uppercase italics is used to denote matrices.

Token Embedding Layer: In the token embeddinglayer, pretrained word embeddings are combined withlearned character embeddings to create a hybrid tokenembedding for each word in the HPI note. The wordembeddings, which are direct mappings from word xi tovector t, were pretrained using word2vec (Mikolov et al.,2013a;b;c) on all of the notes in MIMIC (v30) and onlythe discharge notes. Both the continuous bag of words(CBOW) and skip gram models were explored.

Let z1:l be the sequence of characters comprising theword xi. Each character is mapped to its embedding ci,and all embeddings are input into a bidirectional LSTM,which ultimately outputs c, the character embedding of

the word xi.

The output of the token embedding layer is the vectore, which is the result of concatenation of the word em-bedding, t, and the character embedding, c.

Contextual Representation Layer: The contextualrepresentation layer takes as input the sequence of wordembeddings, e1:k, and outputs an embedding of thecontextual representation for each word in the HPInote. The word embeddings are fed into a bi-directionalLSTM, which outputs hj , a concatenation of the hiddenstates of the two LSTMs for each word.

Label Scoring Layer: At this point, each word xi isassociated with a hidden representation of the word, hj .In the label scoring layer, we use a fully connected neuralnetwork with one hidden layer to output a score associ-ated with each of the 10 categories for each word. LetW ∈ R10Xk and b ∈ R10. We can compute a vector ofscores s = W · h+ b where the ith component of s is thescore of class i for a given word.

Label Sequence Optimization Layer: The Label Se-quence Optimization Layer computes the probability ofa labeling sequence and finds the sequence with the high-est probability. In order to condition the label for eachword on the labels of its neighbors, we employ a linearchain conditional random field (CRF) to define a globalscore, Q, for a sequence of words and their associatedscores s1, ..., sm and labels, y1, ..., yn:

Q(y1, ..., yn) = b[y1] +

m∑t=1

st[yt] +

m−1∑t=1

T [yt, yt+1] + ε[ym]

(1)where T is a transition matrix ∈ R10X10 and b, ε ∈ R10

are vectors of scores that describe the cost of beginningor ending with a label.

The probability of a sequence of labels is calculated byapplying a softmax layer to obtain a probability of asequence of labels:

P (y1, ..., yn) =eQ(y1,...,yn)∑

y1,...,yneQ(y1,...,yn)

(2)

Cross-entropy loss, −log(P (y)), is used as the objectivefunction where y is the correct sequence of labels andthe probability P (y) is calculated according to the CRF.

3.3.3. Running the Model

We evaluated our model on the 515 annotated history ofpresent illness notes, which were split in a 70% train set,15% development set, and a 15% test set. The modelis trained using the Adam algorithm for gradient-basedoptimization (Kingma & Ba, 2014) with an initial learn-ing rate = 0.001 and decay = 0.9. A dropout rate of

Extractive Summarization of Electronic Health Record Discharge Notes

Figure 1. LSTM Model for sequential HPI word classifica-tion. Xi: token i, Z1 to Zl: characters of Xi, t: word em-beddings, c: character embeddings, e1:k: hybrid token em-beddings, h1:j : contextual representation embedings, s1:m

: score vectors where s[i] is the score for category i,y1:m: output word labels.

0.5 was applied for regularization, and each batch size =20. The model ran for 20 epochs and was halted early ifthere was no improvement after 3 epochs.

We evaluated the impact of character embeddings, thechoice of pretrained w2v embeddings, and the addition oflearned word embeddings on model performance on thedev set. We report performance of the best performing

model on the test set.

4. Results and Discussion

Figure 2. Relationship between the number of non-dischargenotes (top), number of non-discharge CUIs (middle), or thenumber of hours the patient spent outside the ICU (bottom)and the recall for each discharge summary.

4.1. Upper Bound on Summarization

For each of the 55,177 discharge summary reports inthe MIMIC database, we calculated what fraction ofthe CUIs in the discharge summary could be found inthe remaining notes for the patient’s current admission

Extractive Summarization of Electronic Health Record Discharge Notes

(by hadm id) and in their entire longitudinal medicalrecord (by subject id). Table 2 shows the CUI recall aver-aged across all discharge summaries by both subject idand hadm id. The low recall suggests that cliniciansmay incorporate information in the discharge note thathad not been previously documented in the EHR. Fig-ure 2 plots the relationship between the number of non-discharge notes for each patient and the CUI recall (top)and the number of total CUIs in non-discharge notes andthe CUI recall (middle). The number of CUIs is a proxyfor the length of the notes, and as expected, the CUI re-call tends to be higher in patients with more and longernotes. The bottom panel in Figure 2 demonstrates thatrecall is not correlated with the patient’s length of stayoutside the ICU, which indicates that our upper boundcalculation is not severely impacted by only having ac-cess to the patient’s notes from their stay in the ICU.

Finally, Table 3 shows the recall for the sex, chief com-plaint, procedure, discharge medication, and HPI dis-charge summary sections averaged across all the dis-charge summaries. The procedure section has the high-est recall of 0.807, which is understandable because pro-cedures undergone during an inpatient stay are mostlikely to be documented in an EHR. The recall for eachof these five sections is much higher than the overall re-call in Table 2, suggesting that extractive summarizationmay be easier for some sections of the discharge note.

Overall, this upper bound analysis suggests that we maynot be able to recreate a discharge summary with extrac-tive summarization alone. While CUI comparison allowsfor comparing medically relevant concepts, cTAKES’sCUI labelling process is not perfect, and further work,perhaps through sophisticated regular expressions, isneeded to define the limits of extractive summarization.

Table 2. Discharge CUI recall for each patient’s currenthospital admission (by hadm id) and their entire EHR(by subject id), averaged across all discharge summaries

Comparison Average RecallBy subject id 0.431By hadm id 0.375

4.2. Labeling History of Present Illness Notes

Table 4 compares dev set performance of the model us-ing various pretrained word embeddings, with and with-out character embeddings, and with pretrained versuslearned word embeddings. The first row in each sec-tion is the performance of the model architecture de-scribed in the methods section for comparison. Modelsusing word embeddings trained on the discharge sum-maries performed better than word embeddings trained

Table 3. Average Recall for five sections of the discharge sum-mary. Recall for each patient’s sex was calculated by exam-ining the structured data for the patient’s current admission,and recall for the remaining sections was calculated by com-paring CUI overlap between the section and the remainingnotes for the current admission.

Section Average RecallSex 0.675Chief Complaint 0.720Procedure 0.807Discharge Medication 0.580HPI 0.665

on all MIMIC notes, likely because the discharge sum-mary word embeddings better captured word use in dis-charge summaries alone. Interestingly, the continuousbag of words embeddings outperformed skip gram em-beddings, which is surprising because the skip gram ar-chitecture typically works better for infrequent words(Google, 2013). As expected, inclusion of character em-beddings increases performance by approximately 3%.The model with word embeddings learned in the modelachieves the highest performance on the dev set (0.886),which may be because the pretrained worm embeddingswere trained on a previous version of MIMIC. As a re-sult, some words in the discharge summaries, such asmi-spelled words or rarer diseases and medications, didnot have associated word embeddings. Performing a sim-ple spell correction on out of vocab words may improveperformance with pretrained word embeddings.

We evaluated the best performing model on the test set.The Learned Word Embeddings model achieved an ac-curacy of 0.88 and an F1-Score of 0.876 on the test set.Table 5 shows the precision, recall, F1 score, and supportfor each of the ten labels, and Figure 3 shows the con-fusion matrix illustrating which labels were frequentlymisclassified. The demographics and patient movementlabels achieved the highest F1 scores (0.96 and 0.93 re-spectively) while the vitals/labs and medication historylabels had the lowest F1 scores (0.40 and 0.66 respec-tively). The demographics section consistently occurs atthe beginning of the HPI note, and the patient move-ment section uses a limited vocab (transferred, admit-ted, etc.), which may explain their high F1 scores. Onthe other hand, the vitals/labs and medication historysections had the lowest support, which may explain whythey were more challenging to label.

Words that belonged to the diagnosis history, patientmovement, and procedure/results sections were fre-quently labeled as symptoms/signs (Figure 3). Diagno-sis history sections may be labeled frequently as symp-toms/signs because symptoms/diseases can either be de-

Extractive Summarization of Electronic Health Record Discharge Notes

scribed as part of the patient’s diagnosis history or cur-rent symptoms depending on when the symptom/diseaseoccurred. However, many of the misclassification errorsmay be due to inconsistency in manual labelling amongannotators. For example, sentences describing both pa-tient movement and patient symptoms (e.g. ”the patientwas transferred to the hospital for his hypertension”)were labeled entirely as ’patient movement’ by someannotators while other annotators labeled the differentclauses of the sentence separately as ’patient movement’and ’symptoms/signs.’ Further standardization amongannotators is needed to avoid these misclassifications.Future work is needed to obtain additional manual an-notations where each HPI note is annotated by multipleannotators. This will allow for calculation of Cohen’skappa, which measures inter-annotator agreement, andcomparison of clinician and non-clinician annotator reli-ability.

Future work is also needed to better understand com-monly mislabeled categories and explore alternativemodel architectures. Here we perform word level labelprediction, which can result in phrases that contain mul-tiple labels. For example, the phrase ”history of neckpain” can be labeled with both ’diagnosis history’ and’symptoms/signs’ labels. Post-processing is needed tocreate a final label prediction for each phrase. Whilephrase level prediction may resolve these challenges, itis difficult to segment the HPI note into phrases for pre-diction, as a single phrase may truly contain multiplelabels. Segmentation of sentences by punctuation, con-junctions, and prepositions may yield the best phrasechunker for discharge summary text.

Finally, supplementing the word embeddings in ourLSTM model with CUIs may further improve perfor-mance. While word embeddings do well in learning thecontextual context of words, CUIs allow for more explicitincorporation of medical domain expertise. By concate-nating the CUI for each word with its hybrid token em-bedding, we may be able to leverage both data drivenand ontology driven approaches.

5. Conclusion

In this paper we developed a CUI-based upper boundon extractive summarization of discharge summaries andpresented a NN architecture that jointly classifies wordsin history of present illness notes. We demonstrate thatour model can achieve excellent performance on a smalldataset with known heterogeneity among annotators.This model can be applied to the 55,000 discharge sum-maries in MIMIC to create a dataset for evaluation ofextractive summarization methods.

Table 4. F1-Scores on the development set under variousmodel architectures. The first model in each section is thesame model described in the methods section. Top perform-ing models in each section are in bold.

Model F1w2v CBOW; Discharge Summaries 0.873w2v Skip Gram; Discharge Summaries 0.831w2v CBOW; All MIMIC Notes 0.862w2v Skip Gram; All MIMIC Notes 0.809

With Character Embeddings 0.873Without Character Embeddings 0.847

Pretrained Word Embeddings 0.873Learned Word Embeddings 0.886

Table 5. Precision (P), Recall (R), F1 Score, and Numberof supporting examples for each of the 10 label categories forthe Learned Word Embedding model on the test set.

Label P R F1 SupportDemographics 0.96 0.95 0.96 1489DiagnosisHistory 0.83 0.82 0.83 3156MedicationHistory 0.80 0.56 0.66 557ProcedureHistory 0.85 0.87 0.86 1835Symptoms/Signs 0.81 0.83 0.82 2235Vitals/Labs 0.83 0.26 0.40 486Procedures/Results 0.86 0.72 0.78 1114Meds/Treatments 0.93 0.87 0.90 4165PatientMovement 0.89 0.97 0.93 10933Other 0.90 0.85 0.88 1815Average/Total 0.88 0.88 0.88 27785

Figure 3. Confusion Matrix for the Learned Word Embed-ding model on the test set. True labels are on the x-axis, andpredicted labels are on the y-axis.

Extractive Summarization of Electronic Health Record Discharge Notes

Acknowledgments

We would like to thank our annotators, AndrewGoldberg, Laurie Alsentzer, Elaine Goldberg, AndyAlsentzer, Grace Lo, and Josh Donis. We would alsolike to acknowledge Pete Szolovits for his guidance andfor providing the pretrained word embeddings and Tris-tan Naumann for providing the MIMIC CUIs.

References

Allahyari, Mehdi and Kochut, Krys. Semantic taggingusing topic models exploiting wikipedia category net-work. In Semantic Computing (ICSC), 2016 IEEETenth International Conference on, pp. 63–70. IEEE,2016.

Allahyari, Mehdi, Pouriyeh, Seyedamin, Assefi, Mehdi,Safaei, Saeid, Trippe, Elizabeth D, Gutierrez, Juan B,and Kochut, Krys. Text summarization techniques: Abrief survey. arXiv preprint arXiv:1707.02268, 2017.

Alsentzer, L. personal communication.

Christensen, T. and Grimsmo, A. (n.d.). Instant avail-ability of patient records, but diminished availabilityof patient information: A multi-method study of gpsuse of electronic patient records. BMC Medical Infor-matics and Decision Making, 8(12), 2008.

Dernoncourt, Franck, Young Lee, Ji, and Szolovits, Pe-ter. Neural networks for joint sentence classificationin medical paper abstracts. pp. 694700, 12 2016.

Edmundson, H. P. New methods in automatic ex-tracting. J. ACM, 16(2):264–285, April 1969. ISSN0004-5411. doi: 10.1145/321510.321519. URL http:

//doi.acm.org/10.1145/321510.321519.

Feblowitz, J. C., Wright, A., Singh, H., Samal, L., andSittig, D. F. Summarization of clinical information: Aconceptual model. Journal of Biomedical Informatics,44:688–699, 2011.

Ferreira, Rafael, de Souza Cabral, Luciano, Lins,Rafael Dueire, e Silva, Gabriel Pereira, Freitas,Fred, Cavalcanti, George DC, Lima, Rinaldo, Simske,Steven J, and Favaro, Luciano. Assessing sentencescoring techniques for extractive text summarization.Expert systems with applications, 40(14):5755–5764,2013.

Goldstein, A. and Shahar, Y. An automated knowledge-based textual summarization system for longitudi-nal, multivariate clinical data. Journal of Biomedi-cal Informatics, 61(Supplement C):159 – 175, 2016.ISSN 1532-0464. doi: https://doi.org/10.1016/j.jbi.

2016.03.022. URL http://www.sciencedirect.com/

science/article/pii/S1532046416300156.

Google. Google Code Archive word2vec, 2013. URLhttps://code.google.com/archive/p/word2vec/.

Kingma, Diederik and Ba, Jimmy. Adam: Amethod for stochastic optimization. arXiv preprintarXiv:1412.6980, 2014.

Kripalani, Sunil, Lefevre, Frank, Phillips, Christopher,V Williams, Mark, Basaviah, Preetha, and Baker,David. Deficits in communication and informationtransfer between hospital-based and primary carephysicians. JAMA : the journal of the American Med-ical Association, 297:831–41, 03 2007.

Liu, P. and Pan, X. Google Research Blog textsummarization with tensorflow, 2016. URLhttps://research.googleblog.com/2016/08/

text-summarization-with-tensorflow.html.

Luhn, H. P. The automatic creation of literature ab-stracts. IBM J. Res. Dev., 2(2):159–165, April 1958.ISSN 0018-8646. doi: 10.1147/rd.22.0159. URL http:

//dx.doi.org/10.1147/rd.22.0159.

Mikolov, Tomas, Chen, Kai, Corrado, Greg, and Dean,Jeffrey. Efficient estimation of word representations invector space. arXiv preprint arXiv:1301.3781, 2013a.

Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado,Greg S, and Dean, Jeff. Distributed representations ofwords and phrases and their compositionality. In Ad-vances in neural information processing systems, pp.3111–3119, 2013b.

Mikolov, Tomas, Yih, Wen-tau, and Zweig, Geoffrey.Linguistic regularities in continuous space word rep-resentations. In hlt-Naacl, volume 13, pp. 746–751,2013c.

Moen, H., Peltonen, L.M., Heimonen, J., Airola, A.,Pahikkala, T., Salakoski, T., and Salantera, T. Com-parison of automatic summarisation methods forclinical free text notes. Artificial Intelligence inMedicine, 67(Supplement C):25 – 37, 2016. ISSN0933-3657. doi: https://doi.org/10.1016/j.artmed.2016.01.003. URL http://www.sciencedirect.com/

science/article/pii/S0933365716000051.

Nallapati, Ramesh, Zhou, Bowen, Gulcehre, Caglar, Xi-ang, Bing, et al. Abstractive text summarization usingsequence-to-sequence rnns and beyond. arXiv preprintarXiv:1602.06023, 2016.

Extractive Summarization of Electronic Health Record Discharge Notes

Natarajan, K., Stein, D., Jain, S., and Elhadad,N. (n.d.). An analysis of clinical queries in an elec-tronic health record search utility. International Jour-nal of Medical Informatics, 79(7):515–522, 2010.

NLM. UMLS Reference Manual. National Library ofMedicine, 2009.

Paulus, Romain, Xiong, Caiming, and Socher, Richard.A deep reinforced model for abstractive summariza-tion. arXiv preprint arXiv:1705.04304, 2017.

Pivovarov, R. Electronic Health Record SummarizationOver Heterogenous and Irregularly Sampled ClinicalData. PhD thesis, Department of Biomedical Infor-matics, Columbia University, 2016.

Rim, Kyeongmin. Mae2: Portable annotation tool forgeneral natural language use. in proceedings of the12th joint acl-iso workshop on interoperable semanticannotation. 05 2016.

Rush, Alexander M, Chopra, Sumit, and Weston, Jason.A neural attention model for abstractive sentence sum-marization. arXiv preprint arXiv:1509.00685, 2015.

Savova, Guergana K, Masanz, James J, Ogren, Philip V,Zheng, Jiaping, Sohn, Sunghwan, Kipper-Schuler,Karin C, and Chute, Christopher G. Mayo clinical textanalysis and knowledge extraction system (ctakes):architecture, component evaluation and applications.Journal of the American Medical Informatics Associ-ation, 17(5):507–513, 2010.

Singh, H., Spitzmueller, C., Petersen, N. J., Sawhney,M. K., and Sittig, D. F. (n.d.). Information overloadand missed test results in electronic health record-based settings. JAMA Internal Medicine, 173(8),2013.