i2b2 Rheumatoid Arthritis DBPDefining RA in the electronic health

record for future studies

Elizabeth Karlson, MDAssociate Professor of Medicine

Harvard Medical SchoolBrigham and Women’s Hospital

Background: Partners Resources

• i2b2: “Informatics for Integrating Biology and the Bedside”

• RPDR: “Research Patient Data Repository”• Natural Language Processing (HiTEX)• Gold standard dataset:

– Training set: 500 manual chart reviews – Validation set: 400 manual chart reviews

Coded data

• ICD-9 codes for RA

• ICD-9 codes for related phenotypes– Lupus (SLE), psoriatic arthritis (PsA), juvenile

inflammatory arthritis (JIA)

• Lab results for RA related antibodies– Rheumatoid factor (RF), anti-CCP

• Medications– physician entry, escripts

NLP Concepts

NLP queries– Rheumatoid arthritis– RA-related antibodies

• Anti-CCP/RF/seropositive• Result coded as positive/negative

– RA Medications• Coded as any mention

– Radiographs: RA erosions• Coded as any erosion

Approach to develop RA cohort

RA Martn=29, 432

Predicted RA Casesn=3,585

≥ 1 ICD RAn=25,830

ORAnti-CCP n=3,602

↑ Sensitivity ↑ Specificity

Training setn=500

Validation setn=400

Classificationalgorithm

training

Classification algorithmStep 1: Develop gold standard training setStep 2: Identify variables important for predicting RAStep 3: Develop algorithm

Chart review results

• RA Mart, N=32,000– ICD9 = 714.xxx

OR– CCP test ordered

• Manual chart review for 500 patients– 20% validation rate– definite RA=100– possible/no RA= 400

Comparison of NLP to manual chart review

• Precision of NLP queries– Methotrexate 100%– Etanercept 100%– CCP+ 98.7%– Seropositive 96%– Erosion 88%

Approach to develop RA cohort

Classification algorithm Step 2: Define variables(Vivian Gainer, Sergey Goryachev, Qing Zeng-Treitler, Shawn Murphy)• Codified data

– ICD9 billing codes– Electronic medication prescription– CCP, RF lab results

• Narrative data extracted using natural language processing (NLP), i.e. from physician notes, radiology reports– Erosions– RF positive, CCP positive, seropositive– RA medications

RA Martn=29, 432

Predicted RA Casesn=3,585

≥ 1 ICD RAn=25,830

ORAnti-CCP n=3,602

↑ Sensitivity ↑ Specificity

Training setn=500

Validation setn=400

Classificationalgorithm

training

Approach to develop RA cohort

Classification algorithm

Step 3: Develop algorithm(Tianxi Cai)

• Penalized logistic regression with adaptive LASSO• Parsimonious predictors selected based on BIC

RA Martn=29, 432

Predicted RA Casesn=3,585

≥ 1 ICD RAn=25,830

ORAnti-CCP n=3,602

↑ Sensitivity ↑ Specificity

Training setn=500

Validation setn=400

Classificationalgorithm

training

Model RA

PPV (%)

Sensitivity (%)

Difference in PPV

Algorithms

Narrative +

Codified 3585 94 63 reference

Codified only 3046 88 51 6

NLP only 3341 89 56 5

Published administrative codified criteria

≥ 3 ICD9 RA 7960 56 80 38

≥1 ICD9RA + med 7799 45 66 49

Top 5 predictive variables for RA

i2b2 RA cohort

Liao, et al., Arthritis Care & Research 2010

*Consortium of Rheumatology Researchers of North America

i2b2 Virtual RA Cohort Studies

• Case-control cohort– ~4,000 RA cases– ~13,000 matched non-RA controls

• Age, gender, race and health care utilization

• Samples collected from 1500 cases/1500 controls for genotyping– Genetic risk score predicts RA with same magnitude

as in GWAS (Kurreman, 2010)– CAD outcomes in RA cases being validated in i2b2

• Pharmacogenetics Research Network (PGRN)

i2b2 RA Project:

• Selected codified data from RPDR

• Performed NLP queries for RA features

• Developed algorithm based on:

coded + NLP data

Liao, 2010

PGRN Methods:

• Select codified data from RPDR (meds)

• Perform NLP queries for RA disease activity features

• Develop algorithm (s) based on:

Meds + NLP data

PGRN Specific Aims

• Aim 1: Define RA disease activity level in the EMR

• Aim 2: Develop an algorithm to predict RA disease activity from EMR data

• Aim 3: Define temporal relations between RA medications and disease activity to define to define treatment response in RA

Background

• In RA, disease activity score (DAS28) is considered the gold standard tool to evaluate disease activity and response to treatment in clinical practice

• DAS28 has 2 components:– Disease activity level– Change in disease activity level

Van Gestel AM et al. Arthritis Rheum 1996; 39: 34-40

• Disease activity level scored as low, moderate, high

• Disease activity change scored as low, moderate, high

Research Methods

• Construct a virtual cohort of RA patients (N=5906)• Review charts for disease activity (document level)

– Remission– Low– Moderate Remission/Low vs. High/Moderate– High– Indeterminant

• Annotate charts for disease activity features (Knowtator) – Disease_disorder– Symptoms (reported pain, stiffness, swelling)– Signs (objective tenderness, limited range of motion, synovitis)– Anatomic site (relations with signs and symptoms)– RA medication signature– RA labs, level of inflammation (CRP, ESR)– Patient functioning (activities of daily living)

NLP Methods

• Move from keyword matching in i2b2 to ontology mapping in PGRN

• Customize cTAKES for– RA medications– RA anatomic sites

• Find relations between entities • Define new modules

– RA medication changes (start/stop)– Reasons to stop medications– Lab values– Patient functioning status

NLP Analytic Approaches

1- Internal gold standard datasets– N=200 BWH annotated notes– N= 200 MGH annotated notes

2- Analyses– Study whether MD summary (1-3 sentences) predicts disease

activity– SVM: construct vectors based on features and relations to

predict disease activity– Bag of concepts to predict disease activity 

2- External gold standard datasets: – DAS28 scores from standardized tool at MGH matched to

clinical note– DAS28 scores from BRASS matched to clinical note

Future work

• Define temporal relations between anti-TNF medication use (eg. new starts) and pre and post start disease activity to define response to therapy– Construct disease activity timeline (patient

level)– Construct medication timeline (patient level)

Use NLP to define temporal sequence of medication start and adverse event

Questions?