text mining the maude database

Company Confidential - For Internal Use Only

Copyright © 2010, SAS Institute Inc. All rights reserved.

Signal Detection of Adverse Medical Device Events in the FDA MAUDE Database Eric Brinsfield, MS Presenter & Research Collaborator David Olaleye, MSCE, PhD Author & Primary Research Statistician SAS Institute Inc. Cary, NC

2



Disclosure Statement

Both presenters are employees of SAS Institute

We have no conflicts of interest

Disclaimer

The views and opinions expressed in the following PowerPoint slides are those of the individual presenters and should not be attributed to ISPE or to SAS Institute.

3



Study Objectives

3

Determine if text mining can be used to:

Detect signals of adverse events in

spontaneous reporting data

Better understand or triage signals

generated by traditional disproportionality

methods

Phase 1:

Evaluate unsupervised text mining

Using FDA MAUDE database

Focused on stents

http://images.google.com/imgres?imgurl=http://www.blisstree.com/files/50/2007/02/coronarystents.jpg&imgrefurl=http://www.blisstree.com/articles/drug-eluting-coronary-stents-in-the-new-england-journal-of-medicine-50/&usg=__s5k1uH-ieOTlEQNrLY9Q8jnxJu4=&h=720&w=463&sz=64&hl=en&start=5&um=1&tbnid=DVgWOfqRlLl0SM:&tbnh=140&tbnw=90&prev=/images?q=coronary+stent&hl=en&sa=X&um=1

4



Definitions

Safety Signal

“A report or reports of an event with an unknown causal relationship to treatment that is recognized as worthy of further exploration and continued surveillance.”

» Council for International Organizations of Medical Sciences (CIOMS)

“Recognition” is often the results of

Analytical or automatic signal detection methods that look for unexpected patterns in data sources such as:

» Spontaneously reported data

» Observational healthcare data

» Insurance claims data

http://www.mimg.com/images/stent.gif

5



Methods

5

Data Source = MAUDE

FDA Spontaneous Reporting System Database for Med. Devices

MAUDE - Manufacturer and User Facility Device Experience

Contains the narrative entered by the reporter

Target Devices

endovascular graft system and coronary stent devices

devices classified as a stent in the “product_category_code”:

» MAF, MIH, NIN, NIO, NIP, NIM, and NIQ

» http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPCD/classif

ication.cfm?ID=896

http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPCD/classification.cfm?ID=896

http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfPCD/classification.cfm?ID=896

http://images.google.com/imgres?imgurl=http://www.blisstree.com/files/50/2007/02/coronarystents.jpg&imgrefurl=http://www.blisstree.com/articles/drug-eluting-coronary-stents-in-the-new-england-journal-of-medicine-50/&usg=__s5k1uH-ieOTlEQNrLY9Q8jnxJu4=&h=720&w=463&sz=64&hl=en&start=5&um=1&tbnid=DVgWOfqRlLl0SM:&tbnh=140&tbnw=90&prev=/images?q=coronary+stent&hl=en&sa=X&um=1

6



MAUDE (2001-2008)

Spontaneous “safety” reports on medical devices

Strengths:

Only surveillance system which covers devices marketed in the entire US

Largest number of case reports on adverse outcomes and malfunctions for medical devices

Provides opportunity to detect signals of new, rare and unusual adverse clinical outcomes

» Usually warrants further investigation

Includes narrative description of events

» Better than AERS which does not include narrative

7



Characteristics of Spontaneous Report

Suitable for hypotheses generation; not for confirmation or rates computation

Exhibit under-reporting and other reporting biases; lack of control group, etc.

Data quality, ascertainment, accuracy and completeness of information are usually poor

Includes events and incidents not causally related to medical device exposure

Does not distinguish between label versus off-label uses of approved products

Contains minimal or no patient history and other potential causal factors

Do not provide estimates of exposure (worse in drugs)

8



Source Population All MAUDE reports received between 2006 – 2010

(N=35954)

Study Device Cohort MAUDE reports for stents

with product codes:

MAF, MIH, NIN, NIO,

NIP, NIM, and NIQ

Device-Adverse Outcomes Pairs

(N=28)

Study Events • Death (D)

• Injury (I)

• Malfunctions (M)

• Other (O)

Study Design: Case-Series

9



Analysis Steps

1. Attempt analysis using text mining alone

Ignore device

Evaluate if general text mining provides any insights

2. Perform standard disproportionality analysis on structured data

PRR, EBGM, Adj. Residual

3. Identify device-AE pairs that have:

High scores

Especially in the “Other” classification

4. Investigate the device with text mining

Include all outcome classifications

10



Text Mining Process

Parse terms to create documents-terms frequency matrix

Use singular value decomposition (SVD) to measure association and perform hierarchical clustering

Use entropy method to cluster SVDs for documents classification

11



Sample Narrative - Injury

Example of Manufacturer Report - Injury

the product labeling for a p154 states that this product is indicated for use in pts with obstruction of major biliary ducts. the product labeling also states that the stent may be increased post-placement by expanding with a larger diameter balloon. the following was obtained through conversation with the user facility on 1/15/98. after deciding that a ptca procedure in a renal artery did not yield adequate results, the md attempted to place a medium biliary stent in the artery. the physician reported to co. that he had difficulty visualizing the stent and that it was difficult to place. he deployed the stent, but was not satisfied with the outcome. in response, he decided to place a second stent inside of the first. however, the stents interlocked and the physician decided to remove both stents, he was able to withdraw the stent up to sheath tip in the femoral artery, but needed a vascular surgeon to completely remove them. info regarding the type of removal procedure has not been provided to co. the physician further stated that he believes that the first stent had not fully opened. it was mounted on a meditech glidex balloon.

http://images.google.com/imgres?imgurl=http://www.fda.gov/cdrh/MDA/DOCS/p030025.jpg&imgrefurl=http://www.fda.gov/cdrh/mda/docs/p030025.html&usg=__ay3LIlegYn1u57nPATsKrFCttBI=&h=271&w=370&sz=21&hl=en&start=11&um=1&tbnid=cncmyBiL69am4M:&tbnh=89&tbnw=122&prev=/images?q=coronary+stent&hl=en&sa=X&um=1

12



Concept Linking and Exploration

13



Text mining clusters over all reports

14



Text Mining Over All Reports

Results are interesting

But inconclusive

Lose track of the device

But may detect new trends for all stent devices in study

Cannot really do comparisons due to lack of denominator. (Same problems as always with spontaneous reports.)

Next, run disproportionality to narrow the focus…

15



Disproportionality Results for Carotid Stent (NIM)

Adverse Event Freq

Adjusted

Residual

MGPS

(EBGM)

Death 276 0.8 0.86728

Injury 2043 1.1 1.130264

Malfuntion 271 0.4 0.424803

Other 24 1.6 3.648191

* Adjusted Residual: Flagged at values over 1.5

EBGM: Flagged at values over 2.0

16



Target for Text Mining Evaluation

NIM showed high proportion of “Other”

Based on relative percentage of reports

Based on signal scoring algorithms

All methods suggested a flag

Although only 24 cases, the method could show promise

Run text mining against all NIM reports

Include all outcomes to fully understand reports

Look for possible explanations or hypotheses

17



Text Mining Clusters for NIM N

+percutaneous +unk dissection performance +unstable repaired

+malfunction 'device remains implanted'

376

reactions collapsing +'premature deployment' cracks +'product quality

issue' replaced performance collapsed

41

malfunction fractures +fracture +'inaccurate delivery' fractured

malfunctions drift +'premature deployment'

846

+normal dissection +unk 'no information' +na collapsing unk fractured 69

'device issue' broke 'no known device problem' +break broken +'shaft

break' reaction reactions

227

abnormal +fracture +continuous +bent cracks unk +break fractured 65

filter +na 'difficult to advance' breaks reaction +bent +malfunction

replaced

150

'device remains implanted' collapsed 'no flow' performance +crack

+break +collapse filter

44

fractured repair +crack breaks +collapse +'product quality issue' 'no

flow' reaction

31

18



19



Manual Review of Text for “Other”

Most were not adverse events that persisted

Some seemed like “FYI” reports.

Two included notification of a formal study

Most patients still had the stent in place (assumed)

Some cases of installation problems:

Potential installer error

Most did not involve an adverse event

20



20

Conclusion

Text mining shows promise for recognizing primary words and patterns

Hard to form hypotheses from bulk text mining on spontaneous database

Combination with disproportionality analysis creates signals that can be further analyzed with text mining

Terms in the “Other” category overlap with other categories

21



21

Next Steps

Need further analysis that includes:

Large target group for further triage

“Other” was too small in this case

Preferred term matching and encoding

To clean up fuzziness and reduce clusters

Content categorization

Look for more structure and combine with ontologies

Sentiment analysis

Determine if overall sentiment was good or bad

22



Final Thoughts

“What the future portends is more and more information — Everests of it. There won’t be anything we won’t know. But there will be no one thinking about it.”

From:

New York Times - August 13, 2011

The Elusive Big Idea By NEAL GABLER

Neal Gabler is a senior fellow at the Annenberg Norman Lear Center at the University of

Southern California and the author of “Walt Disney: The Triumph of the American Imagination.”

We need to help make time for thinking.



Thank You

Contacts:

Eric Brinsfield [email protected]

David Olaleye [email protected]

http://www.sas.com/

mailto:[email protected]

text mining the maude database

Documents

employees of sas institute

internal use onlycopyright

drugs7 company confidential

nc company confidential

narrative6company confidential

unsupervised text mining

general text mining

data quality