Medical Record Reviews – Medical Record Reviews – The Rules of the RoadThe Rules of the Road

David H. Rubin, MDDavid H. Rubin, MD

Department of PediatricsDepartment of Pediatrics

INTRODUCTIONINTRODUCTION

• Any study that uses pre-recorded patient Any study that uses pre-recorded patient focused data as the primary source of focused data as the primary source of information in a research studyinformation in a research study Physician, nurses notesPhysician, nurses notes Ambulance call reportsAmbulance call reports Diagnostic testsDiagnostic tests Clinic, administrative, government recordsClinic, administrative, government records Computerized databasesComputerized databases

WHY SELECT THIS DESIGN?WHY SELECT THIS DESIGN?

• Allows research to address issues that cannot Allows research to address issues that cannot be addressed with prospective studiesbe addressed with prospective studies Effect of harmful exposures (no randomization Effect of harmful exposures (no randomization

possible)possible) Effect of potentially beneficial exposuresEffect of potentially beneficial exposures Occurrence of rare eventsOccurrence of rare events Studies of patterns of disease or behaviorStudies of patterns of disease or behavior Quality assurance studiesQuality assurance studies Studies where cases may be shared (trauma Studies where cases may be shared (trauma

database)database) Pilot studies for prospective studies Pilot studies for prospective studies

DATA QUALITYDATA QUALITY

• ““Free form” quality of medical records may Free form” quality of medical records may increase missing and/or erroneous dataincrease missing and/or erroneous data

• Handwriting may be illegible or Handwriting may be illegible or uninterruptibleuninterruptible

• May miss examining potential casesMay miss examining potential cases

• Computer vs paper recordsComputer vs paper records

• Data abstraction techniques require Data abstraction techniques require standardizationstandardization

MISSING OR CONFLICTING MISSING OR CONFLICTING DATADATA

• Missing information may lead to nonresponse Missing information may lead to nonresponse biasbias Subjects with missing information may be very Subjects with missing information may be very

different from subjects without any missing different from subjects without any missing informationinformation

• Missing values managementMissing values management Case deletion – bias and reduced sample size are Case deletion – bias and reduced sample size are

problemsproblems Case insertion – use what you have and compare Case insertion – use what you have and compare

missing/non missing cases for differences missing/non missing cases for differences

SAMPLE SIZESAMPLE SIZE

• Usually determined based on the Usually determined based on the summary measure and the size/width summary measure and the size/width of the confidence interval desired of the confidence interval desired An interval with a greater CI (eg 99% CI v An interval with a greater CI (eg 99% CI v

95% CI) is wider and more likely includes the 95% CI) is wider and more likely includes the true population valuetrue population value

The width of the CI depends on sample size The width of the CI depends on sample size

SAMPLINGSAMPLING

• Select all cases within a given time frameSelect all cases within a given time frame• For nonconsecutive sampling it is best to choose For nonconsecutive sampling it is best to choose

probability samplingprobability sampling Provides equal opportunity for each eligible case to Provides equal opportunity for each eligible case to

be selectedbe selected Use random number generatorUse random number generator

• Triage levelTriage level

• Incidental sampling – choosing most easily Incidental sampling – choosing most easily accessible casesaccessible cases

• Systematic sampling – choosing every xSystematic sampling – choosing every x thth case case

RELIABILITYRELIABILITY

• Very important Very important • Any differences in data extraction by 2 different Any differences in data extraction by 2 different

people?people?• KappaKappa

Value ranges from -1 (perfect disagreement) to 1 Value ranges from -1 (perfect disagreement) to 1 (perfect agreement)(perfect agreement)

K = [observed agreement (%) – expected agreement K = [observed agreement (%) – expected agreement (%) / [100% - expected agreement (%)](%) / [100% - expected agreement (%)]

Try to achieve kappa of 0.6 or better (60% Try to achieve kappa of 0.6 or better (60% agreement)agreement)

MINIMUM REQUIREMENTS FOR MINIMUM REQUIREMENTS FOR MEDICAL RECORD REVIEWSMEDICAL RECORD REVIEWS

(Lowenstein, 2005)(Lowenstein, 2005)

1)1) Explicit protocols for case selection/exclusionExplicit protocols for case selection/exclusion2)2) Abstractor trainingAbstractor training3)3) Precise definitions of key variablesPrecise definitions of key variables4)4) Use of standardized abstraction and coding Use of standardized abstraction and coding

formsforms5)5) Monitoring of abstractor performanceMonitoring of abstractor performance6)6) Blinding of abstractors to study hypothesis and Blinding of abstractors to study hypothesis and

patient groupspatient groups7)7) Testing of interrater reliabilityTesting of interrater reliability

QUALITY OF MEDICAL RECORD QUALITY OF MEDICAL RECORD REVIEWSREVIEWS

(Badcock, 2005)(Badcock, 2005)

• Observational study of medical record reviews Observational study of medical record reviews published in several emergency medicine published in several emergency medicine journalsjournals

• 107 articles analyzed107 articles analyzed• Clear aim reported in 93%Clear aim reported in 93%• Standard abstraction forms: 51%Standard abstraction forms: 51%• Interrater reliability: 25%Interrater reliability: 25%• Ethics approval: 68%Ethics approval: 68%• Sample size/power: 10%Sample size/power: 10%

MEDICAL RECORD STUDIES IN MEDICAL RECORD STUDIES IN EM RESEARCHEM RESEARCH

(Worster, 2005)(Worster, 2005)

• From 1993 -1998 assessed medical From 1993 -1998 assessed medical record review studies in 6 EM journalsrecord review studies in 6 EM journals

• 79 (14%) medical review studies in 563 79 (14%) medical review studies in 563 original research articles original research articles

• Adherence to methodological criteriaAdherence to methodological criteria Sampling method: 99%Sampling method: 99% Abstractor blinding to hypothesis: 4%Abstractor blinding to hypothesis: 4% Interobserver agreement for the 12 criteria Interobserver agreement for the 12 criteria

ranged from 57% - 95%ranged from 57% - 95%

ACCURACY IN HOSPITAL ACCURACY IN HOSPITAL DISCHARGE SUMMARIESDISCHARGE SUMMARIES

(Callen 2009)(Callen 2009)

• Retrospective analysis of 966 handwritten Retrospective analysis of 966 handwritten and 842 electronically generated and 842 electronically generated discharge summaries in Australiadischarge summaries in Australia

• 12.1% of handwritten and 13.3% of 12.1% of handwritten and 13.3% of electronic summaries contained errorselectronic summaries contained errors

• Medication omission was biggest problemMedication omission was biggest problem

• NO difference in training levelNO difference in training level

MISSING INFORMATION IN ED MISSING INFORMATION IN ED CHARTSCHARTS(Richason 2009)(Richason 2009)

• Examined case reports in 4 ED journals Examined case reports in 4 ED journals from 2000-2005 and used 11 reporting from 2000-2005 and used 11 reporting standardsstandards

• 1,316 case reports identified1,316 case reports identified

• Poor reporting of Poor reporting of Co-morbiditiesCo-morbidities OutcomesOutcomes Concurrent medicationsConcurrent medications

METHODOLOGY FOR RETROSPECTIVE METHODOLOGY FOR RETROSPECTIVE REVIEWS IN CHILD PSYCHIATRYREVIEWS IN CHILD PSYCHIATRY

• Conceive questionConceive question• Literature reviewLiterature review• Proposal methodsProposal methods• Create data abstraction instrument and Create data abstraction instrument and

manualmanual• Sample sizeSample size• Obtain IRB approvalObtain IRB approval• Pilot studyPilot study

SAMPLE SIZESAMPLE SIZE(Gearing 2006)(Gearing 2006)

• Estimate 10 charts per variable (Sackett, Estimate 10 charts per variable (Sackett, 1991)1991)

• Others estimate 5-7 charts/variableOthers estimate 5-7 charts/variable• Convenience sampling – select cases over Convenience sampling – select cases over

specific time periodspecific time period• Quota sampling – predetermined number Quota sampling – predetermined number

sampledsampled• Systematic sampling – every “nth” case Systematic sampling – every “nth” case

chosenchosen

PRACTICAL ISSUESPRACTICAL ISSUES

• Check all possible CPT codes for Check all possible CPT codes for diagnosis or procedure codediagnosis or procedure code Febrile seizure may have been coded as Febrile seizure may have been coded as

seizureseizure Gastroenteritis may have been coded as viral Gastroenteritis may have been coded as viral

syndromesyndrome

• Pilot your Data Abstraction FormPilot your Data Abstraction Form• Create detailed “Codebook” for your studyCreate detailed “Codebook” for your study

Especially critical if > 1 researcher on studyEspecially critical if > 1 researcher on study

REFERENCESREFERENCES• Worster A, Haines T. Advanced statistics: understanding Worster A, Haines T. Advanced statistics: understanding

medical record review (MRR) studies. Acad Emerg Med medical record review (MRR) studies. Acad Emerg Med 2004;11:187-192.2004;11:187-192.

• Lowenstein SR. Medical record reviews in emergency Lowenstein SR. Medical record reviews in emergency medicine: the blessing and the cure. Annals Emerg Med medicine: the blessing and the cure. Annals Emerg Med April 2005;45(4):452-455.April 2005;45(4):452-455.

• Babcock D et al. The quality of medical record review Babcock D et al. The quality of medical record review studies in the international emergency medicine literature. studies in the international emergency medicine literature. 2005;45(4):444-447.2005;45(4):444-447.

• Worster A. et al. Reassessing the methods of medical Worster A. et al. Reassessing the methods of medical record review studies 2005;45:448-451.record review studies 2005;45:448-451.

• Gearing et al. Methodology for Retrospective chart review Gearing et al. Methodology for Retrospective chart review in child adolescent psychiatry. J Can Acad Child Adoles in child adolescent psychiatry. J Can Acad Child Adoles Psychiatry 15:3:2006Psychiatry 15:3:2006

REFERENCESREFERENCES

• Callen JJ, McIntosh. Accuracy of Callen JJ, McIntosh. Accuracy of medication in hospital discharge medication in hospital discharge summaries. Int J. Med Inform. 2009;Oct 1summaries. Int J. Med Inform. 2009;Oct 1

• Richardson TP et al. Case reports Richardson TP et al. Case reports describing treatrments in the ED. BMC describing treatrments in the ED. BMC Emerg Med.2009. June 15:9:10. Emerg Med.2009. June 15:9:10.