RTI International is a trade name of Research Triangle Institute

www.rti.org

Nancy Berkman, PhD Meera Viswanathan, PhDEmail: berkman@rti.org Email: viswanathan@rti.org

Development of a Tool to Evaluate the Quality of Non-randomized Studies of

Interventions or ExposuresPresented by

Nancy D Berkman, PhD & Meera Viswanathan, PhDPresented at

AHRQ 2009 Annual ConferenceBethesda, MarylandSeptember 15, 2009

www.rti.org

Acknowledgements

• Project funding provided by

– Phase 1:• Grant from RTI Independent Research and Development

(IR&D) funds

– Phase 2:• Contract from Agency for Healthcare Research and

Quality (AHRQ), U.S. Department of Health and Human Services, through the Evidence-based Practice Centers program (EPC)

www.rti.org

Context for the Project

• Increasing demand to include non-randomized studies in systematic literature reviews and comparative effectiveness reviews to capture– The effects of interventions or exposures on a more broadly defined

population than can be observed through RCTs– Topics where RCTs would be logistically or ethically inappropriate– Longer term outcomes and harms (side effects)

• The trade-off for wider applicability of findings among observational studies, compared with RCTs, is a potentially wider range of sources of bias, including in selection, performance, detection of effects, and attrition.

www.rti.org

Background: Rating the Quality of Non-randomized Studies

The quality (internal validity) of each study included in a review needs to be evaluated:

• Well-established criteria and instruments exist for evaluating the quality of RCTs, but not non-randomized (observational) studies

• PIs conducting systematic reviews generally lack access to validated and adaptable instruments for evaluating the quality of observational studies.

• Each new review often develops its own quality rating tool, “reinvents the wheel”, leading to inconsistent standards within and across reviews.

www.rti.org

Project Goals

To create a practical and validated tool for evaluating the quality of non-randomized studies of interventions or exposures that is:

• Reflects a comprehensive theoretical framework: captures all relevant domains

• Broad applicability: can be used "off the shelf" by different PIs • Modifiable: can be adapted to different topic areas • Easy to use and understand: can be used by reviewers with

varying levels of expertise or experience • Validated: users can be confident of their evaluation of study

quality • Advances the methodology in the field• Disseminated widely

www.rti.org

Methods: Phase 1

Item development

• Reviewed the literature on the evaluation of the quality of observational studies

• Collected quality review items used in early tools to evaluate non-RCTs through – Published literature– 90 AHRQ-sponsored EPC reviews

• Categorized all potential items into the 12 quality domains identified in Evaluating non-randomized intervention studies (Deeks et al., 2003)

www.rti.org

Methods: Phase 1 (continued)

Item Bank development

• Selected the best items for measuring each of the included domains

• Modified selected items where necessary to ensure that critical domains were included and to improve readability

• Developed a pre-specified set of responses

• Developed explanatory text to be used by PIs and abstractors to individualize as well as standardize interpretation

www.rti.org

Methods: Phase 2

Technical Expert Panel input– Conceptual framework to ensure that we included all

relevant domains– Face validity

• Cognitive interviews with potential users– Readability– Conceptualization

• Validation– Content/face validity– Inter-rater reliability testing

www.rti.org

Conceptual Underpinnings of the Instrument

Evaluation of quality can rely on either a description of methods or an assessment of validity and precision

• Methods description approach– Follows the reporting structure of many manuscripts– Relies less on judgment than on reporting

• Validity and precision approach– What we really care about– More challenging to evaluate– Greater reliance on judgment

www.rti.org

Domains for quality evaluation approaches

Methods description approach• Background/context• Sample definition and selection• Intervention/exposure• Creation of treatment groups• Follow-up• Specification of outcomes• Analysis: comparability of

groups• Analysis: outcomes• Interpretation

Validity and precision approach • Selection bias• Performance bias• Information bias• Detection bias• Attrition bias• Reporting bias• Precision

www.rti.org

Tool Results

• Comprehensive: bank of 39 questions• Modifiable: includes relevant items appropriate for all

non-randomized study types• Easy to use: instructions for PIs and abstractors to assist

in appropriate interpretation of questions. Example:

What is the level of detail in describing the intervention or exposure? [PI: specify which details need to be stated, e.g., intensity, duration, frequency, route, setting, and timing of intervention/exposure. For case-control studies, consider if the condition, timing, frequency, and setting of symptoms is provided in the case definition]

www.rti.org

Next Steps

• Finalize inter-rater reliability results• Publish findings and disseminate the tool

• Proposed Phase III:– Design specific validation including inter-rater reliability

testing by study type– Reduce the number of questions needed to address

specific domains– Develop a web-based platform for generating design and

topic-specific instruments from the item bank.