appropriate use of cardiovascular technology · appropriate use criteria appropriate use of...

Journal of the American College of Cardiology Vol. 61, No. 12, 2013© 2013 by the American College of Cardiology Foundation ISSN 0735-1097/$36.00

APPROPRIATE USE CRITERIA

Appropriate Use of Cardiovascular Technology2013 ACCF Appropriate Use Criteria Methodology UpdateA Report of the American College of Cardiology Foundation Appropriate Use Criteria Task Force

Published by Elsevier Inc. http://dx.doi.org/10.1016/j.jacc.2013.01.025

AppropriateUse CriteriaTask Force

priate Use Criteria Methodohttp://dx.doi.org/10.1016/j.jac

Robert C. Hendel, MD, FACC, FAHA,FASNC

Manesh R. Patel, MD, FACCJoseph M. Allen, MAJames K. Min, MD, FACCLeslee J. Shaw, PHD, FACC, FASNC, FAHA

logy Update. J Am Coll Cardiol 2013;61:1305–17.c.2013.01.025.

American Codepartment at

Pamela S. Douglas, MD, MACC, FAHA, FASEChristopher M. Kramer, MD, FACC, FAHARaymond F. Stainback, MD, FACC, FASESteven R. Bailey, MD, FACC, FSCAI, FAHAJohn U. Doherty, MD, FACC, FAHARalph G. Brindis, MD, MPH, MACC,

Michael J. Wolk, MD, MACC FSCAI, ex officio

Introduction

The past several decades have seen rapid and extensivechanges in the practice of cardiology, especially in theinnovation and utilization practices of imaging, interven-tional, and electrophysiology procedures. Enhanced radio-nuclide imaging techniques, evolution of echocardiography,development of cardiac magnetic resonance (MR), andcoronary computed tomography (CT) angiography tech-niques, as a well as drug-eluting stents and cardiovascularimplantable electronic devices, have revolutionized howpatients are diagnosed and treated. Although these devel-opments have resulted in direct patient benefits includingimproved survival and enhanced quality of life, there hasbeen an accompanying increase in resource utilization andhealthcare costs. Although declines in utilization of manycardiovascular procedures have been observed as of late,during the years preceding 2005, the growth rates were attimes substantial as these technologies were adopted. Theperceived high rate of growth of expenditures related tocardiovascular procedures has precipitated payers to initiateutilization constraints to markedly reduce spending andreimbursement. Various payer initiatives have created anonerous burden leading to costly administrative require-ments, including physician profiling and prior authorization(1). These general programs are also, in part, driven bymarked geographic variability in utilization, which under-score the need for further guidance regarding optimalpatient selection for procedures (2,3). Professional efforts to

better define quality have identified the importance ofmatching procedures and patients (4).

In response to the imperative for improving the utiliza-tion of cardiovascular procedures in an efficient and con-temporary fashion, the American College of CardiologyFoundation (ACCF), along with imaging subspecialty so-cieties and other organizations, developed the first set ofAppropriate Use Criteria (AUC) in 2005, focusing onindications for radionuclide imaging (5). A concurrentpublication defined in some detail the methods involved inthe construction of these criteria (6). During the ensuing 7years, there have been numerous other AUC publications(Figure 1)—including revisions to several of the originalcriteria—that reflect expansion of the AUC concept andadvances within the specific disciplines, as well as method-ological changes. The aim of the current paper is to reviewrecent modifications in the methods for developing AUC,most notably substantial alterations in the nomenclatureemployed for appropriateness categorization (detailed laterin the text).

Although the methods for AUC construction haveevolved, the core process remains rooted in the applicationof the validated, prospectively based modified RAND Ap-propriateness Method (7,8) as was described in the initialmethods article (6). Readers interested to learn more aboutthe RAND method and adaptations developed by ACCFare encouraged to review these papers, as well as the currentpaper. Because the RAND method offered a general ap-proach to constructing criteria, most of the approaches

This document was approved by the American College of Cardiology FoundationBoard of Trustees in January 2013. The American College of Cardiology Foundationrequests this document be cited as follows: Hendel RC, Patel MR, Allen JM, Min JK,Shaw LJ, Wolk MJ, Douglas PS, Kramer CM, Stainback RF, Bailey SR, Doherty JU,Brindis RG. Appropriate Use of Cardiovascular Technology: 2013 ACCF Appro-

Copies: This document is available on the World Wide Web site of the American Collegeof Cardiology (http://www.cardiosource.org). For copies of this document, please contactElsevier Inc. Reprint Department, fax (212) 633-3820, e-mail [email protected].

Permissions: Multiple copies, modification, alteration, enhancement, and/or dis-tribution of this document are not permitted without the express permission of the

llege of Cardiology Foundation. Please contact Elsevier’s [email protected].

http://dx.doi.org/10.1016/j.jacc.2013.01.025

http://www.cardiosource.org

mailto:[email protected]

mailto:[email protected]

emiss

1306 Hendel et al. JACC Vol. 61, No. 12, 2013Appropriate Use Criteria Methodology Update March 26, 2013:1305–17

taken by ACCF in the application of the method areinterpretations required for implementation rather thanchanges to the basic framework of the RAND approach. Assuch, to address the feedback of the clinical cardiologists,other patient-care professionals, payers, and regulators, theACCF AUC Task Force (formerly the AppropriatenessCriteria Working Group) has refined its methods over time.The basic methodology is shown in Figure 2. This paperprovides an update describing the various revisions that haveimproved the rigor and consistency of the AUC develop-ment process. A comparison of the major changes betweenthe original and updated methods is summarized in Table 1.Additionally, this paper describes the modifications in methodsdesigned to facilitate the creation of multimodality AUC, aninitiative aimed to summarize, in 1 document, current recom-mendations regarding multiple procedures. A brief summary ofstudies examining the AUC process and their evaluation andimplementation to date is also provided.

Defining Appropriate Use

In the first series of AUC documents, there was strictadherence to the terms of appropriateness established byRAND methodology, including the title, “appropriateness

Figure 1. Timeline of ACCF AUC Development, Including Current P

ACCF � American College of Cardiology Foundation; ACR � American College of Radcardiography; ED � emergency department; HF � heart failure; IHD � ischemic heacoronary revascularization; RNI � radionuclide imaging; SPECT MPI � single-photon

criteria” (6–8). More recent documents have adopted the

term “Appropriate Use Criteria.” This change reflects themore active function of the AUC in promoting evidence-based, effective use of cardiovascular technologies. Anotherchange since the first publication of AUC has been theexpansion of this initiative from imaging to devices andprocedures including coronary revascularization, diagnosticcatheterization, implanted cardiac defibrillators, and cardiacresynchronization devices. Although imaging remains a criticalfocus of the AUC, the ACCF recognizes that there is a needto address the clinical applications of the appropriate utilizationof all cardiovascular procedures. In response, the AUC TaskForce has expanded the list of potential topic areas.

The definition of appropriate use is largely consistent acrosstechnologies and procedures, and includes consideration ofbenefits and risks; although specific definitions for terms andsurrounding assumptions are modified based on the mostclinically relevant aspects. The basic definition states:

An appropriate diagnostic or therapeutic procedure is one inwhich the expected clinical benefit exceeds the risks of theprocedure by a sufficiently wide margin such that the procedureis generally considered acceptable or reasonable care.

For diagnostic imaging procedures, benefits include incre-mental information, which when combined with clinical

e of Documents in Development

; CT � computed tomography; Dx Cath � diagnostic catheterization; Echo � echo-ase; MR � magnetic resonance; PVD � peripheral vascular disease; Revasc �

ion computed tomography myocardial perfusion imaging.

ipelin

iologyrt dise

judgment augment efficient patient care, weighed against

1307JACC Vol. 61, No. 12, 2013 Hendel et al.March 26, 2013:1305–17 Appropriate Use Criteria Methodology Update

the expected negative consequences (risks include the po-tential hazard of missed diagnoses, radiation, contrast,and/or unnecessary downstream procedures).

For therapeutic procedures such as revascularization orimplanted cardiac defibrillator/cardiac resynchronizationtherapy, the benefits include survival or health outcomes(such as improved symptoms, functional status, and/orquality of life) weighed against the risks of the procedureand subsequent related care.

These definitions may vary slightly but always emphasizewhether a test or procedure is a reasonable care option forthe population of patients described by the indication afterconsideration of benefits and risks.

Terminology Clarifications

The specific terminology used to define appropriate proce-dural utilization has been discussed extensively since thepublication of the first AUC document by ACCF. Theintent of the AUC effort was to focus on patient popula-tions, case mix over time, and quality improvement whileinforming, but not dictating, care for individual patients.The 3 categories were meant to reflect a continuum ofbenefits and risks to various patient populations. However,the categorical nature of AUC inherently placed individualpatient cases into 3 distinct groups that were often viewed asabsolute. Unfortunately, AUC efforts to describe the relativeproportion of patients who might or might not benefit froma procedure over time were obscured by concern over howindividual cases would be adjudicated. Such interpretationoften led to misperceptions by all stakeholders about whena procedure may be considered for a patient. For instance,

Figure 2. Schematic Diagram of Methodology for ACCF Appropriat

Overall, more than 50 professionals are involved in the creation of each AUC documAdapted from Patel et al. (6).

should procedures classified as Uncertain but still used be

considered questionable care choices? Should procedurescategorized as Inappropriate always be avoided or couldthere be mitigating circumstances in an individual patientmaking the procedure a reasonable choice? Should allprocedures that are classified as Appropriate be performedin patients or may some patients go without the procedure?In order to address these issues, the AUC Task Forcesurveyed numerous stakeholders to consider clarifications tothe terminology during the summer of 2012. The feedbacksuggested a change to the terms was necessary, as thecurrent AUC terminology was unable to properly commu-nicate the goals of each stakeholder in a consistent fashion.The result was a recommendation by the AUC Task Forceto modify the terms and clarify the definitions. The updatedterms and definitions more closely reflect their applicationin practice, including an expected distribution between eachAUC category for every population, methods for document-ing exceptions, and proper application to individual pa-tients. These changes reflect the continued commitment bythe AUC Task Force and the College to measuring,benchmarking, and improving the appropriateness of pro-cedures for patients while reducing use in populations whowill rarely benefit. As such, the contemporary, reviseddefinitions for the 3 categories described by AppropriateUse documents herein will be the following:

Median Score 7 to 9: Appropriate Care

An appropriate option for management of patients in thispopulation due to benefits generally outweighing risks;effective option for individual care plans although not always

e Criteria Document Development
e Us
ent.

necessary, depending on physician judgment and patient


specific preferences (i.e., procedure is generally acceptableand is generally reasonable for the indication).

Median Score 4 to 6: May be Appropriate Care

At times, an appropriate option for management of patientsin this population due to variable evidence or agreementregarding the benefits/risks ratio, potential benefit based onpractice experience in the absence of evidence, and/or variabil-ity in the population; effectiveness for individual care must bedetermined by a patient’s physician in consultation with thepatient based on additional clinical variables and judgmentalong with patient preferences (i.e., procedure may be accept-able and may be reasonable for the indication).

Median Score 1 to 3: Rarely Appropriate Care

Rarely an appropriate option for management of patients in

Table 1. Summary Comparison of Differences Between Origina

Original Methods

Name of Document

● Appropriateness Criteria

Appropriateness Topics

● Cardiovascular imaging

Appropriateness Terms

● Appropriate, Uncertain, Inappropriate

● Short definitions

Process

Step 1: Indication Development and Literature Review

Writing Group

● Parent task force served as the writing group and developedindications

External Review of Indications

● Ad hoc review by a few individuals

Step 2: Rating Panel Composition (previously Technical Panel)

Number of Panelists

● Any number

Balance of Rating Panel Expertise

● General balance of different experts; no specific limitations on membersspecific expertise or orientation

● No formal survey of expertise; collection of CVs only

Use of Prior Documents

● Instructed not to consider prior documents

Step 3: Rating Panel Process

Roles of Facilitator, Writing Group, and Methodologist

● Appointed facilitator and selected task force membersprovide writing group and methods expertise

Standardized Rating Package

● No standardized listing of rating package components

Step 4: Rating Tabulation

Understanding Agreement and Disagreement

● BIOMED Concerted Action definition used

this population due to the lack of a clear benefit/risk

advantage; rarely an effective option for individual careplans; exceptions should have documentation of the clinicalreasons for proceeding with this care option (i.e., procedureis not generally acceptable and is not generally reasonablefor the indication).

Although the definition of Appropriate generally hasbeen an effective and widely accepted term, concerns existthat use of the term may stimulate overuse of a procedure inwhom alternative management strategies may exist. The taskforce continues to emphasize that an Appropriate procedure isa reasonable option but may not uniformly be necessary forsuch patients. AUC do not supplant the need to formulatedecisions with individual patients. Necessity reviews have beenundertaken by RAND to determine when procedures rated asappropriate are required. Although important, ACCF AUChas not undertaken such reviews to help define areas of

Updated Methods

Updated Methods

● Appropriate Use Criteria (AUC)

● All cardiovascular technologies and procedures

● Appropriate, May be Appropriate, Rarely Appropriate

● Expanded definitions including population application and how AUCinform individual procedure decisions

● Up to 10 experts appointed by multiple societies develop indications● Task force liaison appointed● Indications written to harmonize with prior AUC documents

● Extensive review by up to 50 experts appointed by multiplesocieties, with subsequent integration of comments

● Odd number required

● Explicit balance of �50% engaged primarily in technologyunder evaluation

● Survey of expertise to evaluate panel balance

● Prior AUC documents explicitly included

● Pre-assigned roles of non-expert facilitator/moderator, writinggroup liaison, and methodology expert from task force

● Formal components of rating package include preamble,instructions, assumptions, definitions, indications,guideline/indication mapping, and evidence tables/references

● For larger panels, interpercentile range adjusted for symmetry used

l and

with

potential underuse, but ACCF and the American Heart


Association (AHA) clinical practice guidelines and especiallyperformance measures are well suited for addressing suchquestions.

All of the AUC publications have emphasized thatpatients in the Uncertain category may be reasonable can-didates for the procedure. The task force continues toemphasize that this category represents a patient populationfor whom individual care must be determined by thephysician working directly with the patient. Althoughadditional research may at times help contribute to futureunderstanding of procedure use in this population, thecurrent state of science and clinical experience indicates thatphysicians and patients may consider the procedure amongcare options for this population. As such, this rating should benot be used as the primary basis for denying coverage andreimbursement. This understanding has been emphasized andincluded in the deliberations of each rating panel. To addressthese issues, as well as to obviate any ambiguity associated withthe term, the task force has determined that future publishedAUC will replace the term Uncertain with May Be Appropriate.

As mentioned in the preceding text, concerns have beenraised indicating substantial misperception related to theterm Inappropriate, even though this term was extracteddirectly from the UCLA/RAND methodology. The intentof the term was related to practice patterns, which allows fora small percentage of Inappropriate use (6). Unfortunately,misunderstanding has been present regarding this categoryas it applies to individual cases and AUC patterns oforganizations and physicians over time. The inherent limi-tation of the AUC are the attempt to categorize individualpatients with 3 to 4 simple characteristics, which serve wellfor a population but may not capture specific individualpatients. For these patients, the AUC have consistentlyemphasized the need to document these differences whileavoiding overuse of the procedures in these patient popula-tions. Therefore, the AUC Task Force has substituted theterm Rarely Appropriate for the previous term Inappropriate.However, physicians should be aware that procedures in thiscategory should have unique individual patient circumstancesthat are documented to justify their use, and be especiallycautious to avoid procedures that, in clinical guidelines, indi-cate potential patient harm if the procedure is performed.

Appropriate Use Criteria Development

Process Revisions

Step 1: Topic Selection, Nomination Process, andWriting Group Composition

Topics are selected by the AUC Task Force after carefulreview of the current variation in utilization, stakeholderneeds, procedure volume and cost, available evidence, andfeasibility. Input is uniformly sought from ACCF councilsand committees, cardiovascular and imaging specialty soci-eties, noncardiologists, and relevant stakeholders such as
health payers, policymakers, and patients. Although in some
instances, AUC development can provide a logistical frame-work for the implementation of evidence synthesized byclinical practice guidelines, AUC are often of particularimportance when gaps in the medical literature, clinicalevidence, or guidance publications are present. In particular,the AUC are aimed at guiding both diagnostic and thera-peutic procedures based upon specific, commonly encoun-tered clinical scenarios, many of which cannot be studied inrandomized trials. The AUC Task Force fosters up-to-dateguidance that encompasses the collective expert opinion of amultidisciplinary group of clinical, payer, policy, and otherstakeholders. In other words, AUC are both evidence-basedand utilize expert consensus. Nominations for the writinggroup, reviewers, and the rating panel are solicited from abroad set of partnering organizations and societies andselected by the task force in consultation with partneringsocieties and the writing group. Relationships with industryand intellectual bias based on clinical and professional expertiseis considered during the nomination and selection process.Further information about relationships with industry can befound on page 14, and a discussion of the composition of thegroups can be found within each of the group descriptions onthe following pages.

The writing group composition has evolved over time butcontinues to include members with significant professionalexpertise and to broadly represent multiple stakeholders. Incontrast to the initial AUC documents’ writing groups thatwere predominantly composed of members from the AUCWorking Group, writing groups are now composed of 6 to10 members from multiple societies and diverse organiza-tions, allowing for broader representation across disciplines.A substantial portion of writing group members remainexperts in the technique under consideration to ensure thatindications are constructed to capture the clinical applica-bility and limitations of the technologies or therapy underconsideration.

Further, an AUC Task Force member is now appointedto each writing group to serve as a liaison to providemethodological and operational oversight. The full AUCTask Force also serves to review and approve relevantclinical indications, review literature summaries, provideguidance on methodological issues, ensure harmonization ofindications, definitions, and assumptions across AUC doc-uments, whenever possible, and to foster finalization of theAUC documents in a time-efficient fashion.

Step 2: Indication Development and Literature Review

Substantial effort has been made to standardize the wording forthe clinical indications, assumptions and definitions as theyspan different documents for the various modalities/procedures.

Clinical IndicationsAs clinical indications or scenarios are developed, consis-tency, clarity, and utility are emphasized to support a
foundation for meaningful evaluation. Whenever possible,

R


indications are grouped under common headings. Thisstructured approach, when applied to cardiac imaging,commonly includes construction of tables for patient diag-nosis and risk assessment, current symptomatology, priortesting, previous revascularization, evaluation for a changein clinical status, and consideration of special clinical cir-cumstances. At times, additional groups of scenarios may beincluded, such as the use of routine surveillance testingeither early or late after a procedure or test. The AUCincreasingly attempt to address such scenarios, recognizingthat the clinical trial evidence base is often quite limitedand, given the cost, may never be achievable in the currenthealthcare environment. AUC seek to establish widespreadconsensus around the timeframes during which such testingis unlikely to yield important clinical information. Althoughinitially it was felt that the AUC should not attempt to beall-inclusive but rather focus on common, real-world situa-tions; external feedback regarding the early AUC docu-ments highlighted gaps in indications or unclear clinicalscenarios. The AUC Task Force responded to this feedback,by significant expansion and revision of the indications in

Figure 3. Example of Hierarchy of Potential Test Ordering Based oisk Assessment of Ischemic Heart Disease

CABG � coronary artery bypass graft surgery; CV � cardiovascular; PCI � percutaneous co

subsequent documents (9). The ensuing applications ofrevised AUC are reflective of this expansion, noting that theadded indications have markedly improved the utility of thedocuments (10).

When possible, AUC now provide a hierarchy of indica-tions to guide use of the AUC in a systematic fashion,tailored for each modality, and assist in applying a particularclinical situation to 1 of the indications, an example ofwhich is depicted in Figure 3. This approach also greatlyfacilitates AUC evaluation and implementation, as it per-mits an ordered and reproducible way to apply AUC.

The assumptions present within the AUC have beenbroadened over time to assist in implementation and areaimed at issues such as the competency to perform proce-dures, inclusion criteria, exclusion criteria including consid-eration of common contraindications, standard protocols,and understanding of common procedure risks. For exam-ple, the details regarding the structure and performance ofthe imaging laboratory, catheterization laboratory, or oper-ating suite are now more explicit, as these are likely to beincluded in evaluation for accreditation. These assumptions

nical Presentation for the Detection and
n Cli
ronary intervention; Pre-op � pre-operative.


ensure that the rating process proceeds with a commonbaseline so that the rating is based upon clinical issues ratherthan variability in competence, test quality, or other issues.Although important, these issues are beyond the scope ofAUC.

DefinitionsCommonly used terms and scenarios, such as “ischemicequivalent,” are used consistently across documents andhave been redefined for improved clarity (10–12). Forexample, the incorporation of electrocardiographic abnor-malities as a clinical finding in the definition of an ischemicequivalent now permit extension of this indication beyondsymptoms. In addition, assessment of risk in asymptomaticindividuals has been modified to reflect the concept ofglobal risk, which may be based on any of the majorliterature-based risk scores, including the Adult TreatmentPanel III, modified Framingham risk, or the Reynolds’sscore, to incorporate the most recent data and include itemssuch as family history in the evaluation (10–12). In fact, theAUC has helped to highlight the need for better definitionsand methods around risk stratification, pre-test probability,and findings from noninvasive tests.

To ensure consistency across AUC as well as otherguidance documents, the task force relies heavily on ACCF/AHA Clinical Practice Guidelines for many of its defini-tions and assumptions. Additionally, the task force, has hada continued dialog with the Clinical Practice GuidelinesTask Force to ensure harmonization.

Guideline Mapping and Evidence ReviewOnce the clinical indications are developed, a table mappingthe indications to recommendations of other guidancedocuments, in particular ACCF/AHA practice guidelines,is constructed to preserve and encourage agreement acrossvarious clinical policy documents as related to the specificAUC technology being evaluated. Practice guidelines oftencontain a comprehensive summary of the available literatureand serve as the primary source of evidence review for theAUC process. As these guidelines are updated, the writinggroups are instructed to update the assumptions, indica-tions, and definitions accordingly. New guidelines, such asthe pre-operative guidelines (13), are incorporated into theAUC as they are written and revised, often resulting inchanges in the wording of clinical indications. For evidencenot covered by the Guidelines, the writing group assemblesthe most up-to-date scientific evidence as it may relate tothe topic and specific clinical indications.

Rating Panel MaterialsAfter these various components have been drafted, a stan-dardized literature review and ratings package is prepared byeach writing group following a pre-set template. Thetemplate includes a preamble, instructions on the ratingprocess, definitions, assumptions, abbreviations, indicationtables, and evidence/guideline mapping tables. These items
are then reviewed and approved by the parent AUC Task
Force. The task force and writing committees have contin-ually expanded and clarified these materials, providingsupportive documentation sufficient for reviewers to providefeedback and the members of the rating panel to render ascore on each clinical scenario that is reflective of the currentevidence base.

Step 3: External Review of Clinical Indications

Subsequent review of the indications by the AUC TaskForce and external peer reviewers is aimed at furtherimprovement of the definitions of the clinical scenarios.This was previously done by an ad hoc group of experts, butthis process has now been formalized to include a detailedexternal review by representatives from key participatingmedical specialty societies, as well as those representinghealth services research and those with familiarity withclinical practice guidelines and other key stakeholder’spolicies. This often involves more than 40 individuals in thereview of the draft set of indications. This process is crucialto indication development, because revisions are not allowedfollowing final rating panel voting, as this would violate thebasic tenets of the modified Delphi methodology.

Step 4: Panel Rating

Rating Panel Composition

The AUC Task Force has always attempted to maintain abalance on the rating panels (previously referred to as thetechnical panel) of specialists using the technology andother professionals who are referrers, general cardiologists,outcome specialists, and/or generalists who care for germanepatient populations, as well as the payer community. Over-all, a diverse panel composition ensures the production ofbalanced, equitable, and reproducible ratings (6–8). Spe-cialists whose key area of interest is the primary focus of theAUC are a minority of rating panel members. A survey ofpractice experience is sent to all rating panel membersbefore formalizing their appointment to the panel. Thisinformation is used to ensure that the task force and writinggroup, who determine the composition of the rating panel,are able to consider an accurate description of the individ-uals’ expertise, interests, and relationships before selectionto serve on the panel. This approach has been recentlychallenged by some who believe that the rating panel shouldbe composed either exclusively or by a majority of specialistswithin the field under evaluation, stating the view that suchsubspecialists have a unique and greater understanding ofthe field (14). However, the AUC Task Force believes thatAUC should be as evidence-based as possible and that allmembers of the rating panel should be able to determine thebalance of clinical benefits and risks of a particular proce-dure or technology based on practice experience and rele-vant literature, especially when provided with evidencetables and practice guidelines. In addition, such nonexpertindividuals often represent the community of practitioners
ordering the test or procedure and provide an important


perspective. As such, the AUC Task Force believes that theuse of a majority of “nonspecialists” within a rating panelpermits a diversity of perspectives and enhances externalcredibility.

After using an even number of panel members on the firstAUC, which required rounding to determine the finalmedian score and resulting AUC category, the task forcedecided that all rating panels must be composed of an oddnumber of individuals so that the final median score reflectsthe whole number score of an actual rating panel member.

Initial Panel Ratings and MeetingThe first round of indication ratings are performed inde-pendently by each rater. Following this, participation in aface-to-face meeting is required. In addition to the panelmembers, the AUC Task Force has standardized specificroles for several other individuals during this meeting. Amoderator who has not been involved in indication con-struction establishes the goals, procedural rules, and facili-tates the meeting. This individual typically does not partic-ipate in rating the specific technology or procedure underreview and does not practice the procedure being reviewed.The moderator may serve as an alternate panel member ifunforeseen circumstances prevent a panelist from complet-ing the process. During the face-to-face meeting, thewriting group liaison member is also present to answerquestions specific to the indications and to assist in anymodification or clarification of indications recommended bythe rating panel. Lastly, a member of the AUC Task Forceis present specifically to address methodological issues asthey pertain to the AUC.

During the rating panel meeting, a standardized presen-tation is given providing an overview of the AUC develop-ment process and a review of the assumptions and defini-tions along with an outline of the indication tables and keyclinical parameters used in the document. The panelists areprovided with their own votes and a tabulation of all votespresented in an anonymous fashion. All indications arediscussed using a “round robin” discussion style to ensure allpanel members have the opportunity to lead and participatein the discussion. Particular attention is paid to indicationswith widely divergent ratings to ensure that there is auniform understanding of the clinical scenario. Followingthe face-to-face meeting, the panelists then independentlyrescore all indications. In a very few cases in which there isa wide dispersion of scores and where further indicationrewording would better align the indication with evidenceand/or practice experience, a third rating of the specificindication(s) in question may be undertaken. The medianscore for each clinical scenario becomes the final indicationrating.

Final Rating TabulationThe final scores are reported in discrete categories (Appro-priate, May Be Appropriate, and Rarely Appropriate—aswell as with the numerical median rating, with anonymized
individual scores available in an online appendix).
In addition to the final indication score, all indications areassessed to measure the level of agreement among the panel.When the number of rating panelists is less than or equal to15 members, the definitions for agreement from BIOMEDConcerted Action on Appropriateness are used, as previ-ously described (6,8). For panels with more than 16 mem-bers, this definition is applied first, but a second statisticalmethod described in the RAND Appropriateness Method(7) is also applied. Both measures examine dispersion in theratings and identify whether most ratings are near the finalmedian (agreement) or clustered at opposite sides of therating scale (disagreement). By definition, indications la-beled as having disagreement would automatically becomeMay Be Appropriate. However, to date, no indication hasbeen qualified under this designation that was not alreadyuncertain.

Multimodality Appropriate Use Criteria

The original methods paper (6) discussed the expectationthat an evaluation of the efficient use of technology amongalternative technologies or procedures would have to beconsidered in the future across similar clinical indications. Itwas recognized early on that this would likely be necessaryfor noninvasive cardiovascular testing and other procedures,such as coronary angiography and revascularization, andother interventional therapeutic procedures.

Coronary Revascularization

The AUC Task Force made a first attempt at such anevaluation in the Appropriateness Criteria for CoronaryRevascularization (15). The writing group first discussed thevariables essential for the consideration of any form ofcoronary revascularization. This led to the framework forthe clinical indications categorized by several domains:acuity, prior bypass surgery status, presenting symptoms,ischemia severity on noninvasive testing or fractional flowreserve, and coronary anatomy, as well as adequacy ofmedical therapy. The rating panel was then asked to rateeach indication for the appropriateness of coronary revas-cularization, independent of the mode (either percutaneouscoronary intervention or coronary artery bypass graft sur-gery). In a select set of clinical indications with a highburden of ischemia and/or coronary artery disease, eachprocedure (percutaneous coronary intervention and coro-nary artery bypass graft surgery) was then rated separately. Aseparate table presented the relative appropriateness of eachprocedure for this limited set of indications.

Diagnostic Testing and Cardiovascular Imaging

A document in progress will focus on multiple tests used toevaluate patients presenting with signs and symptoms sug-gestive of coronary artery disease or follow-up of knowncoronary disease. This AUC will continue the ongoingprocess of establishing similar assumptions, definitions,
evidence synthesis, and clinical indications for several tests


that could be used for patient diagnosis and management ofa given clinical indication/scenario.

This multimodality AUC will evaluate clinical scenariosthat are similar and often identical to those published inprior AUC documents. However, it will also include testingtechniques not previously covered, such as: exercise tread-mill testing, so as to include the entire spectrum of cardiactesting for ischemic heart disease. One key aspect of thiswork is that only the category of appropriate use will bereported rather than a numerical value. The 1 to 9 valuesused in the AUC development process are used as a guide tohelp rating panels put procedures into the 3 categories. Therating panel is asked to provide a numerical value corre-sponding with their final desired category for the procedure.The wider range of numerical values provides individualraters with the ability to grade the procedure on a contin-uum, thus allowing raters with slightly different viewpointsto dialog but also allow the final ratings to converge into asingle message. Given the potential minor variability acrossa rating panel in their final scores within a category, thenumerical scores are less reliable. All raters understand it isthe final categories that describe more generally how oftenthe procedure will be considered. Therefore, numericalvalues should not be used to apply gradations and compar-isons between procedures scored in the same category.Central to the rating in this document will be an evaluationof whether any test is needed and if true differences inappropriate use of different tests are present for specificpatient populations. Although 1 goal of this documentcould have been to determine the ranking of one modalityversus the others, there is limited comparative evidence forspecific imaging modalities. As such, comparative rankingof tests within an appropriateness category is not the focusof this document beyond known limitations of specific testsfor certain populations that may result in different appro-priate use categorizations. In most cases, it is anticipatedthat the clinician may have his/her choice of several Appro-priate procedures or that several will be rated as May BeAppropriate or Rarely Appropriate. Beyond the ratings inthis document, clinicians may selectively identify proce-dures/scenarios where other secondary considerations mayapply for specific patients, including safety, cost, localexpertise, availability, patient preference, and so on.

Relationship With Clinical PracticeGuidelines and Other Clinical Policy

Consistency of the AUC with clinical practice guidelines iscritical to avoid confusion on the part of healthcare practi-tioners and payers. As mentioned previously, before therating of the indications, the AUC Task Force and eachwriting group emphasize the need for harmonization amongrecommendations for tests and procedures, which is enabled
through the construction of guideline/indication mapping
tables, references, and careful review during the develop-ment process.

Within the ACCF, the Science and Clinical PolicyCommittee is charged with ensuring communication andcollaboration among key documents and standards develop-ment groups, such as the task forces and committees forexpert consensus documents, clinical practice guidelines,AUC, data standards, and performance measures. An opendialog among members of these groups is necessary toensure that the ACCF maintains the strengths of eachdocument type while promoting collaboration to ensureconsistency and complementary work products. A paradigmof how these documents interrelate has been proposed (16).

Relationships With Industry andOther Entities

Although AUC are not prescriptive, they do define theappropriateness of technology. The ACCF and the AUCTask Force have focused considerable attention on avoidingreal or perceived relationships with industry and otherentities (RWI) that might impact the rating of a test/procedure for a specific application. In addition to fulldisclosure of all RWI by all individuals involved with thedocument, AUC documents are governed by a policymandating that the writing group chair and fewer than 50%of the rating panel have relevant relationships with industry.The writing group itself is not required to have fewer than50% without RWI, as it does not participate in the rating ofthe indications (17).

The task force determined a modification of what consti-tutes relevant RWI was necessary for the writing group becauseof the unique role of AUC and the multiple groups ofindividuals that are involved in the document developmentprocess. In addition, the rating panel is balanced for manyfactors, including clinical and professional bias, as has alreadybeen described in the preceding text, further strengthening therange of viewpoints reflected in the AUC.

Implementation and Evaluation

Publication of the AUC is not the final step towards optimalutilization of diagnostic and therapeutic techniques. Asdemonstrated in Figure 2, the process of AUC constructionitself is iterative, building on published evidence, expertconsensus, past experience, feedback, and new medicalliterature and guidelines. Additionally, information gar-nered from the measurement of appropriateness of actualuse in the community provides guidance in the constructionof new or revised AUC (9,18).

To ensure optimal utilization of diagnostic and therapeu-tic techniques, measurement of appropriate use in “real-world” clinical practice based on AUC is necessary to assesspractice performance, to provide direction for educational
and continuous quality initiatives efforts and to provide


valuable real-world data to inform future AUC efforts. Agrowing number of publications have focused on the eval-uation of appropriateness of use of technology, highlightingpotential targets for education and utilization improvement(19,20).

Although most of these studies have involved retrospec-tive chart review, it is clear that the evaluation of appropriateuse is possible and more actionable when AUC are incor-porated as a decision support tool in the evolution ofelectronic health records and web-based programs. Severalstudies also are now available regarding prospective appli-cation of appropriate use as well whether their utilizationpromotes improved patient-centered outcomes and/or effi-ciency within the diagnostic work-up (21–24).

In this light, a hierarchical structure is generally needed soas to clearly and reproducibly assign each case to a specificAUC scenario. To this end, AUC writing groups are nowbeing encouraged to develop standardized ordering sheetsand design patient care algorithms to summarize the indi-cations (10–12). The uniform and consistent documenta-tion of AUC in clinical practice will greatly assist in thevalidation of AUC as well as in the potential development ofquality metrics for laboratory or physician standards ofappropriate use. To further support this important direc-tion, the ACCF/AHA recently developed a methodologyfor the creation of a new form of quality metric, termed“appropriate use metrics” that is based heavily on thedevelopment of performance measures (25).

Utilization management companies and private healthplans have often cited the AUC as justification for theircoverage and review policies. However, many of the pro-grams are applied to individual cases without considerationof patterns of care over time for a physician/practice orfeedback regarding performance. As such, these programsviolate the spirit of AUC that are designed to inform caredecisions but are best measured over a patient populationrather than rigidly adjudicating care for individual patients.

Clinicians have at times suggested that AUC do notsufficiently reflect the realities of clinical practice and sug-gested that, as a result, the AUC are not valid. The taskforce encourages informal feedback and more formal studiesto help improve the criteria while supporting clinical judg-ment needed for individual patient care. Decision makingshould be guided, but not dictated, by AUC. Some proce-dures rated as Rarely Appropriate should still occur. Otherprocedures rated as Appropriate may be reasonable to forgo.Importantly, AUC should be used to assess the overallpatient mix undergoing procedures and help physicianstabulate the proportion of patients under their care who fallinto each AUC category and clinical indication, and bench-mark it to others’ data. By using the AUC to betterunderstand practice patterns, physicians can engage ineducational discussions with peers and patients about theimportant clinical variables and patient-specific characteris-tics that underpin current procedure use. This information
can be used to adapt to changes in patient mix over time that
can help to optimize the selection of patients for a givenprocedure based on the benefit and risk thresholds of eachphysician and patient.

Conclusions

The refinements to AUC methodology, presented in thispaper, reflect the responsiveness of the ACCF and its AUCTask Force to queries from the clinician and payer commu-nity for a fair, evidence-based, and practical means to guideprocedural utilization. Current revisions to the AUC meth-odology have strengthened the clinical relevance of thesedocuments that are carefully constructed yet continue toundergo evolution to meet the needs of contemporarypractice patterns and the developing evidence base withincardiovascular medicine. The focus of AUC is to encourageoptimal patient care by the professional stewardship oftechnology utilization within cardiovascular medicine. Theeffort aims to join with all cardiovascular practitioners andstakeholders in providing the optimal clinical decisionmaking to foster high quality of cardiovascular care for theirpatients, and to work toward patterns of care that promoteappropriate utilization and minimize use that lacks sufficientvalue whenever possible. These documents have been wel-comed by many in the cardiovascular community, includingphysicians, patients, and policymakers, and have been in-corporated into processes of clinical care, including educa-tion, accreditation, and quality improvement programs.Additionally, the AUC now seem to be having an impact onperformance of tests in certain patient populations, with thegoal of substantial reduction in waste due to unnecessarytests and procedures. The result is that patients may be morelikely to benefit from cardiovascular technology using pro-cedures selected for having the potential to favorably impactpatient outcomes; however, there are very limited data tosupport such a claim at this time. Just as pharmacologicalagents are increasingly targeted toward specific biologicalactions and patient genetic markers that are meaningful totreatment to produce greater benefit at lower risk, tests andprocedures are more effective when matched to patients whohave specific clinical markers suggesting higher benefit torisk. Nevertheless, the AUC intent is as a guiding docu-ment; the final decision to proceed with testing or aprocedure remains at the bedside where patient–physicianinteraction simply cannot be universally policy-based andmust be done in the context of a discussion about treatmentand patient goals, which is never a black or white decision.

The refinements to AUC methodology, presented in thispaper, reflect the AUC Task Force’s commitment to adaptand respond to the ever-evolving needs of cardiovascularpractice. Over time, the task force continues to focus onreflecting the evolution of contemporary practice patternsand accruing scientific evidence. The AUC Task Forceremains steadfast in its aim to ensure a patient-centered, profes-
sional stewardship of technology application within cardiovascular


medicine. To this end, the process joins with all potential stake-holders—including patients, cardiovascular specialists, primarycare physicians, and other specialists, and regulatory and payerbodies—in support of optimal clinical decision making andensuring high-quality, cardiovascular care.

Staff

American College of Cardiology FoundationWilliam A. Zoghbi, MD, FACC, PresidentThomas E. Arend, Jr, Esq, CAE, Interim Chief Staff

OfficerWilliam J. Oetgen, MD, MBA, FACC, Senior Vice

President, Science and QualityJoseph M. Allen, MA, Director, TRIP (Translating

Research Into Practice)Z. Jenissa Haidari, MPH, Senior Research Specialist,

Appropriate Use CriteriaErin A. Barrett, MPS, Senior Specialist, Science and

Clinical Policy

Appendix A. Appropriate Use ofCardiovascular Technology: 2013 ACCFAppropriate Use Criteria MethodologyUpdate Writing Group

Robert C. Hendel, MD, FACC, FAHA, FASNC—Chair,2013 ACCF Appropriate USe Criteria Methodology Up-date; Director of Cardiac Imaging and Outpatient Services,Division of Cardiology, Miami University School of Med-icine, Miami, FL

Manesh R. Patel, MD, FACC—Assistant Professor ofMedicine, Division of Cardiology, Duke University Medi-cal Center, Durham, NC

Joseph M. Allen, MA—Director, TRIP (TranslatingResearch Into Practice), American College of Cardiology
Foundation, Washington, DC
James K. Min, MD, FACC—Director of Cardiac Imag-ing Research and Co-Director of Cardiac Imaging, Cedars-Sinai Heart Institute, Los Angeles, CA

Leslee Shaw, PhD, FACC, FASNC—Professor ofMedicine, Emory University School of Medicine,Atlanta, GA

Michael J. Wolk, MD, MACC—Chair, Task Force,Past President, American College of Cardiology Founda-tion and Clinical Professor of Medicine, Weill-CornellMedical School, New York, NY

Pamela S. Douglas, MD, MACC, FAHA, FASE—Past President, American College of Cardiology Foun-dation; Past President American Society of Echocardi-ography; and Ursula Geller Professor of Research inCardiovascular Diseases, Duke University Medical Cen-ter, Durham, NC

Christopher M. Kramer, MD, FACC, FAHA—Professor of Medicine and Radiology, Director, Cardiovas-cular Imaging Center, University of Virginia Health Sys-tem, Charlottesville, VA

Raymond F. Stainback, MD, FACC, FASE—MedicalDirector of Noninvasive Cardiac Imaging, Texas HeartInstitute at St. Luke’s Episcopal Hospital; Clinical Associ-ate Professor of Medicine, Baylor College of Medicine,Houston, TX

Steven R. Bailey, MD, FACC, FSCAI, FAHA—Chair,Division of Cardiology, Professor of Medicine and Radiol-ogy, Janey Briscoe Distinguished Chair, University of TexasHealth Sciences Center, San Antonio, TX

John U. Doherty, MD, FACC—Professor of Medicine,Jefferson Medical College of Thomas Jefferson University,Philadelphia, PA

Ralph G. Brindis, MD, MPH, MACC, FSCAI—Clinical Professor of Medicine, Department of Medicineand the Philip R. Lee Institute for Health Policy Studies,
University of California, San Francisco, CA


APPENDIX B. APPROPRIATE USE OF CARDIOVASCULAR TECHNOLOGY: 2013 ACCF APPROPRIATE USE CRITERIA
METHODOLOGY UPDATE—RELATIONSHIPS WITH INDUSTRY AND OTHER ENTITIES (RELEVANT)
Participant ConsultantSpeaker’s

Bureau

Ownership/Partnership/

Principal Research

Institutional,Organizational,

or OtherFinancial Benefit

ExpertWitness

Appropriate Use Criteria Task Force

Michael J. Wolk None None None None None None

Steven R. Bailey None None None None None None

John U. Doherty None None None None None None

Pamela S. Douglas None None None None None None

Robert C. Hendel None None None None None None

Christopher M. Kramer None None None None None None

James K. Min None None None None None None

Manesh R. Patel None None None None None None

Leslee Shaw None None None None None None

Raymond F. Stainback None None None None None None

Joseph M. Allen None None None None None None

This table represents the relevant relationships with industry and other entities that were disclosed by participants at the time of participation. It does not necessarily reflect relationships with industryat the time of publication. A person is deemed to have a significant interest in a business if the interest represents ownership of 5% or more of the voting stock or share of the business entity, or
ownership of $10,000 or more of the fair market value of the business entity; or if funds received by the person from the business entity exceed 5% of the person’s gross income for the previous year.A relationship is considered to be modest if it is less than significant under the preceding definition. Relationships in this table are modest unless otherwise noted.
1

1

1

1

1

1

REFERENCES

1. Hendel RC. Utilization management of cardiovascular imaging pre-certification and appropriateness. J Am Coll Cardiol Img 2008;1:241–8.

2. O’Connor GT, Quinton HB, Traven ND, et al. Geographic variationin the treatment of acute myocardial infarction: the CooperativeCardiovascular Project. JAMA 1999;281:627–33.

3. Dartmouth Medical School Center for the Evaluation of ClinicalSciences, Maine Medical Center. The Dartmouth Atlas of Cardiovas-cular Health Care. Chicago, IL: AHA Press, 1999.

4. Douglas P, Iskandrian AE, Krumholz HM, et al. Achieving quality incardiovascular imaging: proceedings from the American College ofCardiology-Duke University Medical Center Think Tank on Qualityin Cardiovascular Imaging. J Am Coll Cardiol 2006;48:2141–51.

5. Brindis RG, Douglas PS, Hendel RC, et al. ACCF/ASNC appropri-ateness criteria for single-photon emission computed tomographymyocardial perfusion imaging (SPECT MPI): a report of the Amer-ican College of Cardiology Foundation Quality Strategic DirectionsCommittee Appropriateness Criteria Working Group and the Amer-ican Society of Nuclear Cardiology. J Am Coll Cardiol 2005;46:1587– 605.

6. Patel MR, Spertus JA, Brindis RG, et al. ACCF proposed method forevaluating the appropriateness of cardiovascular imaging. J Am CollCardiol. 2005;46:1606–13.

7. Brook RH, Chassin MR, Fink A, et al. A method for the detailedassessment of the appropriateness of medical technologies. Int JTechnol Assess Health Care 1986;2:53–63.

8. Fitch K. The RAND/UCLA Appropriateness Method User’s Man-ual. Santa Monica, CA: Rand, 2001.

9. Ward RP, Al-Mallah MH, Grossman GB, et al. American Society ofNuclear Cardiology review of the ACCF/ASNC appropriatenesscriteria for single-photon emission computed tomography myocardialperfusion imaging (SPECT MPI). J Nucl Cardiol 2007;14:e26–38.

10. Hendel RC, Berman DS, Di Carli MF, et al. ACCF/ASNC/ACR/AHA/ASE/SCCT/SCMR/SNM 2009 appropriate use criteria forcardiac radionuclide imaging: a report of the American College ofCardiology Foundation Appropriate Use Criteria Task Force, theAmerican Society of Nuclear Cardiology, the American College ofRadiology, the American Heart Association, the American Society ofEchocardiography, the Society of Cardiovascular Computed Tomog-raphy, the Society for Cardiovascular Magnetic Resonance, and theSociety of Nuclear Medicine. J Am Coll Cardiol 2009;53:2201–29.

11. Douglas PS, Garcia MJ, Haines DE, et al. ACCF/ASE/AHA/
ASNC/HFSA/HRS/SCAI/SCCM/SCCT/SCMR 2011 appropriate
use criteria for echocardiography: a report of the American College ofCardiology Foundation Appropriate Use Criteria Task Force,American Society of Echocardiography, American Heart Association,American Society of Nuclear Cardiology, Heart Failure Society ofAmerica, Heart Rhythm Society, Society for CardiovascularAngiography and Interventions, Society of Critical Care Medicine,Society of Cardiovascular Computed Tomography, and Society forCardiovascular Magnetic Resonance. J Am Coll Cardiol 2011;57:1126–66.

2. Taylor AJ, Cerqueira M, Hodgson JM, et al. ACCF/SCCT/ACR/AHA/ASE/ASNC/NASCI/SCAI/SCMR 2010 appropriate use cri-teria for cardiac computed tomography: a report of the AmericanCollege of Cardiology Foundation Appropriate Use Criteria TaskForce, the Society of Cardiovascular Computed Tomography, theAmerican College of Radiology, the American Heart Association, theAmerican Society of Echocardiography, the American Society ofNuclear Cardiology, the North American Society for CardiovascularImaging, the Society for Cardiovascular Angiography and Interven-tions, and the Society or Cardiovascular Magnetic Resonance. J AmColl Cardiol 2007;56:1864–94.

3. Fleisher LA, Beckman JA, Brown KA, et al. 2009 ACCF/AHAfocused update on perioperative beta blockade incorporated into theACC/AHA 2007 guidelines on perioperative cardiovascular evaluationand care for noncardiac surgery. J Am Coll Cardiol 2009;54:e13–118.

4. Marso SP, Teirstein PS, Kereiakes DJ, et al. Percutaneous coronaryintervention use in the United States: defining measures of appropri-ateness. J Am Coll Cardiol Intv 2012;5:229–35.

5. Patel MR, Dehmer GJ, Hirshfeld JW, et al. ACCF/SCAI/STS/AATS/AHA/ASNC 2009 appropriateness criteria for coronary revascularization:a report by the American College of Cardiology Foundation Appropri-ateness Criteria Task Force, Society for Cardiovascular Angiography andInterventions, Society of Thoracic Surgeons, American Association forThoracic Surgery, American Heart Association, and the American Soci-ety of Nuclear Cardiology. J Am Coll Cardiol 2009;53:530–53.

6. Antman EM, Peterson ED. Tools for guiding clinical practice fromthe American Heart Association and the American College of Cardi-ology: what are they and how should clinicians use them? Circulation.2009;119:1180–5.

7. ACCF Relationship With Industry and Other Entities: Policies andProcedures for the Development of Guidelines and Other ClinicalDocuments. 2010. Available at: http://www.cardiosource.org/Science-And-Quality/Practice-Guidelines-and-Quality-Standards/Relationships-
With-Industry-Policy.aspx. Accessed December 17, 2012.
http://www.cardiosource.org/Science-And-Quality/Practice-Guidelines-and-Quality-Standards/Relationships-With-Industry-Policy.aspx



2

2

2


18. Ward RP. Appropriateness criteria for echocardiography: an impor-tant step toward improving quality. J Am Soc Echocardiogr 2009;22:60–2.

19. Hendel RC, Cerqueira M, Douglas PS, et al. A multicenter assess-ment of the use of single-photon emission computed tomographymyocardial perfusion imaging with appropriateness criteria. J Am CollCardiol 2010;55:156–62.

20. Ward RP, Krauss D, Mansour IN, et al. Comparison of the clinicalapplication of the American College of Cardiology Foundation/American Society of Echocardiography Appropriateness Criteria foroutpatient transthoracic echocardiography in academic and commu-nity practice settings. J Am Soc Echocardiogr 2009;22:1375–81.

21. Allen JM. A National Initiative to Improve Utilization of CardiacImaging: The FOCUS Learning Community and Performance Improve-ment Module. Circ Cardiovasc Qual Outcome 2011;4 Suppl:231.

22. Chinnaiyan KM, Peyser P, Goraya T, et al. Impact of a continuous
quality improvement initiative on appropriate use of coronary com-puted tomography angiography. Results from a multicenter, statewide
registry, the Advanced Cardiovascular Imaging Consortium. J AmColl Cardiol 2012;60:1185–91.

3. Barnes E. ACC-Based Software Boosts Appropriateness of HeartScans. 2012. Available at: http://www.auntminnie.com//index.aspx?sec�ser&sub�def&pag�dis&ItemID�100015. Accessed No-vember 27, 2012.

4. Druz RS, Phillips LM, Sharifova G. Clinical evaluation of theappropriateness use criteria for single-photon emission-computedtomography: differences by patient population, physician specialty, andpatient outcomes. ISRN Cardiol 2011;2011:798318.

5. Bonow RO, Douglas PS, Buxton AE, et al. ACCF/AHA methodol-ogy for the development of quality measures for cardiovascular tech-nology: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures.J Am Coll Cardiol 2011;58:1517–38.

Key Words: ACCF Appropriate Use Criteria y Delphi process ymethodology y multimodality y rating panel.

http://www.auntminnie.com//index.aspx?sec=ser%26sub=def%26pag=dis%26ItemID=100015

http://www.auntminnie.com//index.aspx?sec=ser%26sub=def%26pag=dis%26ItemID=100015