validity of utilization

LettersCorrespondance

A burning issue

The new series in CMAJ on the en-vironment1 is long overdue. Withrespect to the issue of medical-waste in-cineration, 2 years ago I set the year2000 as the target date for shuttingdown our hospital incinerator. Giventhe current rate of progress, I am think-ing of re-establishing that date as theyear 3000.

Alban Goddard-HillPhysicianBelleville, Ont.

Reference1. McCally M. Environment and health: an

overview [commentary]. CMAJ 2000;163(5):533-5.

Anticoagulation therapy forpatients with atrial fibrillation

Robert Hart makes several astuteobservations in his recent letter1and in general I agree that warfarintherapy is not well used in atrial fibril-lation; it is overused among low-riskpatients and underused among high-risk patients. Perhaps the problem ismore with the treatment itself thanwith the physician using it. Adjusted-dose warfarin treatment is a complextherapy that requires assiduous andongoing monitoring to achieve goodresults, with a narrow therapeutic win-dow. It ties patients to the medical sys-tem, interferes with travel and compli-cates use of alcohol and of manycommon medications. Although adecade has passed since we learnedthat warfarin is beneficial in atrial fib-rillation, many patients with atrial fib-rillation who are at a high risk forstroke are not receiving adequate pro-phylaxis. With new antithrombinagents on the horizon and more effec-tive antiplatelet agents (alone and incombination) already available, per-haps our efforts should be directed to-ward discovering effective antithrom-botic control for atrial fibrillation that

is safer than warfarin therapy and eas-ier to manage.

Stuart J. ConnollyProfessor of MedicineMcMaster UniversityHamilton, Ont.

Reference1. Hart RG. Anticoagulation therapy for patients

with atrial fibrillation [letter]. CMAJ2000;163(8):956-7.

Validity of utilizationreview tools

We agree with Norman Kalantand colleagues that it is impor-tant to validate the use of utilization re-view tools in Canada,1 but we feel thatthe methodology they used for theirstudy does not reflect the manner inwhich the tools are implemented andcannot adequately support their conclu-sions.

Whereas actual utilization review ac-tivity uses current criteria, the re-searchers chose criteria that are now 4years old. Utilization review at the 2largest Vancouver hospitals has shownthat approximately 10% of inpatientdays meet criteria for subacute care, yetthe researchers failed to use the sub-acute care criteria.

In addition, the sample size was verylimited, both in number and scope (i.e.,75 charts were reviewed for cardiologyonly). Generalization as to the validityof the entire tool is thus suspect.

Finally, implementation in ourhealth region includes a secondary re-view process that improves upon toolvalidity as well as inter-rater reliabilitytests for the reviewers. Kalant and col-leagues did not include a secondary re-view process in their study and theyquestioned its usefulness given the fre-quent divergence of clinical opinionamong individual physicians.1 Howvalid is it to use 3 cardiologists as agold standard?

Although utilization review is not aperfect science, it is one of many im-

portant strategies that we can employto determine how best to improve ourhealth system.

Yoel Robens-ParadiseChairVancouver Richmond Health BoardQuality Steering CommitteeJoan ChocholikUtilization Management CoordinatorQuality, Utilization, Information SupportTeam

Vancouver Hospital & Health SciencesCentre

Sue EmmonsDirectorQuality, Utilization, Information Support Team


Jan FisherManagerQuality ResourcesVancouver Richmond Health BoardSelina PopeDirectorQuality PromotionChildrens & Womens Health CentreShirley ThornDirector of Strategic InitiativesRichmond Health ServicesBarbara TreriseLeaderUtilization Management & EvaluationServices

St. Pauls HospitalSharmen VigouretProvincial Project ManagerBC Cancer AgencyVancouver, BC

Reference1. Kalant N, Berlinguet M, Diodati JG, Dragatakis

L, Marcotte F. How valid are utilization reviewtools in assessing appropriate use of acute carebeds? CMAJ 2000;162(13):1809-13.

The conclusion reached by Nor-man Kalant and colleagues thatutilization review tools have only alow level of validity when comparedwith a panel of experts, which raises se-rious doubts about their usefulness forutilization review1 is not well sup-ported by the data in this very limitedstudy involving 75 patients in a singlediagnostic group.

CMAJ NOV. 14, 2000; 163 (10) 1235

2000 Canadian Medical Association or its licensors

Return to November 14, 2000Table of Contents

http://www.cma.ca/cmaj/vol-163/issue-10/issue-10.htm

The authors have not recognizedthat these tools are valuable for systemplanning as pointers to potential alter-native levels of care. In utilization man-agement they are guidelines, not rules.It would be foolish even to consider us-ing such tools exclusively in the decision-making process about clinicalmanagement. In our studies we have re-peatedly emphasized that the responsi-ble clinician must at all times make thefinal judgement,2,3 but these guidelinesdo help to stimulate regular review ofthe need for hospitalization in the in-terests of quality care and efficiency.The rate of inappropriate hospitaliza-tion may be debated, but it would bedifficult to deny that a significant prob-lem exists that can only be addressed bybetter system planning.

The ISD (Intensity of service, Sever-ity of illness, Discharge screens) guide-lines are developed and regularly re-vised by a more extensive panel processthan that used by the authors but theydo not take into account whether themore appropriate level of care (for ex-ample, outpatient diagnostics or homeintravenous therapy) is actually avail-able in the local community. Physiciansbecome justifiably upset if a label ofinappropriate is applied in the ab-sence of this assessment when the alter-natives simply do not exist, but plannersneed help in creating them.

There is, of course, no perfect toolfor assessing the appropriateness ofclinical services, but it would be ex-tremely unfortunate if the unjustifiableconclusion of this paper discouragedthe use of utilization review toolswithin the proper context.

Charles J. WrightCentre for Clinical Epidemiology &Evaluation


Karen CardiffCentre for Health Service & PolicyResearch

University of British ColumbiaVancouver, BC

References1. Kalant N, Berlinguet M, Diodati JG, Dragatakis

L, Marcotte F. How valid are utilization review

tools in assessing appropriate use of acute carebeds? CMAJ 2000;162(13):1809-13.

2. Wright CJ, Cardiff K, Kilshaw M. Acute med-ical beds: How are they used in British Colum-bia? Vancouver: Centre for Health Services andPolicy Research, University of British Columbia;1997. Report no HPRU 97:7D.

3. Wright CJ, Cardiff K. The utilization of acutecare medical beds in Prince Edward Island. Van-couver: Centre for Health Services and PolicyResearch, University of British Columbia; 1998.Report no HPRU 98:14D.

Ihave 3 comments on the methodol-ogy used by Norman Kalant and as-sociates in their article on utilization re-view tools.1

First, the AEP (AppropriatenessEvaluation Protocol) is an instrumentto measure a hospitals operating effi-ciency with respect to acute patients,not specific clinical appropriateness.Consequently, I do not consider thelack of agreement between this tool andthe judgement of a panel of experts tobe remarkable: the AEP tool measuresprovision levels whereas the panel ex-pressed a clinical opinion. Second, thequality of clinical documentation maydramatically influence the appropriate-ness of services; appropriateness tendsto be underestimated in retrospectivesurveys. Lastly, the panel of expertsseems to have based its judgement on amethodology that was only partially

structured and that does not lend itselfto standardization.

Aldo MariottoHeadHealth Community ServicePordenone, Italy

Reference1. Kalant N, Berlinguet M, Diodati JG, Dragatakis


Norman Kalant and colleaguesconclude that utilization reviewtools are not valid to assess appropri-ateness of setting.1 We argue that thisconclusion is not supported for severalreasons. First, and most importantly,acute care review tools assume thatsubacute care and acute care are sepa-rate, discrete levels. With only minorexceptions, in Canada subacute care isnormally and appropriately deliveredwithin the acute care setting. One re-port referred to by the authors specifi-cally makes this point in terms of thestructure of the Ontario health system.2This oversight alone is likely to ac-count for a significant proportion ofthe mismatch between the review toolsfor acute care (without including suba-

Letters

1238 JAMC 14 NOV. 2000; 163 (10)

We require 6 to 8 weeks notice to ensure uninterrupted service.Please send your current mailing label, new address and theeffective date of change to:

CMA Member Service Centre

1867 Alta Vista Dr.Ottawa ON K1G 3Y6

tel 888 855-2555 or 613 731-8610 x2307

fax 613 [email protected]

Change of address

cute care) and the opinion of a panel ofexperts.

The authors base their conclusionson a simple kappa statistic. However,they have not adjusted for the noninde-pendent nature of the days of stay ofthe 75 patients reviewed. Our work hasdemonstrated a 30% correlation be-tween the appropriateness of 1 day ofstay and appropriateness of the subse-quent day (unpublished data). Nonin-dependence of observations may am-plify disagreement as measured by thekappa statistic.

Finally, the authors claim that noprevious studies have validated thesetools by comparison with implicit re-view by a panel of physicians. In fact 2published studies used physician panelsto demonstrate validity.3,4

Peter DodekPhysician LeaderIntensive Care UnitSt. Pauls HospitalBarbara TreriseLeaderUtilization Management & EvaluationServices

St. Pauls HospitalC. Brian WarrinerVice-President, MedicineProvidence Health CareVancouver, BC



2. Utilization Steering Committee, Joint Policyand Planning Committee. Non-acute hospital-ization project - final report. Toronto: OntarioMinistry of Health; 1997. Reference documentRD6-3.

3. Strumwasser I, Paranjpe NV, Ronis DL, ShareD, Sell LJ. Reliability and validity of utilizationreview criteria. Med Care 1990;28:95-109.

4. Inglis AL, Coast J, Gray SF, Peters TJ, FrankelS. Appropriateness of hospital utilization. MedCare 1995;9:952-7.

It is hardly surprising that standardutilization review tools developed forUS hospitals are not useful in Canada.1They have been developed for a differ-ent context.

Communities decide how they willdeliver care to their constituents. Forexample, urban communities with amature home care program will require

fewer hospital days than rural commu-nities where it is not cost worthy toprovide a full home care program to allwho need it.

One way to increase agreement be-tween clinicians and administrators is tosupport a process in which clinical ex-perts, administrators and communitiesdecide the appropriate setting for vari-ous forms of care. Custom-made utiliza-tion review tools can then be imple-mented to test each day whetherpatients are in the agreed-on and appro-priate setting. These systems can alsoprompt health care providers to movepeople to the most appropriate setting.

The aim of utilization review is toincrease efficiency and value. In thiscase, a measure of efficiency would bedays of care related to change in health.Consequently the most valuable utiliza-tion review tools will be able to linkhealth care activities with health careresults to help health care providers un-derstand which activities are pertinentto a result and which are superfluous.2

David ZitnerDirectorMedical InformaticsDonald FayShared Care InformaticsNeil RitchieOffice of the DeanFaculty of MedicineDalhousie UniversityHalifax, NS



2. Zitner D, Paterson G, Fay D. Methods in healthdecision support systems, methods for identify-ing pertinent and superfluous activity. In: Tan J,editor. Health decision support systems. Gaithers-burg (MD): Aspen Publishers; 1998.

[The authors respond:]

We based our conclusions on ourresults together with those ofpreviously published studies (Table 3)of general medical, surgical and psychi-atric patients.1 Our sample size was 75for admissions and 461 for subsequentdays (because each day is rated inde-pendently); the average kappa scores

were based on much larger samples(e.g., 759 admissions and 3142 days ofhospitalization for the ISD). We there-fore believe that the conclusions arewell grounded.

With any type of project, techniquesthat are current at the outset may be-come nominally outdated before termi-nation. Since the ISD and the MCAPare proprietary, we are unable to deter-mine if there have been substantivechanges in the tool criteria, but a timeseries of published kappa values doesnot show an increase in validity overthe past decade. The comment aboutthe age of the tools implies that currentversions have higher validity than ear-lier ones; is there evidence of this?

We did not use subacute care crite-ria because we were focusing on acutecare; as noted, we stopped case evalua-tion when the patient was moved to adifferent level of care.

We omitted the secondary reviewfor reasons already given, including atheoretical concern that it would proba-bly decrease validity. Yoel Robens-Par-adise and colleagues assert that a sec-ondary review improves tool validity;what is their evidence for that?

The validity of an expert panel can-not be assessed because there is nothingaccepted as more accurate to which itcan be compared.

Charles Wright and Karen Cardiffscomments on utilization managementare undoubtedly correct, but they areirrelevant to an assessment of tools forutilization review. These writers statethat we have not recognized the valueof these tools for system planning; wewould put it differently that they donot recognize that if the tools fail to ac-complish what they have been designedfor, then they are not valuable. If, forexample, a tool misidentifies a signifi-cant number of days as inappropriate, areviewer searching for the reasons forthe inappropriate stay (when in factthere are none) may be led to form er-roneous conclusions about the relativeimportance of the various reasons forsuch days and then to make inappropri-ate changes in the system. Nobodywould trust a new laboratory test withas low a level of accuracy as that exhib-

Correspondance

CMAJ NOV. 14, 2000; 163 (10) 1239

ited by these tools; why have so manyhospitals accepted them without firstvalidating them? We believe that theonus is on those who choose to usethem to show that they do what theyare supposed to do.

Aldo Mariotto claims that the AEPmeasures operating efficiency (unde-fined), not appropriateness. However,one of the developers of the AEP2stated clearly that its purpose is to as-sess appropriateness of hospitalization;furthermore Coast3 commented on itsinability to measure efficiency. Weagree that the quality of the clinicalrecord is a critical factor in applyingthese tools, whether concurrently orretrospectively; any deficiencies wereconstant for the tools and the panel inthis study. As to structuring the panelreview, the panels task was to arrive ata clinical judgement, normally an idio-syncratic process; to structure the re-view process would have defeated itspurpose.

Peter Dodek and colleagues arguethat a major source of disagreement be-tween a review tool (specifically theISD) and the panelists is the fact thatCanadian hospitals generally do nothave separate subacute care units; thepanel may therefore consider a day inacute care as appropriate because thereis no alternative. This was not of con-cern in our study because we have asubacute cardiac unit adjacent to thecoronary care unit. The original manu-script stated that the panel was alsoasked to recommend, for those days notrequiring care at an acute level, a moresuitable level of care; for this purpose, itwas assumed that all levels of care wereavailable. Unfortunately, this sentencewas deleted to meet space limitations.

Dodek and colleagues criticize us forclaiming to be the first to carry out avalidation study of these tools; indeedall the data in Table 3 are taken fromsuch validation studies, including the 2to which they refer (see our references16 and 20).1 We did, however, pointout that the ISD and the MCAP hadnot previously been validated in Cana-dian studies.

Dodek and colleagues raise an inter-esting question concerning the applica-tion of the kappa statistic to the pool ofall days in hospital (other than the ad-mission day): they suggest that the daysof a given patient are not fully indepen-dent of each other (e.g., if day a is inap-propriate, then it is possible that day a +1 will be inappropriate), and that thismay amplify disagreement as measuredby . kappa. However, it seemsequally likely that nonindependencemay amplify agreement and lead to afalse elevation of kappa. To settle thisquestion, we have calculated kappascores separately for each of the hospitaldays 26; in this way, each kappa valueis based on only 1 observation per pa-tient. The values of kappa are as follows:day 2, 0.40 (n = 72); day 3, 0.123 (n =64); day 4, 0.42 (n = 59); day 5, 0.287 (n= 55); day 6, 0.181 (n = 48). Thus, re-moval of any hypothetical dependenceeffects does not raise the kappa as pre-dicted by Dodek and colleagues; weconclude that nonindependence of con-secutive observations is not responsiblefor the low kappa values found in valida-tion studies.

David Zitner and colleagues de-scribe a hypothetical utilization reviewprocess that approaches the ideal butmight be very time consuming. Weagree with Tu4 that more research is

needed to develop a useful utilizationreview tool.Acknowledgements: We are grateful to Dr. IanShrier and Aude Dufresne for valuable discussions ofthe statistical analyses.

Norman KalantDepartment of MedicineSir Mortimer B. Davis Jewish GeneralHospital

Montreal, Que.Marc BerlinguetMinistre de la sant et des servicessociaux du Qubec

Quebec City, Que.Jean G. DiodatiService de cardiologieHpital du Sacr-Coeur de MontralMontreal, Que.Leonidas DragatakisFranois MarcotteDivision of CardiologySir Mortimer B. Davis Jewish GeneralHospital

Montreal, Que.



2. Restuccia JD. The evolution of hospital utiliza-tion review methods in the United States. Int JQual Health Care 1995;7:253-60.

3. Coast J. Appropriateness versus efficiency: theeconomics of utilization review. Health Pol1996;36:69-81.

4. Tu JV. Utilization review: can it be improved?CMAJ 2000;162(13):1824-5.

Letters

1242 JAMC 14 NOV. 2000; 163 (10)

Submitting letters

Letters may be submitted via our Web site or by mail, courier, email([email protected]) or fax. They should be no more than 300 words long and must besigned by all authors. A signed copy of letters submitted by email must be sentsubsequently to CMAJ by fax or regular mail. Letters written in response to anarticle published in CMAJ must be submitted within 2 months of the articlespublication date. CMAJ corresponds only with the authors of accepted letters.Letters are subject to editing and abridgement.

eLetters

We encourage readers to submit letters to the editor via the eLetters service on ourWeb site (www.cma.ca/cmaj). Our aim is to post by the next business daycorrespondence that contributes significantly to the topic under discussion. eLetterswill be appended to the article in question in eCMAJ and will also be consideredfor print publication in CMAJ. Beginning with the Aug. 22, 2000, issue, eLetterscan be submitted by clicking on the mailbox icon at the end of the HTML text ofany eCMAJ article.

validity of utilization

Documents