Journal of Critical Care (2011) 26, 138–143

Validation of the Intensive Care Delirium ScreeningChecklist in nonintubated intensive care unit patients in aresource-poor medical intensive care setting in South IndiaChristina George DPM, MDa,⁎, Jayakrishnan Sukumaran Nair MDa,Johann Alex Ebenezer MBBS a, Alan Gangadharan MBBSb, Anna ChristuDas MBBSb,Libu Kanakamma Gnanaseelan MDc, K.S. Jacob MD, PhDd

aDepartment of Psychiatry, Dr Somervell Memorial CSI Medical College and Hospital, Kerala 695504, IndiabDr Somervell Memorial CSI Medical College and Hospital, Kerala 695504, IndiacDepartment of Community Medicine, Dr Somervell Memorial CSI Medical College and Hospital, Kerala 695504, IndiadDepartment of Psychiatry, Christian Medical College, Vellore 632002, India

0d

Keywords:Delirium;Sensitivity;Specificity;Validity;Reliability

AbstractObjective: Delirium is a common, difficult-to-diagnose clinical condition in critical care units. The lackof recognition of delirium often results in increased morbidity and mortality. The study aimed todetermine the validity and reliability of the Intensive Care Delirium Screening Checklist (ICDSC) in aresource-poor medical intensive care setting in South India.Materials and methods: Fifty-three patients admitted into the medical intensive care unit of a teachinghospital who were neither mute nor intubated were recruited for the study. Trained residents administeredthe ICDSC to screen for delirium. A consultant psychiatrist used the International Classification ofDiseases, 10th Revision diagnostic criteria for research to determine the presence of delirium.Results: The optimal threshold for screening, as ICDSC total score of 3 or more, was obtained by using areceiver operating characteristic curve. Although a sensitivity and specificity of 75% and 74%,respectively, were obtained at the original cutoff of 4, a sensitivity of 90% and specificity of 61.54% wereachieved with a cutoff of 3. In a subsample of 21 patients, interrater reliability was evaluated and found tobe 0.947 (95% confidence interval, 0.870-0.979). The ICDSC had good internal consistency, withCronbach α of .754 and Guttman split-half coefficient of 0.71. Factor analysis revealed a 2-factorstructure, namely, altered sensorium/psychopathology and sleep-wake cycle problems.Conclusions: Our findings indicate that in nonintubated intensive care unit patients, the ICDSC can beused to screen for delirium but should not be used as a diagnostic instrument in this patient population andthat residents can be trained in the use of the instrument in resource-poor critical care settings. Using adifferent threshold for positivity of 3 rather than 4 appeared to offer improved screening characteristics inthis resource-poor critical care setting.© 2011 Elsevier Inc. All rights reserved.

⁎ Corresponding author. AI, Aiswarya Nagar, Kesavadasapuram, Trivandrum, Kerala, India. Tel.: +9 1934 913 6502; fax: +9 147 122 50239E-mail address: mukkath@yahoo.com (C. George).

883-9441/$ – see front matter © 2011 Elsevier Inc. All rights reserved.oi:10.1016/j.jcrc.2010.11.002

139Delirium screening in a resource-poor medical intensive care setting

1. Introduction 2.3. Instruments

The symptoms of delirium include a constellation ofphysical, biologic, and psychologic disturbances. It ischaracterized by rapid onset and a fluctuating course ofclouding of consciousness, attention deficits, cognitivedisturbances, psychomotor changes, and sleep disturbances[1]. Delirium is a common but underdiagnosed complicationseen in patients admitted to intensive care units (ICUs), withestimated prevalence of delirium being as high as 80% incritical care units [2-4]. The presence of delirium can lead toincreasedmorbidity, such as lengthened hospital stay [5,6], aswell as mortality [7]. The diagnosis of delirium by the In-ternational Classification of Diseases criteria is done using aclinical interview and examination, which is time-consumingand needs expert professionals who may not be readilyavailable in ICU settings in resource-poor countries. Deliriumis severely underestimated by critical care staff [8]. There is aneed for easy-to-use, valid, and reliable instruments that canbe used by available staff such as residents and nurses.

The Intensive Care Delirium Screening Checklist(ICDSC) [9], Confusion Assessment Method for the ICU[3], and the Memorial Delirium Assessment Scale [10] aresuch easy-to-use tools, which have been validated for use inpopulations in the developed world. Recently, the MemorialDelirium Assessment Scale was validated for use in a similarsetting [11]. However, there remains a dearth of instrumentsvalidated for use in resource-poor populations, and researchon the methods of early detection of delirium is warranted inview of the high prevalence and serious implications [4].

This study aims to evaluate the reliability and validity ofdiagnosing delirium with the ICDSC by nonspecialists in aresource-poor ICU setting in South India.

2. Materials and methods

2.1. Study design

This study is a test of diagnostic accuracy attempting totest the validity and reliability of the ICDSC as comparedwith an expert interview based on the InternationalClassification of Diseases, 10th Revision (ICD-10) diagnos-tic criteria for research [1] to diagnose delirium (F05.0,F05.1, and F05.9).

The study was approved by the institutional ethics boardbefore its execution.

2.2. Study setting

The study took place at the Dr Somervell Memorial CSIMedical College, situated in a rural setting, in Karakonam,Trivandrum, India. The hospital is a 550-bed, multispecialtyteaching institute that has 12 beds in the medical ICU and 4beds in the cardiac ICU.

The ICDSC developed in 2001 by Bergeron et al [9],based on Diagnostic and Statistical Manual of MentalDisorders, Fourth Edition (DSM-IV) criteria, is used as ascreening instrument for detecting delirium. Its sensitivity,specificity, and interrater reliability when used by intensivecare staff have been evaluated previously and found to beadequate. Other investigators such as Devlin et al [12]showed that the ICDSC, along with education supportingits use, improved the ability of both nurses and physiciansto detect delirium at the bedside. It can be administered in 7to 10 minutes by the nonspecialist staff in the ICU. Itconsists of 8 items with a score of 0 or 1 for each item,based on the information collected in the previous 24 hours.The ICD-10 diagnostic criteria for research [1], presentedin Table 1, was used to diagnose delirium (F05.0, F05.1,and F05.9) by the consultant psychiatrist as the referencestandard. The ICD-10 includes 2 additional items (psycho-motor and sleep-wake cycle disturbances) compared withthe DSM-IV, which was used in the original studyvalidating the instrument [9]. The ICD-10 criterion waschosen because it is being used in everyday clinical practicein the study setting.

2.4. Sample size estimation

The study wanted to establish a 90% sensitivity of ICDSCto screen for delirium with a precision of 8.

It estimated using the formula

ss =Z2T pð ÞT 1 − pð Þ

c2

and a sample size for a 2-sided test, as 54 was obtained.

2.5. Study population

All consecutive patients to the medical ICU over a periodof 2 months who had been admitted for a period of more than48 hours were assessed for possible inclusion in the study ifthey were (a) older than 12 years and (b) if they or theirrelatives were willing or able to give written consent.

The patients were excluded if they or their relativesrefused consent or if they were intubated or mute. Intubatedpatients were excluded because the previous study validatingthe ICDSC did not include patients who were comatose andstuporous [9], and in the study setting, sedatives wereregularly used on intubated patients. Eleven patients withsuch status were excluded.

2.6. Training of the residents

The residents (A.G. and A.D.) were undergoing their 3-month rotation in the general medicine department. Thetraining for a total duration of 8 hours involved imparting

Table 1 ICD-10 diagnostic criteria for F05 delirium, notinduced by alcohol and other psychoactive substances

A. There is clouding of consciousness, that is, reduced clarity ofawareness of the environment, with reduced ability to focus,sustain, or shift attention.

B. Disturbance of cognition is manifested by both:1. Impairment of immediate recall and recent memory, withrelatively intact remote memory2. Disorientation in time, place, or personC. At least 1 of the following psychomotor disturbances ispresent:1. Rapid, unpredictable shifts from hypoactivity tohyperactivity2. Increased reaction time3. Increased or decreased flow of speech4. Enhanced startle reactionD. Disturbance of sleep or of the sleep-wake cycle, manifestedby at least 1 of the following:1. Insomnia, which in severe cases may involve total sleep loss,with or without daytime drowsiness, or reversal of the sleep-wake cycle2. Nocturnal worsening of symptoms3. Disturbing dreams and nightmares, which may continue ashallucinations or illusions after awakening

E. Symptoms have rapid onset and show fluctuations over thecourse of the day.

F. There is objective evidence from history, physical andneurologic examination, or laboratory tests of an underlyingcerebral or systemic disease (other than psychoactivesubstance–related) that can be presumed to be responsible forthe clinical manifestations in criteria A to D.

Comments: Emotional disturbances such as depression, anxietyor fear, irritability, euphoria, apathy or wondering perplexity,disturbances of perception (illusions or hallucinations, oftenvisual), and transient delusions are typical but are not specificindications for the diagnosis.

Use the 4th character to indicate whether the delirium issuperimposed on dementia or not:F05.0 Delirium, not superimposed on dementiaF05.1 Delirium, superimposed on dementiaF05.8 Other deliriumF05.9 Delirium, unspecified

For a definite diagnosis, symptoms, mild or severe, should be present ineach of the aforementioned areas.

140 C. George et al.

general information on the study, training in administrationof the instrument, eliciting the specific items, and scoring ofthe ICDSC, as well as collection of background data. Thisalso involved a piloting phase of assessing 10 patientsadmitted to the ICU.

2.7. Data collection

The patients were assessed in the ICU by either A.G. orA.D. The study details were explained using a standardinformation sheet, and written consent was obtained fromthe patient where possible or, where this was not possible,

from the relative. Sociodemographic and clinical detailswere recorded in a structured pro forma. The ICDSC wasadministered once on each of the patients by either of theresidents. On the same working day, the consultantpsychiatrist independently screened these patients onceand established the presence, if any, of the diagnosis ofdelirium consistent with ICD-10 diagnostic criteria forresearch (Table 1) by clinical interviews. The treating teamwas informed about the possibility of delirium, andnecessary consultation services were provided whererequested. All investigators remained masked to each other'sfindings until the end of data analysis to minimize observerbias. Twenty-one patients were independently assessed byA.G. and A.D. within 1 to 2 hours of each other to ascertaininterrater reliability.

2.8. Data analysis

The sociodemographic data, clinical profile, and data onICDSC were analyzed using descriptive statistics. Theinterrater reliability of ICDSC was calculated using 2-way,mixed-effects, average-measure intraclass correlation coef-ficient statistic. The internal consistency of ICDSC wasassessed by Cronbach α and Guttman split-half coefficient.The data-structuring was assessed with principal axisfactoring, followed by varimax rotation with Kaisernormalization. A receiver operating characteristic (ROC)curve was obtained to evaluate the performance of ICDSC toclassify delirium. The data were analyzed using SPSSversion 13.0 (SPSS, Chicago, Ill).

3. Results

Fifty-nine patients who were admitted into the ICUconsecutively and stayed for longer than 48 hours, fulfillingselection criteria, were recruited into the study. They werepredominantly male (n = 35; 59.3%), and 10 (16.9%) wereilliterate. Nine (15.3%) individuals had dementia, whereas8 (13.6%) fulfilled the criteria for harmful use of alcohol/alcohol dependence syndrome, and 13 (22.0%) had a historyof depression or psychosis. Six (10.2%) had been admittedwith a diagnosis of organophosphorus poisoning (Table 2).

All 59 patients were evaluated by the consultantpsychiatrist. Twenty (33.9%) fulfilled the ICD-10 diagnosticcriteria for research for delirium. When compared with thisreference standard, ICDSC classified delirium adequately(area under ROC curve, 0.837). This ROC curve is displayedin Fig. 1. The sensitivity, specificity, positive predictivevalue, and negative predictive value for various cutoff scoresof ICDSC are presented in Table 3. A diagnostic cutoff valueof the ICDSC total score of 3 or more provided 90%sensitivity and 61.54% specificity to diagnose delirium.Positive and negative predictive values for this cutoff scorewere 54.5% and 92%, respectively.

Table 2 Baseline characteristics of patients assessed fordelirium a (N = 59)

Withoutdelirium(n = 39)

Withdelirium(n = 20)

Total(N = 59)

Demographic featuresAge (y), mean (SD) 51 (28.99) 76 (21.21) 60 (50.24)Sex, n (%)Male 23 (59) 12 (60) 35 (59.3)Female 16 (41) 8 (40) 24 (40.7)Ethnicity, n (%)Tamil 3 (7.7) 4 (20) 7 (11.9)Malayalee 36 (92.3) 16 (80) 52 (88.1)Religion, n (%)Hindu 11 (28.2) 3 (15) 14 (23.7)Christian 28 (71.8) 17 (85) 45 (76.3)Level of education,n (%)Illiterate 9 (23.1) 1 (5) 10 (16.9)Up to secondaryeducation

25 (64.1) 17 (85) 42 (71.2)

College or universitydegree

5 (12.8) 2 (10) 7 (11.9)

Residence, n (%)Rural 38 (97.4) 19 (95) 57 (96.6)Urban 1 (2.6) 1 (5) 2 (3.4)

Clinical features, n (%)Dementia 5 (12.8) 4 (20) 9 (15.3)Alcohol dependencesyndrome/harmfuluse of alcohol

4 (10.3) 4 (20) 8 (13.6)

History of depression 6 (15.4) 2 (10) 8 (13.6)History of psychosis 2 (5.1) 3 (15) 5 (8.5)Cardiac diseases 27 (69.2) 5 (25) 32 (54.2)Renal diseases 15 (38.5) 4 (20) 19 (32.2)Neurologic diseases 14 (35.9) 8 (40) 22 (37.3)Hepatic dysfunction 5 (12.8) 1 (5) 6 (10.2)Pulmonary disease 9 (23.1) 2 (10) 11 (18.6)Hypertension 23 (59) 12 (60) 35 (59.3)Diabetes 26 (66.7) 11 (55) 37 (62.7)Infectious diseases 9 (23.1) 3 (15) 12 (20.3)Use of anticholinergicdrugs

6 (15.4) 6 (30) 12 (20.3)

Organophosphoruspoisoning

0 (0) 6 (30) 6 (10.2)

Electrolyte imbalance 6 (15.4) 3 (15) 9 (15.3)a None of the baseline patient characteristics showed significant

association with the presence of delirium.

Table 3 Sensitivity and specificity of ICDSC compared withdelirium diagnosis by ICD-10 diagnostic criteria for research

Cutoffscore

Sensitivity(%)

Specificity(%)

Positivepredictivevalue (%)

Negativepredictivevalue (%)

≥0 100 0 33.9 100≥1 100 15.38 37.74 100≥2 100 33.33 43.47 100≥3 90 61.54 54.54 92≥4 75 74.36 60 85≥5 60 82.05 63.16 80≥6 40 94.87 80 75.51≥7 25 97.44 83.33 71.69≥8 15 100 100 69.64

Figures in bold indicate cutoff scores of ICDSC to diagnose delirium.

Fig. 1 Receiver operating characteristic curve based on the totalscore (0-8) obtained on the ICDSCon the day of assessment. A cutoffscore of 3 provided a sensitivity of 90% and a specificity of 61.53%.

141Delirium screening in a resource-poor medical intensive care setting

The residents independently evaluated 21 patients toestablish interrater reliability. The interrater agreement ofICDSC total score was 0.947 (95% confidence interval,0.870-0.979). All 59 assessments by the residents wereassessed for internal consistency. The ICDSC had aCronbach α coefficient of .754 and a Guttman split-halfcoefficient of 0.71 for all items. There were adequatemeasures of sampling adequacy at greater than 0.05.

After principal axis factoring, there were 2 factors whichhad eigenvalues of at least 1. Factor I had an eigenvalue of3.070 and explained 38.376% of variance observed in ourdata. Factor II had an eigenvalue of 1.433 and explained17.908% of variance observed in our data.

The factor loadings for the individual items of the ICDSCare given in Table 4. The factor loadings of the ICDSC are asfollows: altered level of consciousness, inattention, disorien-tation, hallucination, delusion or psychosis, psychomotoragitation or retardation, inappropriate speech, or mood wasloaded on factor I (altered sensorium/psychopathology);whereas sleep-wake cycle disturbance and symptom fluctu-ation were loaded on factor II (sleep-wake cycle problems)(Table 4).

Table 4 Factor loadings for the items of ICDSC

Itemno.

Scale item Factor I Factor II

Psychopathology/altered sensorium

Sleep-wakecycle disturbances

2 Altered level ofconsciousness

0.574 0.481

3 Inattention 0.633 −0.0034 Disorientation 0.737 0.2725 Hallucination/

delusion/psychosis

0.708 0.173

6 Psychomotoragitation

0.640 −0.053

7 Inappropriatespeech or mood

0.665 0.075

1 Symptomfluctuation

0.204 0.823

8 Sleep-wake cycledisturbance

−0.086 0.897

Total varianceexplained

38.376% 17.908%

Bold numbers indicate significant factor loadings of the items of ICDSC.

142 C. George et al.

4. Discussion

This study was conducted in view of the seriousimplications of undetected delirium in ICU settings [5,6].

4.1. Primary and secondary outcome measures

The instrument had adequate sensitivity and specificityfor detecting delirium compared with the reference standardat a cutoff of 3. Therefore, adequate validity was demon-strated. The interrater reliability at 0.947 was excellent, andinternal consistency was adequate. This study was able todelineate a 2-factor structure for the scale. The diagnosis ofdelirium was higher (33.9%) in this study compared with theoriginal validation study (16%).

The study undertaken determined a cutoff score of 3 inmedically ill critical care patients on the ICDSC as providingthe most optimal sensitivity and specificity. The previousvalidation study with the ICDSC set in Canada suggested acutoff score of 4 [9]. The difference may be accounted for bythe variations in study settings, such as being set in a lowincome country, and the differing culture of subjects arguingfor local validation of instruments being required before itsuse. The other factors contributing may be the higherincidence of delirium in this series (33.9%) vs 16% in theoriginal validation study [9] and the exclusion of ventilatedpatients. The use of the ICD-10 criteria (Table 1) as thereference standard in this study, in lieu of the DSM-IV in theoriginal study, may have also contributed to the difference incutoff scores. Sleep-wake and psychomotor disturbances areobligatory items in ICD-10 and not in the DSM-IV. Theinstrument had a low predictive value (54.5%), and this may

lead to an overidentification of potentially delirious patients,probably necessitating the availability of professionals aftersuch screening.

The interrater reliability of 0.94 demonstrated in our studyamong junior residents was similar to earlier data-testingreliability between nurses and nurses and physicians [9]. Theprevious validation study of the ICDSC showed an internalconsistency of 0.71 to 0.79 [9], which was confirmed by thepresent investigation.

This is the first attempt to determine the factor structure forthis instrument. A 2-factor structure of altered sensorium/psychopathology and sleep-wake cycle disturbances (Table 4)were delineated, showing that the scale is broadly represen-tative of the ICD-10 criteria for delirium (Table 1). It is tobe noted that sleep-wake and psychomotor disturbancesare obligatory items in ICD-10 and not in the DSM-IV. Thefactors, altered sensorium/psychopathology, and sleep-wakecycle disturbances may serve to delineate the symptoms ofdelirium into cognitive and biologic dimensions.

4.2. Strengths and limitations

There is a dearth of evidence on the validity of usingscreening instruments for the detection of delirium inresource-poor countries, with, to our knowledge, only oneprevious similar attempt [11]. The strengths of the study arethe involvement of nonspecialist staff, such as residents; theuse of feasible training; pragmatic selection criteria; and thechoice of a standard reference.

Furthermore, the factor structure has been examined,besides the reliability and validity of the instrument. This isthe first attempt to examine the factor structure of the ICDSCin literature.

The present finding of a different optimal cutoff ascompared with the original study [9] further emphasizes theneed for validation of such screening tools in local settingsbefore its use.

The limitation of the study is its modest sample size, whichcould have affected the power of the study, whereas theexclusion of mute and ventilated patients may hinder thegeneralization of the findings to all patients in an ICU setting.

4.3. Interpretation and implications

An effective screening instrument for delirium is said tobe one that elicits the core features of delirium, can becompleted quickly, can be administered by nonspecialiststaff, and demonstrates adequate reliability and validity [13].The ICDSC was able to fulfill these criteria as an effectivescreening tool in the low-resource setting it was tested in.

Delirium is a frequently encountered clinical problem inICU settings all over the world. The relative scarcity ofpsychiatric professionals and low staff-to-patient ratios indeveloping countries [14] necessitate the development offeasible methods for available staff to identify delirium. The

143Delirium screening in a resource-poor medical intensive care setting

results of our study indicate that the ICDSC when used byavailable staff, such as residents with brief training, inresource-poor setting can assist in the detection of delirium inresource-poor critical care settings. A policy of incorporatingthe use of structured screening instruments such as theICDSC by staff such as residents (nonspecialists) and criticalcare nurses may enhance the detection of delirium. Suchprompt and precise identification of delirium may improvethe outcome by possibly mitigating the adverse outcomesassociated with delirium such as lengthened hospital stay andincreased mortality rates of critical care patients [5,6] inresource-poor settings.

References

[1] World Health Organization. The ICD-10 classification of mental andbehavioural disorders: diagnostic criteria for research. Geneva: WHO;1992.

[2] Roberts B, Rickard CM, Rajbhandari D, et al.: Multicentre study ofdelirium in ICU patients using a simple screening tool. Aust Crit Care2005 Feb; 18(1):6, 8-9, 11-4 passim

[3] Ely EW, Inouye SK, Bernard GR, et al. Delirium in mechanicallyventilated patients: validity and reliability of the confusion assessmentmethod for the intensive care unit (CAM-ICU). JAMA 2001;286:2703-10.

[4] Van Eijk MM, Slooter AJ, Kesecioglu J, et al. Delirium in the intensivecare unit. Ned Tijdschr Geneeskd 2008;152(51-52):2768-73.

[5] Ely EW, Gautam S, Margolin R, et al. The impact of delirium in theintensive care unit on hospital length of stay. Intensive Care Med 2001;27:1892-900.

[6] Thomason JW, Shintani A, Peterson JF, et al. Intensive care unitdelirium is an independent predictor of longer hospital stay: aprospective analysis of 261 non-ventilated patients. Crit Care 2005;9:R375-81.

[7] Ely EW, Shintani A, Truman B, et al. Delirium as a predictor ofmortality in mechanically ventilated patients in the intensive care unit.JAMA 2004;291(14):1753-62.

[8] Spronk PE, Riekerk B, Hofhuis J, et al. Occurrence of delirium isseverely underestimated in the ICU during daily care. Intensive CareMed 2009;35:1276-80.

[9] Bergeron N, Dubois MJ, Dumont M, et al. Intensive Care DeliriumScreening Checklist: evaluation of a new screening tool. IntensiveCare Med 2001;27:859-64.

[10] Breitbart W, Rosenfeld B, Roth A, et al. The Memorial DeliriumAssessment Scale. J Pain Symptom Manage 1997;13(3):128-37.

[11] Shyamsundar G, Raghuthaman G, Rajkumar Anto P, et al. ValidationofMemorial DeliriumAssessment Scale. J Crit Care 2009;24(4):530-4.

[12] Devlin JW, Fong JJ, Schumaker G, et al. Use of a validated deliriumassessment tool improves the ability of physicians to identify deliriumin medical intensive care unit patients. Crit Care Med 2007;35:2721-4.

[13] Devlin JW, Fong JJ, Fraser GL, et al. Delirium assessment in thecritically ill. Intensive Care Med 2007;33:929-40.

[14] Prayag S. ICUs worldwide: critical care in India. Crit Care 2002;6(6):479-80.