department of health care policy | health care policy - … · 2005-05-13 · international journal...

International Journal of Methods in Psychiatric Research Volume 13 Number 2 93

The World Mental Health (WMH) SurveyInitiative Version of the World HealthOrganization (WHO) CompositeInternational Diagnostic Interview (CIDI)RONALD C KESSLER1 T BEDIRHAN UumlSTUumlN2

1 Department of Health Care Policy Harvard Medical School Boston MA USA2 Global Programme on Evidence for Health Policy World Health Organization Geneva Switzerland

ABSTRACT This paper presents an overview of the World Mental Health (WMH) Survey Initiative version of theWorld Health Organization (WHO) Composite International Diagnostic Interview (CIDI) and a discussion of themethodological research on which the development of the instrument was based The WMH-CIDI includes a screeningmodule and 40 sections that focus on diagnoses (22 sections) functioning (four sections) treatment (two sections)risk factors (four sections) socio-demographic correlates (seven sections) and methodological factors (two sections)Innovations compared to earlier versions of the CIDI include expansion of the diagnostic sections a focus on 12-monthas well as lifetime disorders in the same interview detailed assessment of clinical severity and inclusion of informationon treatment risk factors and consequences A computer-assisted version of the interview is available along with adirect data entry software system that can be used to keypunch responses to the paper-and-pencil version of the inter-view Computer programs that generate diagnoses are also available based on both ICD-10 and DSM-IV criteriaElaborate CD-ROM-based training materials are available to teach interviewers how to administer the interview aswell as to teach supervisors how to monitor the quality of data collection

Key words Composite International Diagnostic Interview epidemiologic research design psychiatric diagnostic interview question wording methods

IntroductionThis paper discusses methodological issues involvedin designing the World Mental Health (WMH)Survey Initiative version of the World HealthOrganization (WHO) Composite InternationalDiagnostic Interview (CIDI) the interview used inthe US National Comorbidity Survey Replication(NCS-R Kessler and Merikangas 2004) TheWMH-CIDI was developed by WHO for use in theWHO WMH Survey Initiative The latter is a seriesof face-to-face household surveys carried out withcoordination by WHO in 28 countries around theworld (Kessler 1999 Kessler and Uumlstuumln 2000)These surveys aim to obtain valid information aboutthe prevalence and correlates of mental disorders inthe general population unmet need for treatment of

mental disorders treatment adequacy among patientsin treatment for mental disorders and the societalburden of mental disorders The focus of the currentpaper is on the sections of the WMH-CIDI that assesspsychopathology although a few words also are saidabout other sections of the instrument

Historical overviewThe first fully structured psychiatric diagnostic inter-view that could be administered by trained layinterviewers was the Diagnostic Interview Schedule(DIS) (Robins Helzer Croughan and Ratcliff 1981)The DIS was developed by Lee Robins and hercolleagues at Washington University with supportfrom the National Institute of Mental Health for usein the Epidemiologic Catchment Area (ECA) Study

IJMPR 132 3rd v4 25604 1026 am Page 93

Kessler and Uumlstuumln94

(Robins and Regier 1991) The ECA was a landmarkcommunity-based survey of mental disorders carriedout in selected neighbourhoods in five US communi-ties The wide dissemination of ECA results inhigh-profile publications led to replications in othercountries as well as to the development of other struc-tured diagnostic interviews The most widely used ofthese instruments is the WHO CIDI (World HealthOrganization 1990) The CIDI is an expansion of theDIS that was developed under the auspices of WHOby an international task force under the supervisionof Lee Robins to address the problem that DIS diag-noses are exclusively based on the definitions andcriteria of the American Psychiatric Associationrsquos(APA) Diagnostic and Statistical Manual (DSM) ofMental Disorders (Robins Wing Wittchen HelzerBabor Burke Farmer Jablenski Pickens RegierSartorius and Towle 1988) The WHO was keen toexpand the DIS to generate diagnoses based on thedefinitions and criteria of the WHO InternationalClassification of Disease (ICD) This was especiallyimportant for cross-national comparative research asthe ICD system is the international standard diag-nostic system

The CIDI was designed to encourage communityepidemiological surveys in many countries around theworld To this end a multinational CIDI editorialcommittee translated and field-tested the instrumentin many different countries (Wittchen 1994) whileWHO encouraged researchers around the world tocarry out CIDI surveys beginning in 1990 when theCIDI was first made available These efforts weresuccessful as over a dozen large-scale CIDI surveys inas many countries were completed during the firsthalf of the 1990s The WHO created theInternational Consortium in Psychiatric Epi-demiology (ICPE) in 1997 to bring together andcompare results across these surveys (Kessler 1999)The ICPE has subsequently published a number ofuseful descriptive studies of cross-national similaritiesand differences in prevalence and socio-demographiccorrelates of mental disorders (for example Aguilar-Gaxiola Alegria Andrade Bijl Caraveo-AnduagaDeWit Kolody Kessler Uumlstuumln Vega and Wittchen2000 Alegria Kessler Bijl Lin Heeringa Takeuchiand Kolody 2000 Bijl de Graaf Hiripi KesslerKohn Offord Uumlstuumln Vicente Vollebergh Waltersand Wittchen 2003 WHO InternationalConsortium in Psychiatric Epidemiology 2000)

However the work of the ICPE with this first genera-tion of CIDI surveys was hampered by the fact thatcomparability among the surveys was limited to theassessment of mental disorders Measures of riskfactors consequences patterns and correlates oftreatment and treatment adequacy none of whichwere included in the CIDI were not assessed in aconsistent manner across the surveys

Recognizing the value of coordinating themeasurement of these broader areas of assessmentthe ICPE launched an initiative in 1997 to bringtogether the senior scientists in planned CIDIsurveys prior to the time their surveys were carriedout in order to coordinate measurement Within ashort period of time research groups in over a dozencountries joined this initiative The World HealthOrganization officially established the WHO WMHSurvey Initiative to coordinate this undertaking in1998 Since that time the number of participatingWMH countries has expanded to 28 with an antici-pated combined sample size of over 200000interviews The authors of the current paper are theco-directors of both the ICPE and the WMH SurveyInitiative as well as the principal developers of theWMH-CIDI the expanded version of the WHOCIDI that was created for use in the WMH surveys

An overview of the WMH-CIDIIn the course of expanding the CIDI to includebroader areas of assessment we also took the oppor-tunity to make the diagnostic sections of the CIDImore operational We expanded questions to breakdown critical criteria including the clinical signifi-cance criteria required in the DSM-IV system Weexpanded the diagnostic sections to include dimen-sional information along with the categoricalinformation that existed in previous CIDI versionsWe also expanded the number of disorders includedin the CIDI

The 41 sections in the WMH-CIDI are listed inTable 1 These are not in their order of assessmentThe first section is an introductory screening andlifetime review section the logic of which isdiscussed later in this article There are also 22 diag-nostic sections that assess mood disorders (twosections) anxiety disorders (seven sections)substance-use disorders (two sections) childhooddisorders (four sections) and other disorders (sevensections) Four additional sections assess various

IJMPR 132 3rd v4 25604 1026 am Page 94

WMH Survey Initiative Version of the CIDI 95

kinds of functioning and physical comorbidity Twoassess treatment of mental disorders Four assess riskfactors Six assess socio-demographics Two finalsections are methodological The first of these twoincludes rules for determining which respondents toselect into Part II of the interview and which ones toterminate after Part I of the interview The secondmethodological section consists of interviewer obser-vations that are recorded after the interview hasended

The entire WMH-CIDI takes an average ofapproximately 2 hours to administer in most generalpopulation samples However interview time varieswidely depending on the number of diagnosticsections for which the respondent screens positiveAs mentioned in the last paragraph the interviewhas a two-part structure that allows early terminationof a representative subsample of respondents whoshow no evidence of lifetime psychopathology Thesampling fraction used in this subselection procedureinfluences average interview time Finally a number

of WMH-CIDI sections are optional and can beadministered to subsamples rather than to the entiresample This too reduces average interview length

In addition to the interview schedule we devel-oped an elaborate set of training materials to teachinterviewers how to administer the WMH-CIDI andto teach supervisors how to monitor the quality ofdata collection We developed a computer-assistedversion of the interview (CAI) that can be used withlaptop computers We also developed a direct dataentry (DDE) software system that can be used tokeypunch paper and pencil versions of the interviewFinally we developed computer programs thatgenerate diagnoses from the completed survey datausing the definitions and criteria of the ICD-10 orthe DSM-IV diagnostic systems

Use of the WMH-CIDI requires successfulcompletion of a training programme offered by anofficial WHO CIDI Training and Research Centre(CIDI-TRC) Another innovation associated withthe WMH-CIDI is a state-of-the art interviewer

Table 1 An outline of the WMH-CIDI

I Screening and lifetime review

II Disorders

Mood Major Depression Mania

Anxiety Panic Disorder Specific Phobia Agoraphobia Generalized Anxiety Disorder Post-Traumatic Stress Disorder Obsessive-Compulsive Disorder Social Phobia

Substance abuse Alcohol Abuse Alcohol Dependence Drug Abuse Drug Dependence Nicotine Dependence

Childhood Attention-DeficitHyperactivity Disorder Oppositional-Defiant Disorder Conduct Disorder Separation Anxiety Disorder

Other Intermittent Explosive Disorder Eating Disorders Premenstrual Disorder Non-Affective Psychoses Screen Pathological Gambling Neurasthenia Personality Disorders Screens

III Functioning and physical disorders Suicidality 30-day Functioning 30-day Psychological Distress Physical Comorbidity

IV Treatment Services Pharmacoepidemiology

V Risk factors Personality Social networks Childhood experiences Family Burden

VII Socio-demographics Employment Finances Marriage Children Childhood Demographics Adult Demographics

VII Methodological Part I ndash Part II Selection Interviewer Observations

IJMPR 132 3rd v4 25604 1026 am Page 95


training programme that includes an intelligent 40-hour CD-ROM-based self-study module in additionto a three-day face-to-face training module thatrequires the trainee to travel to an authorized CIDI-TRC The latter is designed for individuals who havesuccessfully completed the self-study module as indi-cated by passing the self-administered testsembedded throughout the CD-ROM Remedialtraining elements are embedded in the CD-ROMwhenever a trainee fails an embedded test Traineeswho successfully complete the certification process atthe end of this program are given access to all WMH-CIDI training materials for use in traininginterviewers and supervisors They are also givencopies of the WMH-CIDI CAI and DDE programsand the computerized diagnostic algorithms A PDFcopy of the WMH-CIDI and contact information forWMH-CIDI training can be obtained from the CIDIWeb page at wwwwhointmsacidi

The validity of CIDI diagnostic assessments A number of DIS and CIDI validity studies werecarried out prior to the time the WMH SurveyInitiative was launched These studies aimed todetermine whether the diagnoses generated by theseinstruments are consistent with those obtained inde-pendently by trained clinical interviewers whoadminister semi-structured research diagnosticinterviews to a probability sample of survey respon-dents who previously completed the DIS or CIDIWittchen (1994) reviewed these studies up throughthe early 1990s Only a handful of DIS or CIDIvalidity studies have been published since that time(Kessler Wittchen Abelson McGonagle SchwarzKendler Knauper and Zhao 1998 Wittchen Uumlstuumlnand Kessler 1999 Brugha Jenkins Taub Meltzerand Bebbington 2001) Results show that DIS andCIDI diagnoses are significantly related to indepen-dent clinical diagnoses but that individual-levelconcordance is far from perfect Some part of thislack of concordance is doubtless due to unreliabilityof clinical interviews Indeed the literature is clearin showing that test-retest reliability is higher fordiagnostic classifications based on DIS-CIDI inter-views than semi-structured clinical interviewsHowever there is also the issue of validity which ispresumably higher in semi-structured clinical inter-views than in fully structured DIS-CIDI interviewsAs a result of concerns about validity considerable

interest existed among the developers of the WMH-CIDI to improve the validity of the CIDI for use inthe WMH surveys

Based on previous evaluations of the CIDI bysurvey methodologists in preparation for the USNational Comorbidity Survey (NCS) (KesslerWittchen et al 1998 Kessler Mroczek and Belli1999 Kessler Wittchen Abelson and Zhao 2000)four main methodological problems were the focusof our work revising the diagnostic sections of the WMH-CIDI One was that respondents mightnot understand some of the CIDI questions anumber of which included multiple clauses andvaguely defined terms A second was that somerespondents might not understand the task impliedby the questions which sometimes required carefulmemory search that was unlikely to be carried out unless respondents were clearly instructed to do so A third was that respondents might not be motivated to answer accurately especially inlight of the fact that many CIDI questions deal withpotentially embarrassing and stigmatizing experi-ences A fourth was that respondents might not beable to answer some CIDI questions accurately especially those that asked about characteristics of mental disorders that are difficult to remember(for example age of onset number of lifetimeepisodes)

A considerable amount of methodologicalresearch has been carried out by survey researcherson each of the four methodological problemsenumerated in the last paragraph (for exampleTurner and Martin 1985 Tanur 1992 SudmanBradburn and Schwarz 1996) This research hasadvanced considerably over the past two decades ascognitive psychologists have become interested inthe survey interview as a natural laboratory forstudying cognitive processes (Schwarz and Sudman1994 1996 Sirken Herrmann Schechter SchwarzTanur and Tourangeau 1999) A number of impor-tant insights have emerged from this work thatsuggest practical ways of improving the accuracy ofself-reported psychiatric assessments As describedbelow we used these insights to help develop theWMH-CIDI The next four sections of the paperprovide a quick review of these insights as well asuse of them to address each of the four method-ological problems enumerated in the lastparagraph

IJMPR 132 3rd v4 25604 1026 am Page 96


Question comprehensionIt is obvious that ambiguous questions are likely to bemisconstrued It is perhaps less obvious though justhow ambiguous most structured questions are andhow often respondents must lsquoread between the linesrsquoIn the first systematic study of this issue Belson(1981) debriefed a sample of survey respondents on aset of standard survey questions and found that morethan 70 of respondents interpreted some questionsdifferently from the researcher leading Belson toconclude that subtle misinterpretations are pervasivein survey situations Similar conclusions have beenobtained in other survey debriefing studies(Oksenberg Cannell and Kanton 1991) Our owndebriefing studies of the CIDI found much the sameresult ndash a great many respondents misunderstoodimportant aspects of key diagnostic questions

How is it possible for so much misunderstandingto occur As Oksenberg and her colleagues discov-ered the answer lies partly in the fact that manyterms in surveys are vaguely defined Beyond this isthe more fundamental fact that the survey interviewsituation is a special kind of interaction in which thestandard rules of conversation ndash rules that help fill inthe gaps in meaning that exist in most speech ndash donot apply Unlike the situation in normal conversa-tional practice the respondent in the surveyinterview often has only a vague notion of the personto whom he or she is talking or the purpose of theconversation (Cannell Fowler and Marquis 1968)The person who asks the questions (the interviewer)is not the person who formulated the questions (theresearcher) and the questioner is often unable toclarify the respondentrsquos uncertainties about theintent of the questions Furthermore the flow ofquestions in the survey interview is established priorto the beginning of the conversation which meansthat normal conversational rules of give-and-take inquestion-and-answer sequences do not apply Thisleads to more misreading than in normal conversa-tions even when questions are seeminglystraightforward (Clark and Schober 1992) aproblem that is compounded when the topic of theinterview is one that involves emotional experiencesthat are in many cases difficult to describe withclarity

Clinical interviews attempt to deal with thisproblem by being lsquointerviewer basedrsquo (Brown 1989)that is by training the interviewer to have a deep

understanding of the criteria being evaluatedallowing the interviewer to query the respondent asmuch as necessary to clarify the meaning of ques-tions and leaving the ultimate judgment about therating with the interviewer rather than the respon-dent Indeed one might say that the interview is insome sense administered to the interviewer ratherthan to the respondent in that the responses ofinterest are responses to interviewer-based questionsof the following sort lsquoInterviewer based on yourconversation with the respondent would you saythat he or she definitely probably possibly probablynot or definitely does not meet the requirements ofCriterion Arsquo Fully structured psychiatric interviewslike the CIDI cannot use this interviewer-basedapproach because by definition they are designed sothat interviewer judgment plays no part in theresponses These lsquorespondent-basedrsquo interviews usetotally structured questions that the respondentanswers often in a yes-no format either after readingthe questions to themselves or after having an inter-viewer read the questions aloud When the criterionof interest is fairly clear there may be little differencebetween interviewer-based and respondent-basedinterviewing It is a good deal more difficult thoughto assess conceptually complex criteria with fullystructured questions

In an effort to investigate the problem of questionmisunderstanding in the CIDI as part of the pilotstudies for an early CIDI survey Kessler and hiscolleagues (Kessler Wittchen et al 1998 Kessler etal 1999) carried out a series of debriefing interviewswith community respondents who were administeredsections of the CIDI and then asked to explain whatthey thought the questions meant and why theyanswered the way they did A great deal of misunder-standing was found However enormous variationacross questions was also found in the frequency ofmisunderstanding Four discriminating features werefound among questions that had high versus lowlevels of misunderstanding

First some commonly misunderstood questionsare simply too complex for many respondents tograsp Second some commonly misunderstood ques-tions involve vaguely defined terms rather thancomplex concepts A third type of commonly misun-derstood CIDI question involves questions about oddexperiences that could plausibly be interpreted inmore than one way such as being asked about seeing

IJMPR 132 3rd v4 25604 1026 am Page 97


and hearing things that others do not Many respon-dents have a tendency to normalize these questionsand respond positively when the correct answer isnegative A fourth type of commonly misunderstoodCIDI question finally involves a contextual misun-derstanding ndash that is a misunderstanding thatderives more from the position of the question in the flow of the interview than from lack of clarity in the question A good example is the evaluation ofCriterion A in the DSM-III-R diagnosis of simplephobia which stipulates that the fear of circum-scribed stimuli must be lsquopersistentrsquo CIDI 10operationalized this criterion by asking lsquoDid thisstrong unreasonable fear continue for months oreven yearsrsquo Although seemingly not ambiguous initself pilot work by Kessler and his colleagues foundthat this question was misunderstood by a great manyrespondents because of the location of the question inthe instrument Specifically this question followedan open-ended question that asked the respondent togive an example of a specific fear In many cases therespondent would respond to this open-ended ques-tion by describing the autonomic arousal symptomsthat occur on exposure to the stimulus such as feelingdizzy or having trouble breathing When the follow-up question was administered right after thisdescription ndash asking whether this fear continued formonths or even years ndash the question was sometimesmisunderstood as asking about the duration of thearousal symptoms The respondent would invariablyanswer no the nausea or dizziness or other physiolog-ical symptoms typically lasted no more than a fewhours and certainly never lasted months or yearsConfusions of this sort can lead to substantial errorsin fully structured clinical assessments especiallywhen they concern required symptoms

Following the work of Kessler et al (2000) wecarried out detailed CIDI debriefing interviews withvolunteer respondents in methodological studies ofthe WMH-CIDI in an effort to pinpoint CIDI ques-tions with each of the above four types ofcomprehension problems Misunderstandings basedon complex questions are addressed by breakingdown the original CIDI questions into less complexsubquestions Especially complex questions arepresented in a respondent booklet (RB) thatprovides a visual aid as the questions are being readby the interviewer Misunderstandings based on thevagueness of terms are addressed by introducing clar-

ifications and examples Misunderstandings based onnormalization of questions about odd experiences areaddressed by prefacing the questions with clarifica-tion that we are actually asking about oddexperiences and that it is important for us to learnhow often these experiences occur Contextualmisunderstandings finally are resolved by reorderingquestions to remove the contextual effects and byadding clarifying clauses in questions where residualconfusion might exist Although the number ofmodifications of this sort are so large that each onecannot be reviewed here the appendix to this paperpresents one example of each of the four types ofmodifications to illustrate the types of changes wemade to the CIDI in developing the WMH-CIDI

Task comprehensionRespondents not only sometimes misunderstandsurvey questions but they also sometimes misunder-stand the fundamental task they are being asked tocarry out Debriefing studies have shown that misun-derstandings of the second sort are especiallycommon with the diagnostic stem questions in theCIDI (Kessler et al 2000) These stem questions arethe first questions asked in each diagnostic sectionThey are used to determine whether a lifetimesyndrome of a particular sort might have everoccurred The questions provide what are in effectbrief vignettes and ask the respondent whether theyever had an experience of this sort If so additionalquestions assess the specifics of the syndrome If notthe remaining questions about this syndrome areskipped Substantial confusion arises from respon-dentsrsquo failure to understand the purpose of such stemquestions In particular only about half of pilotrespondents in the Kessler study interpreted thesequestions as they were intended by the authors of theCIDI namely as a request to engage in activememory search and report episodes of the sort in thequestion The other respondents interpreted thequestion as a request to report whether a memory ofsuch an episode was readily accessible These latterrespondents did not believe that they were beingasked to engage in active memory search and did notdo so Not surprisingly these respondents were muchless likely than those who understood the intent ofthe question to remember lifetime episodes

Why did so many respondents misinterpret theintent of these lifetime recall questions As Marquis

IJMPR 132 3rd v4 25604 1026 am Page 98


and Cannell (1969) discovered in their earlyresearch on standard interview practice respondentsare generally ill-informed about the purposes of theresearch and poorly motivated to participate activelyFurthermore cues from interviewers often reinforcethis inclination to participate in a half-hearted wayFor example when an interviewer asks a questionthat requires considerable thought the respondent islikely to assume in the absence of instructions to thecontrary that the interviewer is operating undernormal conversational rules and as such is reallyasking for an immediate and appropriate answerCannell et al (1981) shows that this conversationalartifact can be minimized by explicitly instructingrespondents to answer completely and accuratelyThe use of such instruction can substantiallyimprove the quality of data obtained in surveysBased on this result we built in clarifying statementsthroughout the WMH-CIDI aimed at informingrespondents that accuracy was important See theappendix for an example of such a statement

MotivationOne problem with emphasizing to respondents theneed to work hard at a series of demanding andpotentially embarrassing recall tasks is that morerespondents than otherwise may refuse the jobRecognition of this problem among survey method-ologists has led to the development of motivationaltechniques intended to increase the chances thatrespondents will accept the job of answeringcompletely and accurately Three techniques thathave proven to be particularly useful in this regardare the use of motivational components in instruc-tions the use of contingent reinforcement strategiesembedded in interviewer feedback probes and theuse of respondent commitment questions

Motivational instructionsThere is evidence that the use of introductoryremarks at the beginning of a survey that clarify theresearch aims can help motivate respondents toprovide a more complete and accurate report thanthey would otherwise (Cannell et al 1981)Debriefing shows that respondents are more willingto undertake laborious and possibly painful memorysearches if they recognize some altruistic benefit ofdoing so Even such an uncompelling rationale as lsquoitis important for our research that you take your time

and think carefully before answeringrsquo has motiva-tional force This is even more so when instructionsinclude statements that have universalistic appealsuch as lsquoaccuracy is important because social policymakers will be using these results to make decisionsthat affect the lives of all of usrsquo Based on thisevidence we developed and presented a statementcontaining a clear rationale for administering theinterview at the onset of the WMH-CIDI interviewschedule and emphasized the importance of thesurvey for social policy purposes See the appendixfor this text In addition in the case of especiallyimportant questions that require long-term recall toanswer correctly the questions are included in theRB and a written instruction is included at the top ofthe page in capital letters urging the respondent tolsquotake your time and think carefully before answeringrsquo

Contingent reinforcementConsistent with research on behavioural modifica-tion of verbal productions through reinforcement(see for example Centers 1964) several surveyresearchers have demonstrated that verbal reinforcerssuch as lsquothanksrsquo and lsquothatrsquos usefulrsquo can significantlyaffect the behaviour of survey respondents (Marquiset al 1969) Based on this observation Cannell andhis associates developed a method for training inter-viewers to use systematic feedback ndash both positiveand negative ndash to reinforce respondent effort inreporting (Oksenberg Vinokur and Cannell 1979a)The central feature of this method is the use of struc-tured feedback statements coordinated with thecontent and timing of instructions aimed at rein-forcing respondent performance It is important torecognize that it is performance that is being rein-forced rather than the content of particular answersFor example a difficult recall question may be pref-aced with the instruction lsquoThis next question may bedifficult so please take your time before answeringrsquoIn contingent feedback instruction interviewers issuesome expression of gratitude whenever the respon-dent seems to consider his or her answer carefullywhether they remember anything or notAlternatively the interviewer might instruct theprecipitous respondent lsquoYou answered that awfullyquickly Was there anything (else) even somethingsmallrsquo Such invitations to reconsider would occurwhenever the respondent gives an immediate answerwhether or not anything was reported

IJMPR 132 3rd v4 25604 1026 am Page 99


Experiments carried out by Cannell and his associ-ates (Miller and Cannell 1977 Vinokur Oksenbergand Cannell 1979) have documented that thecombined use of these contingent reinforcementprobes with instructions explaining the importanceof careful and accurate reporting leads to substantialimprovement in recall of health-related events ingeneral population surveys including validated datesof medical events Importantly their results alsoshow that self-enhancing response biases are reducedwhen these strategies are used as indicated by both adecreased tendency to under-report potentiallyembarrassing conditions and behaviours (forexample gynaecological problems seeing an X-ratedmovie) and a decreased tendency to over-report self-enhancing behaviours (for example the number ofbooks read in the last 3 months reading the editorialpage of the newspaper the previous day) Based onthese results the Cannell contingent feedbackapproach is included as a fundamental part of theinterviewer training materials developed for theWMH surveys

Commitment questionsInstructions that define the nature of interviewerexpectations for respondent behaviour help to estab-lish a perspective on the interview that can havemotivational force The literature on cognitivefactors in surveys contains many examples of thesubtle ways in which perspectives established inquestions subsequently influence respondent behav-iours (see for example Loftus and Palmer 1974)This same literature shows that perspective can havemotivational force when it implies a commonpurpose (Clark et al 1992) That is if a question isposed in such a way that it implies that hard workwill be invested in arriving at an answer it is incum-bent on the respondent either to demur explicitly ortacitly accept the task of working hard as part of thecommon understanding between interviewer andrespondent By answering the question the respon-dent in effect makes a commitment to honour theinjunction implied in the perspective of the questionand this implied commitment in turn creates moti-vation to this task (Marlatt 1972) Based on thistype of thinking Cannell and his colleagues showedthat it is possible to motivate respondents to acceptthe goal of serious and active reporting by asking anexplicit commitment question as part of the inter-

view Experimental studies carried out by Cannelland his associates (Oksenberg et al 1979aOksenberg Vinokur and Cannell 1979b Cannell etal 1981) have shown that commitment questionsimprove accuracy of recall Based on this result weadded a commitment question in the screeningsection of the WMH-CIDI just before the adminis-tration of lifetime diagnostic stem questions See theappendix for the text of this question

The ability to answer accurately

Episodic and semantic memoriesResearch on basic cognitive processes has shown thatmemories are organized and stored in structured setsof information packages commonly called schemas(Markus and Zajonc 1985) When the respondenthas a history of many instances of the same experi-ence that cannot be discriminated the separateinstances tend to blend together in memory to form aspecial kind of memory schema called a lsquosemanticmemoryrsquo a general memory for a prototypical experi-ence (Jobe White Kelley Mingay Sanchez andLoftus 1990 Means and Loftus 1991) For examplethe person may have a semantic memory of whatpanic attacks are like but due to the fact that he hashad many such attacks in his lifetime cannot specifydetails of any particular panic attack In comparisonwhen the respondent has had only a small number oflifetime experiences of a certain sort or when oneinstance stands out in memory as much differentfrom the others a memory can probably be recoveredfor that particular episode This is called an lsquoepisodicmemoryrsquo

In the case of memories of illness experiencesmemory schemas tend to include not only semanticmemories of prototypic symptoms but also personaltheories about causes course and cure (LeventhalNerenz and Steele 1984 Skelton and Croyle 1991)Some of these theories will conceptualize the experi-ence in illness terms and others as a moral failing apunishment from God or a normal reaction to stress(Gilman 1988) These interpretations influence theextent to which different memory cues are capable oftriggering the schemas

The effects of memory schemas and the differencebetween semantic and episodic memories are centralthemes in research on autobiographical memoryIndeed we must determine whether episodic memories

IJMPR 132 3rd v4 25604 1026 am 00


can be recovered and whether the respondent isanswering the questions by referring to episodicmemories or by drawing inferences of what the pastmust have been like on the basis of more generalsemantic memories Research shows that people aremore likely to recover episodic memories for experi-ences that are recent distinctive and unique whilefor experiences that are frequent typical andregular people will rely more on semantic memories(Brewer 1986 Belli 1988 Menon 1994)

Asking questions without knowing the limits of memoryWhen a survey question is designed to ask about aparticular instance of an experience it must be posedin such a way that the respondent knows he is beingasked to recover an episodic memory Furthermorethe researcher must have some basis for assumingthat an episodic memory can be recovered for thisexperience If it cannot a question that asks for sucha memory implicitly invites the respondent to inferor estimate rather than remember and this can haveadverse effects on quality of reporting later in theinterview (Pearson Ross and Dawes 1992) Incomparison when a question is designed to recover asemantic memory or to use semantic memories toarrive at an answer by estimation that should bemade clear

One difficulty with these injunctions in the caseof retrospective recall questions about lifetimepsychiatric disorder is uncertainty about what levelof recall accuracy to expect Therefore as part of theWMH-CIDI pilot work we debriefed pilot respon-dents with an explicit eye towards pinpointingquestions that were difficult to answer When ques-tions of this sort were discovered an attempt wasmade to revise the questions to reduce the memoryproblem either by allowing explicitly for estimation(for example explicitly asking respondents toprovide a rough estimate) by providing categoricalresponses that reduce the complexity of the task orby decomposing the question into sub-questions thatmimic effective memory search processes Examplesof question wording modifications that employ eachof these strategies are presented in the appendix

The lifetime review sectionThe previous four sections of the paper review anumber of strategies that we use to optimize dataquality in the WMH-CIDI either by improving

understanding by enhancing respondent commit-ment or by adjusting questions to recognize thatsome respondents will be less able than others toprovide completely accurate responses We use someadditional strategies to deal with two or more ofthese problems at once The most important of thelatter was a life review section that we administernear the beginning of the interview in an effort toboth motivate and facilitate active memory search inanswering diagnostic stem questions This sectionstarts out with an introduction that explains torespondents that the questions might be difficult to answer because they require respondents to reviewtheir entire lives The introduction then goes on tosay that despite this difficulty it was very importantfor the research that these questions be answeredaccurately The introduction ends with the injunc-tion to lsquoplease take your time and think carefullybefore answeringrsquo and a commitment question thatasks respondents if they were willing to do this

The diagnostic stem questions for all core diag-noses are administered directly following thecommitment question The questions are allincluded in the RB with a written instruction tolsquotake your time and think carefully before answeringrsquoInterviewers were instructed to read the diagnosticstem questions slowly in an effort to emphasize theirimportance and to use motivational probes toencourage active memory search Our intent indeveloping this section was that we could bothexplain the serious and difficult nature of the taskand motivate respondents to engage in activememory search we hoped to stimulate by combiningall the stem questions after a fairly detailed motiva-tional introduction We also recognize based on ourdebriefing studies that CIDI respondents quicklylearn the logic of the stem-branch structure after afew sections and recognize that they can shorten theinterview considerably by saying no to the stem ques-tions This problem has been removed by asking allthe stem questions near the beginning of the inter-view before the logic of the stem-branch structurebecame clear Furthermore respondents told us indebriefing interviews that their energy flagged as theinterview progressed making it much more difficultto carry out a serious memory search later in theinterview than at the beginning

As previously noted Cannell and his associatescarried out experiments that documented powerful

IJMPR 132 3rd v4 25604 1026 am Page 101


effects of commitment questions on the accuracy ofsurvey responses (Oksenberg et al 1979a 1979bCannell et al 1981) A similar experiment wascarried out in conjunction with the NCS to evaluatethe effects of using a lifetime review section inconjunction with commitment and motivationalprobes A random sample of 200 community respon-dents was randomized either to the standard versionof the CIDI or to a version that was identical exceptthat it included the life review section As reportedin more detail elsewhere (Kessler Wittchen et al1998) this experiment documented that the lifereview section led to a significant increase in theproportion of respondents who endorsed diagnosticstem questions For example while 267 of respon-dents in the standard CIDI condition endorsed thelsquosad blue or depressedrsquo stem question for majordepression a significantly higher 406 did so in thelife-review condition Importantly a clinical validitystudy documented that this increased prevalence ofstem endorsement was not accompanied by a reduc-tion in sensitivity with regard to clinical diagnosesdocumenting that additional true cases were discov-ered by the use of the life review section and theaccompanying commitment and motivation probesThese results were the basis of adopting the lifetimereview section in the WMH-CIDI

Substantive modifications of diagnostic sectionsIn addition to the methodological modificationsdescribed above a number of important substantivemodifications were made to the diagnostic assess-ments in the WMH-CIDI aimed at addressingcurrent uncertainties about the prevalence impair-ment and appropriate diagnostic criteria for thedisorders assessed in the interview Perhaps the mostimportant of these uncertainties concerns diagnosticthresholds This uncertainty arose in no small partas a reaction to the results of early DIS and CIDIsurveys which showed that as much as 50 of thegeneral population of some countries meet lifetimecriteria for one or more ICD or DSM mental disor-ders and as many as 30 meet criteria for such adisorder in the past 12 months (WHO InternationalConsortium in Psychiatric Epidemiology 2000 Bijlet al 2003) These percentages seemed implausiblyhigh to many critics leading to the suggestion thatthe lay-administered diagnostic interviews in thesesurveys were upwardly biased (Brugha Bebbington

and Jenkins 1999 Wittchen et al 1999) Howeverclinical calibration studies showed that the preva-lence estimates in these surveys were not upwardlybiased (Kessler Wittchen et al 1998 EatonNeufeld Chen and Cai 2000) leading critics toconclude that the ICD and DSM systems themselvesare overly inclusive (Pincus Zarin and First 1998Regier Kaelber Rae Farmer Knauper Kessler andNorquist 1998 Uumlstuumln Chatterji and Rehm 1998)

This conclusion was instrumental in causing anAPA task force to add a clinical significance crite-rion to many disorders in the DSM-IV in order toremind readers of the basic definition of a mentaldisorder in the introduction of the manual asrequiring clinically significant distress or impair-ment However even when this additionalrequirement was applied post hoc to DIS and CIDIsurveys carried out in the US the 12-month preva-lence of having at least one DSM disorderequivalent to approximately 37 million adults in theUS continued to substantially exceed the numberwho could be helped with current treatmentresources (Narrow Rae Robins and Regier 2002) Inrecognition of this problem several more restrictivedefinitions have been proposed that can be used tonarrow the number of people qualifying for treat-ment (National Advisory Mental Health Council1993 Substance Abuse and Mental Health ServicesAdministration 1993 Regier 2000 Narrow et al2002 Regier and Narrow 2002)

Others however have argued against the proposalto narrow the definition of mental disorders(Mechanic 2003) and have in some cases evenargued that the definitions should be expanded toinclude what would currently be consideredsubthreshold cases (see for example MerikangasZhang Avenevoli Acharyya Neuenschwander andAngst 2003) These critics have noted that researchshows many syndromes currently defined as mentaldisorders to be extremes on continua that appear notto have meaningful thresholds (see PreisigMerikangas and Angst 2001) This is important forresearch purposes because exploration of the fullcontinua rather than the currently established diag-nostic thresholds yields greater power in studies ofgenetic and environmental risk factors (BenjaminEbstein and Lesch 1998)With regard to diagnosticthresholds these critics note that research has shownsubthreshold cases on some of these continua to be

IJMPR 132 3rd v4 25604 1026 am Page 102


quite impaired (for examples Judd Paulus Wells andRapaport 1996) and to have significantly elevatedrisk of serious outcomes such as suicide attempts andhospitalization for emotional problems (KesslerBarber Beck Berglund Cleary McKenas PronkSimon Stang Uumlstuumln and Wang 2003) This meansthat the development of early interventions to treatthese subthreshold cases might prevent progressionalong a given severity continuum thereby reducingthe prevalence of serious cases in a cost-effectivefashion (Eaton Badawi and Melton 1995) Removalof these subthreshold cases from the ICD or DSMsystems in comparison might result in the impor-tance of developing interventions for these cases tobe ignored as well as to a distortion occurring in thereality that mental disorders like physical disordersvary widely in seriousness (Spitzer 1998 Kendell2002)

The final adjudication between these competingviews will doubtless take years to occur and will relyon emerging information about the genetics ofmental disorders as well as on information abouttreatment response across the range of the symptomseverity continuum To the extent that epidemiolog-ical data will play a part they will at a minimumneed to include assessments of subthreshold casesassessments of symptom severity in dimensionalterms and evaluation of the association betweensymptom severity and impairment The WMH-CIDIis designed to do all three of these things as brieflydiscussed in the next three subsections

Subthreshold disordersOur general approach in modifying WMH-CIDIdiagnostic sections is to include as much informationas possible about subthreshold cases with the precisenature of the subthreshold assessment guided by theliterature and our preliminary studies For examplein the case of depression even though the diagnosticcriteria require dysphoria or anhedonia that persistsmost of the day we found that a great many peopleotherwise meet the criteria for a major depressiveepisode except that their symptoms persist only forabout half the day or sometimes less than half theday We include these people in our assessment ofdepressive disorders In addition we include peoplewith as few as two symptoms in their worst lifetimeepisode of depression in order to capture cases ofminor depression even though larger numbers are

required in the ICD and DSM systems to be consid-ered a major depressive episode We also includepeople with depressive episodes as short as 3 days ifthey report ever having episodes of this sort mostmonths for an entire year in a row This is done toallow an assessment of recurrent brief depression(Angst Merikangas Scheidegger and Wicki 1990)

To take a second example in the case of panic wecarry out a complete assessment of people who reportat least one lifetime limited symptom attack or panicattack Age of onset and the circumstancessurrounding the single attack were collected Parallelinformation is collected about the first lifetimeattack among people who meet criteria for a panicdisorder Separate information is obtained from thelatter people about age of onset of the transition frompanic attacks to panic disorder Previous research hasshown that only about half of the people who have asingle lifetime panic attack go on to develop panicdisorder (Eaton Kessler Wittchen and Magee1994) The information collected in the WMH-CIDIallows us to obtain separate prevalence estimates ofpanic attacks and panic disorder as well as to studythe separate predictors of progression from panicattacks to panic disorder A similar distinction ismade between specific fears and phobias with sepa-rate dating of age of onset of the fear and avoidance

As a final example we include a complete assess-ment of generalized anxiety disorder (GAD) forrespondents who report having episodes that persistat least one month the original duration require-ment in DSM-III rather than requiring the 6-monthminimum episode duration stipulated in DSM-IVand ICD-10 The decision by the developers of theDSM to increase the 1-month minimum durationrequirement for GAD in DSM-III to six months inDSM-III-R was based on the fact that the vastmajority of patients with GAD in treatment sampleshad comorbid depression unless their episodes ofGAD persisted for at least six months (Breslau andDavis 1985) However subsequent epidemiologicalresearch showed that pure cases of GAD with shorterdurations exist in the general population but seldomcome to clinical attention because professional help-seeking is often driven by comorbid disorders Yetthis does not mean that people with recurrentepisodes of GAD lasting less than 6 months are notimpaired Indeed the largest and most comprehen-sive study of this matter carried out by Maier et al

IJMPR 132 3rd v4 25604 1026 am Page 103


(2000) in the WHO study of mental disorders inprimary care found that the 1-month durationrequirement is optimal for distinguishing cases ofGAD from non-cases in terms of role impairmentThe decision to require a minimum duration of 1month rather than 6 months in the WMH-CIDI isbased on this result For a similar reason we assesscases of subthreshold GAD who fail to report all thepsycho-physiological symptoms required by DSM-IVor ICD-10

Symptom persistence and severity The focus of the CIDI like that of the DIS before ithas largely been on lifetime disorders although a 12-month version of the CIDI was developed in the lastrevision of the instrument The standard lifetimeversion of the CIDI provides only superficial infor-mation about recent disorders by asking no morethan a single question ndash lsquoHow recently have you had[the disorder]rsquo ndash to learn about recency after theassessment of lifetime symptom clustering Thismakes it impossible to characterize the persistence ofdisorders over the recent past or to know whetherrespondents with a lifetime disorder have met fullcriteria during the recent past As 12-month preva-lence is of great interest for needs assessment thissuperficial consideration of 12-month prevalence is aserious limitation The WMH-CIDI addresses thisproblem by obtaining information about 12-monthsymptoms and persistence of symptoms over the pastyear In the case of panic and intermittent explosivedisorder this is done by asking about number ofattacks in the past 12 months as well as about thenumber of months when the respondent had at leastone attack In the case of episodic disorders such asdepression and GAD 12-month duration is assessedby asking how many weeks out of the past 52 therespondent has been in an episode

Similar data are obtained in the WMH-CIDI toincrease understanding of long-term course In thestandard version of the CIDI information on courseis limited to two questions about age of onset and ageof recency of the disorder The WMH-CIDI expandsthis assessment to ask about persistence in theinterval between these two ages along the same linesused to assess 12-month persistence (for examplewith questions about lifetime number of panic andanger attacks lifetime number of episodes of depres-sion and mania and GAD typical and longest

durations of episodes and number of years in whichthe respondent experienced at least one attack orone episode of the disorder) In keeping with theprior comments on the limits of autobiographicalmemory and consistent with the results of ourmethodological pilot studies we recognize thatrespondents with complex histories ofpsychopathology will be unable to recover episodicmemories in answering these long-term recall ques-tions As a result the questions are worded in such away as to make it clear that we are looking forsemantic memories Even with this limitation inmind though these data can be extremely useful in distinguishing between broad categories of peoplewho have had only one or two a few more than afew or a large number of attacks or episodes ofepisodic disorders

The WMH-CIDI also includes much more exten-sive information on symptom severity than thestandard CIDI Each diagnostic section containsexplicit questions about the depth of the distresscaused by the disorder along with a 12-monthsymptom severity scale based on a fully structuredversion of a standard clinical scale For example thequick self-report version of the Inventory ofDepressive Symptomatology (Rush Gullion BascoJarrett and Trivedi 1996) is used to assess theseverity of 12-month depression while a structuredversion of the Panic Disorder Severity Scale (ShearBrown Barlow Money Sholomskas WoodsGorman and Papp 1997) is used to assess theseverity of 12-month panic These scales are admin-istered only to respondents who meet thesubthreshold diagnostic requirements for the disorderin question and who report that they had symptomsin the past 12 months Our hope in embedding thesestandard clinical symptom severity scales in theWMH-CIDI is that they will help create a crosswalkbetween the findings in epidemiological surveys andthe findings in clinical studies

Internal impairment The issue of impairment is related to the issue ofclinical significance The standard CIDI asks onlyone dichotomous disorder-specific role impairmentquestion for all disorders lsquoDid (the disorder) everinterfere a lot with your life or activitiesrsquo No ques-tions about impairment are asked independent ofdisorders This is inadequate for evaluating whether

IJMPR 132 3rd v4 25604 1026 am Page 104


there is clinically significant role impairment associ-ated with a particular syndrome or for investigatingthe implications of changing diagnostic thresholdson evaluations of impairment As a result theWMH-CIDI has substantially expanded the numberof the within-section questions about the impair-ments caused by individual disorders These arecalled lsquointernalrsquo impairment questions because theyask respondents to evaluate the impairment causedby a given disorder As described below the WMH-CIDI also includes two important sections that assesslsquoexternalrsquo impairment by which we mean overallimpairment in various areas of functioning withoutreference to the cause of the impairment Althoughmost of the WMH-CIDI internal impairment ques-tions focus on the worst lifetime impairment due to aparticular disorder we also include five questions ineach diagnostic section that assess impairmentamong 12-month cases Four of these are theSheehan Disability Scales (Leon Olfson PorteraFarber and Sheehan 1997) which ask respondentsto rate the impairments caused by a focal disorderduring the one month in the past year when it wasmost severe in each of four areas of life (householdduties employment social life and close personalrelationships) on a 0ndash10 scale that uses a visualanalogue scale with impairment categories of none(0) mild (1ndash3) moderate (4ndash6) severe (7ndash9) andvery severe (10) The fifth question asks respondentsto estimate the total number of days out of 365 in thepast 12 months when they were totally unable towork or carry out their other usual activities becauseof the focal disorder

External impairment Two sections of the WMH-CIDI assess externalimpairment The first is the section on 30-day func-tioning which is made up of the WHO DisabilityAssessment Schedule (WHO-DAS World HealthOrganization 1998 Rehm Uumlstuumln Saxena NelsonChatterji Ivis and Adlaf 1999) The WHO-DASassesses both the persistence (number of days in thepast 30) and severity (during the days when difficul-ties in functioning occurred) of difficulties in therespondentrsquos functioning during the 30 days beforethe interview due to all physical and mental healthproblems The dimensions of functioning assessed inthe WHO-DAS are keyed to the major categories inthe WHO International Classification of

Functioning Disability and Health (World HealthOrganization 2001) The second section of theWMH-CIDI that assesses external impairment is the section on employment which includes theWHO Health and Work Performance Questionnaire(HPQ) (Kessler Barber et al 2003) The HPQ is anexpansion of the work impairment section of theWHO-DAS that assesses the workplace costs ofillness in terms of absenteeism decrements in perfor-mance while on the job and critical workplaceincidents (such as work-related accidents) The HPQwas developed in order to provide data to employersand government health policy makers about the indirect costs of illness on the productive capacity ofthe labour force

Why assess both internal and external impairmentIt is important to obtain both internal (disorder-specific) and external (global) assessments ofimpairment Disorder-specific assessments are impor-tant because they can be used to make directcomparisons among different mental and physicaldisorders These direct comparisons are becomingincreasingly central to healthcare resource allocationdecisions as evidence-based medicine becomes thebasis for more and more triage decisions Howeverdisorder-specific assessments are limited by the factthat they require respondents to make inferencesabout the cause of their impairments This can bedifficult especially among the large number ofpeople with comorbid conditions who might have ahard time sorting out which of their conditionscauses various aspects of impairment

It is important to obtain external assessmentsbecause they allow the researcher to overcome thelimitation of disorder-specific assessments by empiri-cally estimating the relative effects of differentdisorders from prediction equations in whichmeasures of the prevalence of these disorders andtheir comorbidities are included as predictors ofglobal impairment However as it is not possible tomake detailed assessments of all possible disorders forinclusion in such prediction equations estimates ofthe impairments due to specific disorders based onanalysis of such equations are necessarily imperfectFurthermore replication of results involving the esti-mated effects of a focal disorder on impairment usingexternal comparisons requires measurement ofexactly the same set of control conditions across

IJMPR 132 3rd v4 25604 1026 am Page 105


studies This is infeasible As a result the internalassessment of impairment is more feasible despite itsconceptual limitations in comparison to the externalassessment of impairment

Ranking the impairments of mental and physical disordersIn order to provide comparative information on theimpairments of mental and physical disorders achecklist of chronic physical disorders is included inthe WMH-CIDI Internal impairment is assessedwith the five questions described above for onerandomly selected chronic condition per respondentThe random sampling strategy was used becausecomprehensive assessment of internal impairmentfor all possible chronic physical disorders would betoo time-consuming for a one-session survey devotedto mental disorders However by taking care to carryout a random selection for each respondent fromamong all the conditions reported by that respon-dent it is possible to weight the internal impairmentdata by the number of conditions reported to recoveran equal-probability sample for each chronic condi-tion for purposes of comparative assessment ofwithin-disorder role impairments

The chronic conditions checklist was modifiedfrom the list used in the National Health InterviewSurvey (NHIS) (National Center for HealthStatistics 2003) to ask about the lifetime occur-rence age of onset and recency of commonlyoccurring chronic conditions that are thought to beassociated with substantial role impairment Anumber of methodological studies have found thatsuch checklists yield valid data about disordersbrought to medical attention or that significantlylimit activities when compared to independentmedical records (Halabi Zurayk Awaida Darwishand Saab 1992 Heliovaara Aromaa KlaukkaKnekt Joukamaa and Impivaara 1993 EdwardsWinn Kurlantzick Sheridan Berk Retchin andCollins 1994 Gross Bentur Elhayany Sherf andEpstein 1996 Kriegsman Penninx van Eijk Boekeand Deeg 1996 Mackenbach Looman and van derMeer 1996) For example moderate to high agree-ment (Cohenrsquos κ Cohen 1960) has been foundbetween self-reports and medical records regardingarthritis (κ= 041) asthma (κ= 055) diabetes (κ=082) and high blood pressure (κ= 073) (Edwards etal 1994) These are lower bound estimates becausethe medical record is not a lsquogold standardrsquo especially

for chronic conditions often not brought to medicalattention (such as arthritis) for poorly definedconditions (such as back pain) and for symptom-based conditions in which the medical record merelyreproduces symptoms that are based on self-report(such as chronic headaches)

In the case of symptom-based conditions anumber of more extensive scales are used instead ofthe single yes-no questions in the chronic conditionschecklist For example we include a brief screeningscale to assess migraines that reproduces physiciandiagnoses much more accurately than a single check-list question (Lipton Dodick Sadovsky KolodnerEndicott Hettiarachchi and Harrison 2003) Othersymptom-based conditions that are assessed withscreening scales include chronic fatigue syndromeirritable bowel syndrome insomnia and unexplainedchronic pain disorder

Comparative analyses of internal impairmentcould add important information to the growing bodyof data that physical disorders often cause substantialrole impairments (Zeiss and Lewinsohn 1988 Wellset al 1989 Wells Stewart Hays Burnam RogersDaniels Berry Greenfield and Ware 1989 StewartGreenfield and Hays 1989 Ormel Von Korff UumlstuumlnPini Korten and Oldehinkel 1994 Hays WellsSherbourne Rogers and Spritzer 1995 van den Bos1995 Verbrugge and Patrick 1995 PenninxBeekman Ormel Kriegsman Boeke van Eijk andDeeg 1996 Kempen Ormel Brilman and Relyveld1997 Ormel Kempen Deeg Brilman van Sonderenand Relyveld 1998 Kempen Sanderman MiedemaMayboom-de and Ormel 2000) and mental disorders(Rhode Lewinsohn and Seeley 1990 BroadheadBlazer George and Tse 1990 Tweed 1993 CoryellScheftner Keller Endicott Maser and Klerman1993 Ormel et al 1994 Ormel et al 1998 Bijl andRavelli 2000) Such results have led to an interestamong health policy researchers in the possibilitythat expanded outreach and guideline-concordanttreatment of impairing chronic disorders mightrepresent an investment opportunity for employers(Kessler Greenberg Mickelson Meneades andWang 2001) as well as for governments (Murray andLopez 1996) However not all mental disorders havebeen studied in this way The WMH-CIDI assess-ment of internal impairment for each mentaldisorder and for one random chronic physical condi-tion allows this to be done

IJMPR 132 3rd v4 25604 1026 am Page 106


The WMH-CIDI assessment of external impair-ment has the potential to be even more important inthis regard A central limitation of the existing liter-ature on the role impairments of chronic conditionsis the lack of attention to the co-occurrence ofmultiple disorders in the same patient Many peoplewith chronic disorders suffer from more than onedisorder (Dewa and Lin 2000) Pure disorders are ingeneral less impairing than co-occurring disorders inclinical samples (Ormel et al 1994) The co-occurrence of mental disorders with chronic physicaldisorders is of special importance in this regard asstrong patterns of co-occurrence with mental disor-ders have been found for a number of commonlyoccurring physical disorders both in general popula-tion samples (Neeleman Ormel and Bijl 2001) andin primary care samples (Berardi Berti CeroniLeggieri Rucci Uumlstuumln and Ferrari 1999)

Clinical studies have found excess impairmentassociated with co-occurring mental disorders amongpeople with chronic physical disorders (SullivanLaCroix Russo and Walker 2001) As effortsincrease to rationalize the allocation of healthcareresources guided by the criteria of evidence-basedmedicine consideration of the role played by co-occurring mental disorders in causing impairmentamong patients with chronic physical disordersbecomes all the more important The inclusion ofthe external impairment in the WMH-CIDI makes itpossible to carry out such analyses in general popula-tion samples by using information about mentaldisorders physical disorders and their comorbiditiesto predict external impairment

Part I and Part II diagnosesThe WMH-CIDI is quite a long instrument with anaverage administration time of approximately 2hours for the full interview This long administrationtime can create practical complications the mostimportant of them being that it is often necessary toadminister the interview in two sessions In order toaddress this length problem a case-control approachis used in developing the WMH-CIDI whereby asubsample of respondents who complete the first halfof the interview (Part I) which includes all core diag-nostic assessments and who report having no lifetimehistory of disorder are terminated at this mid-point ofthe interview All respondents who meet criteria forany lifetime mental disorders in the Part I interview

in comparison are retained in the second half of theinterview (Part II) along with a probability sub-sample of non-cases The default value for thenon-case probability of selection is 25 althoughthis can be changed depending on the interests of theinvestigator This sampling fraction leads to Part IIsamples typically retaining between 33 and 67 ofall Part I respondents This case-control subsamplingfraction yields a high ratio of controls to cases for allbut the most prevalent disorders Statistical poweranalysis show that increasing the number of controlsto cases above these levels yields very little improve-ment in power (Schlesselman 1982) As the mainpurposes of carrying out a WMH-CIDI survey are toestimate prevalence and correlates of mental disor-ders this result implies that the subsampling ofnon-cases into Part II retains most of the efficiency ofthe full sample for central analyses while substantiallyreducing field costs

Once the Part II subsampling approach is in placeit becomes clear that average interview length isreduced whenever a section is moved from Part I toPart II Several important but lengthy diagnosticsections in the WMH-CIDI were included in Part IIfor this reason These include the assessments ofpost-traumatic stress disorder obsessive-compulsivedisorder and non-affective psychosis It should benoted that the high comorbidity of these Part IIdisorders with the disorders assessed in Part I meansthat the great majority of respondents with thesedisorders are selected into the Part II sample leadingto only a small loss of information about the Part IIdisorders by placing them in Part II rather than inPart I In addition disorders that are included in theWMH-CIDI for exploratory purposes are all placedin Part II Included here are eating disorders neuras-thenia nicotine dependence pathological gamblingpremenstrual disorder and a screen for personalitydisorders

Options for additional subsampling of the assess-ment of these exploratory disorders within the Part IIsample (for example only a random 50 of Part IIrespondents receive the assessment of pathologicalgambling) are built into the WMH-CIDI skip logicfor investigators who vary in their level of interest inthese exploratory disorders In addition a series offour diagnostic sections are included in Part II forretrospectively reported childhood and adolescentdisorders These sections which are modelled on

IJMPR 132 3rd v4 25604 1026 am Page 107


those developed by Lee Robins for the DSM-IVversion of the DIS include assessments of Attention-DeficitHyperactivity Disorder Conduct DisorderOppositional-Defiant Disorder and SeparationAnxiety Disorder

Expansions of other WMH-CIDI sectionsAs noted previously the main reason for developingthe WMH-CIDI is to expand the instrument beyondits initial focus on diagnoses to include assessmentsof risk factors consequences and treatment TheWMH-CIDI includes 14 sections of this sort Sixsections assess socio-demographics (employmentfinances marriage children adult and childhooddemographics for the Part I sample adult demo-graphics for the Part II sample) Two sections assesstreatment (services pharmacoepidemiology) Theother six sections assess external impairmentchronic conditions non-specific psychologicaldistress social networks family burden and child-hood experiences We already commented on theexternal impairment and chronic conditionssections A few comments are in order about severalof the other sections

Socio-demographicsBasic information about socio-demographic variablesndash such as age sex race education marital status andemployment status ndash is included in all communitysurveys For the most part though this informationis cross-sectional ndash it assesses the respondentrsquoscurrent status on these variables rather than his orher history This is fine for ascribed socio-economiccharacteristics (such as sex and race) which do notchange over time but it misses important informa-tion about the dynamics of achieved statuses such asmarital status and employment status which changeover time This loss of information can be importantif dynamic information is relevant to mental healthGiven the focus of the WMH-CIDI diagnosticsections on lifetime course we feel that it is impor-tant to include dynamic information about achievedstatuses in the interview schedule As a result sepa-rate sections of the interview are devoted to therespondentrsquos history in each of the three main areasof achieved social status ndash employment marriageand childbearing

The details of the assessment differ across thethree sections but the basic approach is the same

We begin by asking about timing of initial entry intoroles (for example ages at first dating first marriagefirst employment first sexual intercourse firstbecoming pregnant (females) or causing a woman tobecome pregnant (males) first having an abortionfirst giving birth) We then ask about role history(for example age at onset and duration of eachmarriage age of each child stability of employmenthistory) Information about current role incumbency(for example current employment status currentmarital status number and ages of children whichchildren live with the respondent) which is thefocus of the assessment in most surveys is only thefinal part of the assessments The exception is Part Irespondents terminated before the Part II interviewwho are administered a brief socio-demographicbattery that focuses only on current status

The additional socio-demographic sections ndashchildhood demographics adult demographics andfinances ndash also obtain much more detail about theseareas than in most other surveys The section onchildhood demographics asks about age of parentswhen the respondent was born size of sib-ship andbirth order marital status of parents nativitynumber of generations in the country among nativesage at immigration among the foreign born countryof origins for people who were originally fromanother country native tongue education child-hood religion and religiosity urbanicity of childhoodresidence and stability of childhood residence Thesection on adult demographics asks about whetherparents are living or dead age and cause of death ofeach deceased parent race-ethnicity subjectivecloseness of racial-ethnic identification citizenshipreligious preference religiosity amount of timeduring adulthood when the respondent was in a jailor prison or correctional facility amount of timehomeless amount of time institutionalized in ahospital or nursing home and current subjectivesocial class position The section on finances asksboth about objective finances and subjective finan-cial stress With regard to objective financesinformation is obtained both on income and assetsIncome information is obtained for the entire house-hold broken down by income of the respondent therespondentrsquos spouse other family members incomefrom government assistance programmes and otherincome This disaggregation of income is very usefulin analyses of social class and mental illness where it

IJMPR 132 3rd v4 25604 1026 am Page 108


is possible to distinguish associations that might bedue to direct selection (respondent income) assorta-tive mating (spouse income) and other sources

Treatment and pharmacoepidemiologyTreatment like impairment is assessed in the WMH-CIDI both internally and externally In the internal assessment respondents who meetcriteria for a particular disorder are asked at the end ofthe diagnostic section whether they ever soughtprofessional treatment for that disorder and if so atwhat age they first sought this treatment They arealso asked if they ever obtained treatment that theyconsidered helpful for the disorder and if so howmany different professionals they had to see beforethey received helpful treatment Respondents who saythey never received helpful treatment are asked howmany professionals they ever saw for the disorder Thiskind of information when coupled with informationon age at onset of the disorder can be used to studypatterns and predictors of delays in initial treatmentcontact after first onset of a mental disorder

Analyses of this sort in epidemiological samplesconsistently find pervasive delays in initial treatmentcontact after first onset of a mental disorder that areinversely related to age at onset cohort and illnessseverity (Kessler Olfson and Berglund 1998 OlfsonKessler Berglund and Lin 1998) It is also unclearwhether patient reports of being helped would beconfirmed in objective evaluations Nonethelesspatient perception even if not entirely accurate hasto be considered an important dimension of treatmenteffectiveness that has not previously been seriouslyconsidered in psychiatric epidemiological surveys

In the external assessment of treatment theWMH-CIDI asks respondents about ever havingtreatment for problems with their emotions ormental health Questions about inpatient treatmentinclude asking about lifetime hospitalization age offirst hospitalization number of lifetime hospitaliza-tions amount of time spent in hospitals for theseproblems over the life course and hospitalization inthe past 12 months Questions about outpatienttreatment include asking about treatment from eachof a wide range of professionals For each type ofprofessional seen information is recorded on age offirst receiving treatment and age of most recenttreatment For those who received treatment in thepast 12 months information is obtained on number

of visits with each type of professional average dura-tion of time with the professional whether therespondent is still in treatment and if not whetherthe termination of treatment occurred because therespondent completed the course of treatment orquit Summary questions are then asked of all respon-dents who were in any type of 12-month treatmentabout all the money spent out of pocket for treat-ment with all professionals over that interval of timereasons for seeking treatment and reasons for termi-nating treatment among those who did terminateRespondents who did not receive any treatment inthe past 12 months are asked whether they ever feltthat they might need professional help for theiremotions or mental health in the past 12 months Ifso they are asked about reasons for not seekingprofessional help

A separate pharmacoepidemiology section asksabout the use of prescription and non-prescriptionmedications in the past 12 months for lsquoproblems withyour emotions nerves mental health substance useenergy concentration sleep or ability to cope withstressrsquo An exhaustive list of prescription medicationsis provided as a visual aid in answering these ques-tions Interviewers are instructed to use motivationalprobes to encourage respondents to think carefullyand exhaustively to list all medications taken duringthe recall period Interviewers are also instructed tohave respondents get their medicine bottles if theystill have them in order to copy down informationabout the name of the medicine and the recom-mended dose For each medicine taken questions arethen asked about number of days taken out of thepast 365 dose whether the medicine is taken underthe supervision of a doctor If not information iscollected about where the respondent obtains themedicine If taken under the supervision of a doctora question is asked whether the doctor is a psychia-trist some other mental health specialist a primarycare doctor or some other kind of doctor If takenunder professional supervision a question is askedhow about the respondent failed to take the medi-cine at the recommended dose and times Finallyquestions are asked about whether the medicine isstill being taken and if not reasons for terminatinguse including side effects

Non-specific psychological distress The Part II interview includes screening scales about

IJMPR 132 3rd v4 25604 1026 am Page 109


the frequency of non-specific psychological distressduring the 30 days before the interview and in theworst month of the past year These scales weredeveloped originally for use in the NHIS in order toscreen for Serious Mental Illness (SMI) (defined asany DSM anxiety or mood or psychotic disorder asso-ciated with a Global Assessment of Functioningscore less than 60) They have subsequently beenadopted not only by the NHIS but also by a numberof other large ongoing national government healthsurveys in the US as well as by the Australian andCanadian governments for use in their ongoing healthsurveys The 10 questions in the scales were selectedfrom a large item pool aimed at sensitively measuringthe first principal factor of non-specific psychologicaldistress which is consistently found in communitysurveys of distress in the clinically significant range ofits distribution (Furukawa Andrews Slade andKessler 2003)

A validation study carried out in a communitysample showed that the 10-question scale does anexcellent job of screening for SMI (Kessler BarkerColpe Epstein Gfroerer Hiripi Howes NormandManderscheid Walters and Zaslavsky 2003)However the sensitivity and specificity of the scalemight vary across populations making it useful toinclude the scale in WMH-CIDI surveys so that newcalibrations can be made across many different popu-lations When accurate rules of this sort exist thisbrief screening scale can be very useful as an inexpen-sive mental health needs surveillance tool in ongoinggeneral-purpose tracking surveys Calibration rulesand software for transforming scores on this scale intoindividual-level predicted probabilities of SMI andother global measures of disorder are being posted asthey are developed not only for the US but also for allcountries in the WMH Survey Initiative

Family burdenAlthough the questions on internal and externalimpairment do an excellent job of assessing the waysin which mental disorders affect the people who havethese disorders the effects on the families of thementally ill are ignored This is a major gap as thereis clear evidence that mental disorders can createenormous family burdens (Saunders 2003 TsangTam Chan and Chang 2003) In an effort to correctthis situation a separate section was developed forthe WMH-CIDI on family burden Unlike all other

sections of the interview this section treats therespondent as the lsquofamily memberrsquo whose burden isbeing assessed rather than as the person whosemental health is being assessed This was based onconcerns that respondents would be unable toprovide accurate informant information on theburdens their disorders imposed on their loved onesAn additional virtue of this method is that itprovides an easy way to integrate information onburden into population estimates something thatwould be extremely difficult to do without enumer-ating networks if the respondent was treated as thefocal respondent whose illness affected manydifferent family members rather than as a representa-tive family member who might be burdened by theillness of any number of family members

The logic of the section requires us to begin bydefining a network of first-degree relatives This isdone by asking the respondent how many living first-degree relatives he has separately reporting thenumber of parents siblings children and whether ornot he has a spouse The respondent is then told thatwe have a few questions about the health problems ofthese individuals After enumerating the network inthis way the respondent is asked if any of these indi-viduals has any of 12 serious health problems Theseinclude the following in the order they are askedcancer serious heart problems a serious memoryproblem like senility or dementia mental retarda-tion a permanent physical disability like blindnessor paralysis any other serious chronic physicalillness alcohol or drug problems serious depressionserious anxiety schizophrenia or psychosis manic-depression or any other serious chronic mentalillness Note that the threshold of problems is sethigh (for example lsquoserious depressionrsquo rather thanlsquodepressionrsquo) and that open-ended questions areincluded about other serious chronic physical andmental illnesses at the end of the lists of explicitphysical and mental disorders The open ends are putat the end so that the earlier explicit disorders canprovide a context for defining the severity level wewant respondents to be thinking about whenresponding to the open-ended questions

Respondents who report having any first-degreerelatives with any of the 12 health problems are thenasked a series of questions about how these problemstaken as a whole affect the life of the respondentIncluded here are questions about the respondent

IJMPR 132 3rd v4 25604 1026 am Page 110


helping to wash or dress the ill person helping withpractical things (for example paperwork houseworklocal transportation and taking medications) andspending more time with the ill person(s) or givingthem more emotional support than they would if theillness(es) were not present The respondent thenestimates the number of hours per week he or shespends doing things related to the health problems ofthese family members and the amount of money hespends per month because of these problemsEmotional effects of the problems on the respondentare then briefly assessed (for example extent ofworry anxiety depression and embarrassmentcaused by the problems) Finally the SheehanDisability Scales are used to have the respondent ratethe extent to which family member health problemsinterfere with his or her functioning in the same fourareas of functioning used in the assessment ofinternal impairment (household duties employmentsocial life and close personal relationships)

The use of the Sheehan scales makes it possible tocompare the effects on functioning of the respon-dentrsquos own illnesses compared to the illnesses offamily members More generally by creating 48separate variables (for each of the 12 illnesses foreach of the four types of first-degree relatives usingcounts to deal with the situation where more thanone family member of a given type has a particulartype of illness) regression analysis can be used toexamine the relative effects of different types ofillnesses on the various dimensions of family burdenincluded in the assessment It is also possible usingthis same analysis approach to study whether partic-ular aspects of burden are greater for female thanmale focal respondents exposed to the same profileof family illness to examine how burden changeswhen the structural relationship between the focalrespondent and the ill person changes (for examplethe illness of a child versus of a parent) and to eval-uate the effects of network illness comorbidity onrespondent burden As far as we are aware none ofthese has up to now been the subject of systematicanalysis across a wide range of illness categoriesconsidered together in a large-scale communitysurvey although each of them has long been studiedin small-scale focused studies of patients and theirfamilies (for example Chakrabarti Kulhara andVerma 1993 Rupp and Keith 1993) The inclusionof the family burden section in the WMH-CIDI

makes it possible to carry out this systematic kind ofanalysis

Childhood experiencesThe final section in the WMH-CIDI that we want tomention is the section on childhood experiencesThe WMH collaborators agreed early on that theWMH surveys were uniquely positioned to study the lifetime effects of traumatic life experiences witha special emphasis on the long-term effects of child-hood adversities As a result a fairly extensive seriesof questions is included in the WMH-CIDI aboutchildhood experiences The questions about child-hood traumatic events are placed in the traumachecklist within the PTSD section The remainingquestions are included in a separate section on child-hood experiences This section begins by askingwhether the respondent lived with both of his or herbiological parents until age 16 and if not to explainhis or her living situation up to that age The natureand age at each important transition obtained inresponse to this question are recorded for such eventsas death of a parent parental divorce adoption andthe like Respondents who report living with bothparents up to age 16 are then asked whether a parentwas ever away from home for six months or longerdue to such things as hospitalization imprisonmentor military service The respondentrsquos age at and theduration of each such event are recorded A ques-tion is then asked about whether the respondent wasever away from home for 6 months or longer due tosuch things as hospitalization boarding school fostercare or residential treatment The respondentrsquos ageat and the duration of each such event are recorded

The remainder of the section focuses on respon-dent reports about their mothers and fathers or inthe absence of a mother or father during their child-hood the man and woman who served as theequivalents of mother and father Focusing on themain parent figure respondents are asked theirbiological relationship to this individual (forexample grandfather or step-father) as well as thecloseness and warmth of their interpersonal relation-ships during the years the respondent was growingup A modified version of the Parental BondingInstrument (Parker 1989) is used to classify parent-child relationships as either authoritarianauthoritative overprotective or neglectful A modi-fied version of the Conflict Tactics Scale (Straus

IJMPR 132 3rd v4 25604 1026 am Page 111


Hamby Finkelhor Moore and Runyan 1998) is usedto assess the frequency and intensity of parentalviolence towards the respondent during the respon-dentrsquos childhood Questions are also asked aboutneglect and sexual abuse Information is thenobtained on parent education employment statusoccupation if they were employed and the stabilityof their employment during the respondentrsquos child-hood Finally a modified version of the FamilyHistory Research Diagnostic Criteria Interview(Andreasen Endicott Spitzer and Winokur 1977) isused to assess parental psychopathology during therespondentrsquos childhood Separate assessments aremade here of parental depression panic disorderGAD substance-use disorder and antisocial person-ality disorder

Overview and future directions As noted in the introduction the CIDI was origi-nally developed by WHO to be a tool that could helpcoordinate the efforts of psychiatric epidemiologistsaround the world to carry out community surveys inwhich results could be directly compared and cumu-lated by virtue of using the same instrument TheWMH-CIDI continues in this tradition by refiningthe diagnostic assessments in the original CIDI andby adding sections that examine risk factors conse-quences and treatment Like the original CIDI theWMH-CIDI was designed to generate diagnosesusing the definitions and criteria of both the ICDand DSM systems (ICD-10 and DSM-IV) In addi-tion like the original CIDI the WMH-CIDI hasbeen translated into a number of languages using thestandard WHO translation and back-translationprotocol

An important WMH-CIDI development is theestablishment of an explicit protocol for modifyingthe instrument In the past reluctance on the part ofthe WHO CIDI Advisory Committee (WHO CIDI-AC) to work with investigators who proposedmodifications led to idiosyncratic changes made byindividual users in different surveys that reducedcomparability across studies The new protocol formodifying the WMH-CIDI calls for users who wantto modify diagnostic questions to include both theoriginal WMH-CIDI questions and the proposednew questions in their modified version of the instru-ment and to carry out blind clinical follow-upinterviews in a stratified probability sample of

concordant and discordant cases in order to evaluatewhether the new questions increase consistency ofWMH-CIDI diagnoses with clinical diagnoses Incases where the old and new questions cannot logi-cally be included in the same instrument a splitballot approach is stipulated in which random sub-samples receive one or the other in the same study

Stipulations for the design instrumentation andquality control of WMH-CIDI clinical reappraisalstudies have been established to guarantee endorse-ment of results by the WHO CIDI-AC Suggestionsfor such methodological studies and results of thesestudies will be posted on the WMH-CIDI Web pagealong with author attributions Replication of posi-tive results in a second endorsed methodologicalstudy will lead to proposed changes being adopted inthe next revision of the WMH-CIDI A similarsystem of making proposed modifications and expan-sions of the CIDI in ways that do not change the diagnostic questions will also be posted on theWMH-CIDI Web page in order to create a library ofpotentially useful alternative questions for futureusers Included here for example might be expandedquestions about childhood adversity a new sectionon coping or more elaborate questions about thenature of specific fears aimed at subtyping specificphobias As with other proposed modifications andexpansions of the instrument author attributionswill be included with each of these postings

AcknowledgementsThis paper is a report of the National Comorbidity SurveyReplication (NCS-R) The NCS-R is supported by theNational Institute of Mental Health (NIMH U01-MH60220) with supplemental support from the NationalInstitute of Drug Abuse (NIDA) the Substance Abuseand Mental Health Services Administration (SAMHSA)the Robert Wood Johnson Foundation (RWJF Grant044708) and the John W Alden Trust In addition tothese sources the creation of the WMH-CIDI wassupported by unrestricted educational grants from the JohnD and Catherine T MacArthur FoundationGlaxoSmithKline Eli Lilly and Company and PfizerCollaborating NCS-R investigators include Ronald CKessler (Principal Investigator Harvard Medical School)Kathleen Merikangas (Co-Principal Investigator NIMH)James Anthony (Michigan State University) WilliamEaton (The Johns Hopkins University) Meyer Glantz(NIDA) Doreen Koretz (Harvard University) JaneMcLeod (Indiana University) Mark Olfson (ColumbiaUniversity College of Physicians and Surgeons) Harold

IJMPR 132 3rd v4 25604 1026 am Page 112


Pincus (University of Pittsburgh) Greg Simon (GroupHealth Cooperative) Michael Von Korff (Group HealthCooperative) Philip Wang (Harvard Medical School)Kenneth Wells (UCLA) Elaine Wethington (CornellUniversity) and Hans-Ulrich Wittchen (Institute ofClinical Psychology Technical University Dresden andMax Planck Institute of Psychiatry) The authors appre-ciate the helpful comments on earlier drafts of RobertBelli Beth-Ellen Pennell Norbert Schwarz and Hans-Ulrich Wittchen Portions of this paper were previouslypublished in Kessler et al (2000) and are used here with thepermission of the publisher Lawrence Erlbaum AssociatesA complete list of NCS publications and the full text of allNCS-R instruments can be found at httpwwwhcpmedharvardeduncs Send correspondence to NCShcpmedharvardedu

The development of the WMH-CIDI depended on theefforts of a great many individuals Clinical research expertswho worked with us on specific diagnostic sections includethe following Jules Angst (major depression mania)Gavin Andrews (anxiety disorders) James Anthony(substance disorders) Naomi Breslau (post-traumaticstress disorder) Emil Coccaro (intermittent explosivedisorder) Ian Falloon (non-affective psychosis) AbbyFyer (agoraphobia specific phobia) Josep Maria Haro(family burden) Richard Heimberg (social phobia) DavidKatzelnick (obsessive-compulsive disorder) Martin Keller(generalized anxiety disorder) Mark Lenzenweger (person-ality disorders) Armand Lorenger (personality disorders)Bruce Lydiard (irritable bowel syndrome) Susan McElroy(intermittent explosive disorder) Kathleen Merikangas(major depression mania nicotine dependence) DanMroczek (non-specific psychological distress) Tom Roth(sleep disorders) John Rush (major depresson) MarcSchuckit (substance use disorders) Kathy Shear (panicdisorder separation anxiety disorder) Derrick Silove (sepa-ration anxiety disorder) Dan Stein (post-traumatic stressdisorder obsessive-compulsive disorder) Mauricio Tohen(bipolar disorder) Michael von Korff (chronic paindisorder) Lynn Wallisch (pathological gambling) PhilipWang (services pharmacoepidemiology) Ken Winters(pathological gambling) and Hans-Ulrich Wittchen(anxiety disorders) Survey methodology experts whoworked with us on the various question wording studies andrefinements discussed in the first half of the paper includeRobert Belli Charles Cannell Barbel Knauper Beth-EllenPennell and Norbert Schwarz The WMH-CIDI built ofthe pioneering work of Lee Robins in developing theDiagnostic Interview Schedule and the original CIDI andof Hans-Ulrich Wittchen in developing a number of impor-tant CIDI innovations in his M-CIDI Neither the originalDIS nor the CIDI and its extensions would have beenpossible without the vision of Darrel Regier who launchedthe programme of research that led to the DIS and theCIDI and subsequently the WMH-CIDI

ReferencesAguilar-Gaxiola S Alegria M Andrade L Bijl RV

Caraveo-Anduaga JJ DeWit DJ Kolody B Kessler RCUumlstuumln TB Vega WA Wittchen HU The InternationalConsortium in Psychiatric Epidemiology In EDragomireckaacute A Palcovaacute and H Papezovaacute eds SocialPsychiatry in Changing Times Prague Czech RepublicPrague Psychiatric Center 2000

Alegria M Kessler RC Bijl R Lin E Heeringa S TakeuchiDT Kolody B Comparing mental health service usedata across countries In G Andrews ed Unmet Need inMental Health Service Delivery Cambridge UnitedKingdom Cambridge University Press 2000 97ndash118

Andreasen NC Endicott J Spitzer RL Winokur G Thefamily history method using diagnostic criteriaReliability and validity Arch Gen Psychiatry 1977 341229ndash35

Angst J Merikangas K Scheidegger P Wicki W Recurrentbrief depression a new subtype of affective disorder JAffect Dis (Amsterdam) 1990 19 87ndash98

Belli RF Color blend retrievals compromise memories ordeliberate compromise responses Mem Cognit 198816 314ndash26

Belson WA The Design and Understanding of SurveyQuestions Aldershot Gower 1981

Benjamin J Ebstein RP Lesch KP Genes for personalitytraits implications for psychopathology Int JNeuropsychopharmacol 1998 1 153ndash68

Berardi D Berti Ceroni G Leggieri G Rucci P Uumlstuumln TBFerrari G Mental physical and functional status inprimary care attenders Int J Psychiatry Med 1999 29133ndash48

Bijl RV de Graaf R Hiripi E Kessler RC Kohn R OffordDR Uumlstuumln TB Vicente B Vollebergh WA Walters EEWittchen HU The prevalence of treated and untreatedmental disorders in five countries Health Aff(Millwood) 2003 22 122ndash33

Bijl RV Ravelli A Current and residual functionaldisability associated with psychopathology findingsfrom the Netherlands Mental Health Survey andIncidence Study (NEMESIS) Psychol Med 2000 30657ndash68

Breslau N Davis GC DSM-III generalized anxietydisorder an empirical investigation of more stringentcriteria Psychiatry Res 1985 15 231ndash8

Brewer WF What is autobiographical memory In DCRubin ed Autobiographical Memory New YorkCambridge University Press 1986 25ndash49

Broadhead WE Blazer DG George LK Tse CKDepression disability days and days lost from work in aprospective epidemiologic survey JAMA 1990 2642524ndash8

Brown GW Life events and measurement In GW BrownTO Harris eds Life Events and Illness New YorkGuilford Press 1989 3ndash45

IJMPR 132 3rd v4 25604 1026 am Page 113


Brugha TS Bebbington PE Jenkins R A difference thatmatters comparisons of structured and semi-structuredpsychiatric diagnostic interviews in the general popula-tion Psychol Med 1999 29 1013ndash20

Brugha TS Jenkins R Taub N Meltzer H Bebbington PEA general population comparison of the CompositeInternational Diagnostic Interview (CIDI) and theSchedules for Clinical Assessment in Neuropsychiatry(SCAN) Psychol Med 2001 31 1001ndash13

Cannell CF Fowler FJ Jr Marquis KH The influence ofinterviewer and respondent psychological and behav-ioral variables on the reporting in household inter-views Vital Health Stat 1 1968 2 1ndash65

Cannell CF Miller PV Oksenberg L Research on inter-viewing techniques In S Leinhardt ed SociologicalMethodology San Francisco Jossey-Bass 1981389ndash437

Centers R A laboratory adaptation of the conversationalprocedure for the conditioning of verbal operantsJournal of Abnormal and Social Psychology 1964 67334ndash9

Chakrabarti S Kulhara P Verma SK The pattern ofburden in families of neurotic patients Soc PsychiatryPsychiatr Epidemiol 1993 28 172ndash7

Clark HH Schober MF Asking questions and influencinganswers In JM Tanur ed Questions about QuestionsInquiries into the Cognitive Bases of Surveys NewYork Russell Sage Foundation 1992 15ndash48

Cohen J A coefficient of agreement for nominal scalesEducational and Psychological Measurement 1960 2037ndash46

Coryell W Scheftner W Keller M Endicott J Maser J andKlerman GL The enduring psychosocial consequencesof mania and depression Am J Psychiatry 1993 150720ndash7

Dewa CS Lin E Chronic physical illness psychiatricdisorder and disability in the workplace Soc Sci Med2000 51 41ndash50

Eaton WW Badawi M Melton B Prodromes and precur-sors epidemiologic data for primary prevention ofdisorders with slow onset Am J Psychiatry 1995 152967ndash72

Eaton WW Kessler RC Wittchen HU Magee WJ Panicand panic disorder in the United States Am JPsychiatry 1994 151 413ndash20

Eaton WW Neufeld K Chen LS Cai G A comparison of self-report and clinical diagnostic interviews fordepression diagnostic interview schedule for clinicalassessment in neuropsychiatry in the BaltimoreEpidemiologic Catchment Area follow-up Arch GenPsychiatry 2000 217ndash22

Edwards WS Winn DM Kurlantzick V Sheridan S BerkML Retchin S Collins JG Evaluation of NationalHealth Interview Survey Diagnostic ReportingNational Center for Health Statistics Vital Health Stat2 1994 120 1ndash116

Furukawa TA Andrews G Slade T Kessler RC Theperformance of the K6 and K10 screening scales forpsychological distress in the Australian NationalSurvey of Mental Health and Well-Being Psychol Med2003 33 357ndash62

Gilman S Disease and Representation Images of Illnessfrom Madness to AIDS Ithaca NY Cornell UniversityPress 1988

Gross R Bentur N Elhayany A Sherf M Epstein L Thevalidity of self-reports on chronic disease characteris-tics of underreporters and implications for the planningof services Public Health Rev 1996 24 167ndash82

Halabi S Zurayk H Awaida R Darwish M Saab BReliability and validity of self and proxy reporting ofmorbidity data a case study from Beirut Lebanon Int JEpidemiol 1992 21 607ndash12

Hays RD Wells KB Sherbourne CD Rogers W SpritzerK Functioning and well-being outcomes of patientswith depression compared with chronic general medicalillnesses Arch Gen Psychiatry 1995 52 11ndash19

Heliovaara M Aromaa A Klaukka T Knekt P JoukamaaM Impivaara O Reliability and validity of interviewdata on chronic diseases The Mini-Finland HealthSurvey J Clin Epidemiol 1993 46 181ndash91

Jobe JB White AA Kelley CL Mingay DL Sanchez MJLoftus EF Recall strategies and memory for health carevisits Milbank Q 1990 68 171ndash89

Judd LL Paulus MP Wells KB Rapaport MNSocioeconomic burden of subsyndromal depressivesymptoms and major depression in a sample of thegeneral population Am J Psychiatry 1996 1531411ndash17

Kempen GI Ormel J Brilman EI Relyveld J Adaptiveresponses among Dutch elderly the impact of eightchronic conditions on health-related quality of life AmJ Public Health 1997 87 38ndash44

Kempen GI Sanderman R Miedema I Mayboom-de JBOrmel J Functional decline after congestive heartfailure and acute myocardial infarction and the impactof psychological attributes A prospective study QualityLife Res 2000 9 439ndash50

Kendell RE Five criteria for an improved taxonomy ofmental disorders In JE Helzer JJ Hudziak eds DefiningPsychopathology in the 21st Century DSM-V andBeyond Washington DC American PsychiatricPublishing 2002 3ndash17

Kessler R The World Health Organization InternationalConsortium in Psychiatric Epidemiology (ICPE) Initialwork and future directions ndash the NAPE lecture 1998Acta Psychiatr Scand 1999 99 2ndash9

Kessler RC Barber C Beck A Berglund PA Cleary PDMcKenas D Pronk N Simon G Stang P Uumlstuumln TBWang PS The World Health Organization Health andWork Performance Questionnaire (HPQ) J OccupEnviron Med 2003 45 156ndash74

Kessler RC Barker PR Colpe LJ Epstein A Gfroerer JC

IJMPR 132 3rd v4 25604 1026 am Page 114


Hiripi E Howes MJ Normand S-LT Manderscheid RWWalters EE Zaslavsky AM Screening for seriousmental illness in the general population Arch GenPsychiatry 2003 60 184ndash9

Kessler RC Greenberg PE Mickelson KD Meneades LMWang PS The effects of chronic medical conditions onwork loss and work cutback J Occup Environ Med2001 43 218ndash25

Kessler RC Merikangas K The National ComorbiditySurvey Replication (NCS-R) International Journal ofMethods in Psychiatric Research 2004 (this issue)

Kessler RC Mroczek DK and Belli RF Retrospective adultassessment of childhood psychopathology In D ShafferCP Lucas JE Richters eds Diagnostic Assessment inChild and Adolescent Psychopathology New YorkGuilford Press 1999 256ndash84

Kessler RC Olfson M Berglund PA Patterns and predic-tors of treatment contact after first onset of psychiatricdisorders Am J Psychiatry 1998 155 62ndash9

Kessler RC Uumlstuumln TB The World Health OrganizationWorld Mental Health 2000 Initiative HospitalManagement International 2000 195ndash6

Kessler RC Wittchen H-U Abelson JM McGonagle KSchwarz N Kendler KS Knauper B Zhao SMethodological studies of the Composite InternationalDiagnostic Interview (CIDI) in the US NationalComorbidity Survey International Journal of Methodsin Psychiatric Research 1998 7 33ndash55

Kessler RC Wittchen HU Abelson JM Zhao SMethodological issues in assessing psychiatric disorderwith self-reports In AA Stone JS Turrkan CABachrach JB Jobe HS Kurtzman VS Cain eds TheScience of Self-Report Implications for Research andPractice Mahwah NJ Lawrence Erlbaum Associates2000 229ndash25

Kriegsman DM Penninx BW van Eijk JT Boeke AJ DeegDJ Self-reports and general practitioner information onthe presence of chronic diseases in community dwellingelderly A study on the accuracy of patientsrsquo self-reportsand on determinants of inaccuracy J Clin Epidemiol1996 49 1407ndash17

Leon AC Olfson M Portera L Farber L Sheehan DVAssessing psychiatric impairment in primary care withthe Sheehan Disability Scale Int J Psychiatry Med1997 27 93ndash105

Leventhal H Nerenz D Steele DJ Illness representationsand coping with health threats In A Baum SE TaylorJE Singer eds Handbook of Psychology and Health 4Hillsdale NJ Erlbaum 1984 219ndash52

Lipton RB Dodick D Sadovsky R Kolodner K Endicott JHettiarachchi J Harrison W A self-administeredscreener for migraine in primary care The IDMigraine(TM) validation study Neurology 2003 61375ndash82

Loftus EF Palmer JC Reconstruction of automobiledestructions an example of the integration between

language and memory Journal of Verbal Language andVerbal Behavior 1974 13 585ndash9

Mackenbach JP Looman CW Van der Meer JBDifferences in the misreporting of chronic conditionsby level of education the effect on inequalities inprevalence rates Am J Public Health 1996 86 706ndash11

Maier W Gansicke M Freyberger HJ Linz M Heun RLecrubier Y Generalized anxiety disorder (ICD-10) inprimary care from a cross-cultural perspective a validdiagnostic entity Acta Psychiatr Scand 2000 10129ndash36

Markus H Zajonc RB The cognitive perspective in socialpsychology In G Lindzey E Aronson eds TheHandbook of Social Psychology New York RandomHouse 1985 137ndash230

Marlatt GA Task structure and the experimental modifi-cation of verbal behavior Psychol Bull 1972 78335ndash50

Marquis KH Cannell CF A Study of Interviewer-Respondent Interaction in the Urban EmploymentAnn Arbor MI Survey Research Center University ofMichigan 1969

Means B Loftus EF When personal history repeats itselfdecomposing memories for recurring events AppliedCognitive Psychology 1991 5 297ndash318

Mechanic D Is the prevalence of mental disorders a goodmeasure of the need for services Health Aff(Millwood) 2003 22 8ndash20

Menon A Judgments of behavioral frequencies memorysearch and retrieval strategies In N Schwartz SSudman eds Autobiographical Memory and theValidity of Retrospective Reports New York Springer-Verlag 1994 161ndash72

Merikangas KR Zhang H Avenevoli S Acharyya SNeuenschwander M Angst J Longitudinal trajectoriesof depression and anxiety in a prospective communitystudy the Zurich Cohort Study Arch Gen Psychiatry2003 60 993ndash1000

Miller PV Cannell CF Communicating measurementobjectives in the survey interview In DM Hirsch PVMiller FG Kline eds Strategies for CommunicationResearch 6 Beverly Hills CA Sage 1977

Murray CJL Lopez AD The Global Burden of Disease AComprehensive Assessment of Mortality and Disabilityfrom Diseases Injuries and Risk Factors in 1990 andProjected to 2020 Cambridge MA Harvard UniversityPress 1996

Narrow WE Rae DS Robins LN Regier DA Revisedprevalence estimates of mental disorders in the UnitedStates using a clinical significance criterion to recon-cile two surveysrsquo estimates Arch Gen Psychiatry 200259 115ndash23

National Advisory Mental Health Council Health carereform for Americans with severe mental illnessesreport of the National Advisory Mental HealthCouncil Am J Psychiatry 1993 150 1447ndash65

IJMPR 132 3rd v4 25604 1026 am Page 115


National Center for Health Statistics DoDS 2003National Health Interview Survey (NHIS) HyattsvilleMD US Department of Health and Human ServicesCDC 2003

Neeleman J Ormel J Bijl RV The distribution of psychi-atric and somatic ill-health associations with person-ality and socioeconomic status Psychosom Med 200163 239ndash47

Oksenberg L Cannell CF Kanton G New strategies forpretesting survey questions Journal of Official Statistics1991 7 349ndash65

Oksenberg L Vinokur A Cannell CF The effects ofinstructions commitment and feedback on reporting inpersonal interviews In CF Cannell L Oksenberg JMConverse eds Experiments in Interviewing TechniquesDHEW Publication No (HRA) 78-3204 WashingtonDC Department of Health Education and Welfare1979a 133ndash99

Oksenberg L Vinokur A Cannell CF Effects of commit-ment to being a good respondent on interview perfor-mance In CF Cannell L Oksenberg JM Converse edsExperiments in Interviewing Techniques DHEWPublication No (HRA) 78-3204 Washington DCDepartment of Health Education and Welfare 1979b74ndash108

Olfson M Kessler RC Berglund PA and Lin E Psychiatricdisorder onset and first treatment contact in the UnitedStates and Ontario Am J Psychiatry 1998 1551415ndash22

Ormel J Kempen GI Deeg DJ Brilman EI Van SonderenE Relyveld J Functioning well-being and healthperception in late middle-aged and older peoplecomparing the effects of depressive symptoms andchronic medical conditions J Am Geriatr Soc 1998 4639ndash48

Ormel J Von Korff N Uumlstuumln TB Pini S Korten AOldehinkel T Common mental disorders and disabilityacross cultures results from the WHO collaborativestudy on psychological problems in general health careJAMA 1994 272 1741ndash8

Parker G The Parental Bonding Instrument psychometricproperties reviewed Psychiatr Dev 1989 7 317ndash35

Pearson RW Ross M Dawes RM Personal recall and thelimits of retrospective questions in surveys In JM Tanured Questions about Questions Inquiries into theCognitive Bases of Surveys New York Russell SageFoundation 1992 65ndash94

Penninx BW Beekman AT Ormel J Kriegsman DMBoeke AJ Van Eijk JT Deeg DJ Psychological statusamong elderly people with chronic diseases does typeof disease play a part J Psychosom Res 1996 40521ndash34

Pincus HA Zarin DA First M lsquoClinical Significancersquo andDSM-IV Arch Gen Psychiatry 1998 55 1145

Preisig M Merikangas KR and Angst J Clinical signifi-cance and comorbidity of subthreshold depression and

anxiety in the community Acta Psychiatr Scand 2001104 96ndash103

Regier DA Community diagnosis counts [Commentary]Arch Gen Psychiatry 2000 57 223ndash4

Regier DA Kaelber CT Rae DS Farmer ME Knauper BKessler RC Norquist GS Limitations of diagnosticcriteria and assessment instruments for mental disor-ders Implications for research and policy Arch GenPsychiatry 1998 55 109ndash15

Regier DA Narrow WE Defining clinically significantpsychopathology with epidemiologic data In JE HelzerJJ Hudziak eds Defining Psychopathology in the 21stCentury DSM-V and Beyond Washington DCAmerican Psychiatric Publishing 2002 19ndash30

Rehm J Uumlstuumln TB Saxena S Nelson CB Chatterji S IvisF Adlaf E On the development and psychometrictesting of the WHO screening instrument to assessdisablement in the general population InternationalJournal of Methods in Psychiatric Research 1999 8110ndash23

Rhode P Lewinsohn P Seeley J Are people changed bythe experience of having an episode of depression Afurther test of the scar hypotheses J Abnorm Psychol1990 99 264ndash71

Robins LN Helzer JE Croughan JL Ratcliff KS NationalInstitute of Mental Health Diagnostic InterviewSchedule its history characteristics and validity ArchGen Psychiatry 1981 38 381ndash9

Robins LN Regier DA Psychiatric Disorders in AmericaThe Epidemiologic Catchment Area Study New YorkThe Free Press 1991

Robins JN Wing J Wittchen HU Helzer JE Babor TEBurke J Farmer A Jablenski A Pickens R Regier DASartorius N Towle LH The Composite InternationalDiagnostic Interview An epidemiologic instrumentsuitable for use in conjunction with different diagnosticsystems and in different cultures Arch Gen Psychiatry1988 45 1069ndash77

Rupp A Keith SJ The costs of schizophrenia Assessingthe burden Psychiatr Clin North Am 1993 16413ndash23

Rush AJ Gullion CM Basco MR Jarrett RB Trivedi MHThe Inventory of Depressive Symptomatology (IDS)psychometric properties Psychol Med 1996 26477ndash86

Saunders JC Families living with severe mental illness aliterature review Issues Ment Health Nurs 2003 24175ndash98

Schlesselman J Case-control Studies Design ConductAnalysis New York Oxford University Press 1982

Schwarz N Sudman S Autobiographical Memory and theValidity of Retrospective Reports New York SpringerVerlag 1994

Schwarz N Sudman S Answering QuestionsMethodology for Determining Cognitive andCommunicative Processes in Survey Research 1 edn

IJMPR 132 3rd v4 25604 1026 am Page 116


San Francisco Jossey-Bass Publishers 1996Shear MK Brown TA Barlow DH Money R Sholomskas

DE Woods SW Gorman JM Papp LA Multicentercollaborative panic disorder severity scale Am JPsychiatry 1997 154 1571ndash5

Sirken MG Herrmann DJ Schechter S Schwarz N TanurJM and Tourangeau R Cognition and Survey ResearchNew York John Wiley amp Sons 1999

Skelton JA Croyle RT Mental Representation in Healthand Illness New York Springer-Verlag 1991

Spitzer RL Diagnosis and need for treatment are not thesame Arch Gen Psychiatry 1998 55 120

Stewart A Greenfield S Hays RD Functional status andwell-being of patients with chronic conditions JAMA1989 262 907ndash13

Straus MA Hamby SL Finkelhor D Moore DW RunyanD Identification of child maltreatment with theParent-Child Conflict Tactics Scales development andpsychometric data for a national sample of Americanparents Child Abuse Negl 1998 22 249ndash70

Substance Abuse and Mental Health ServicesAdministration Final notice establishing definitionsfor (1) children with a serious emotional disturbanceand (2) adults with a serious mental illness Fed Regist1993 58 29422ndash5

Sudman S Bradburn N Schwarz N Thinking aboutAnswers The Application of Cognitive Processes toSurvey Methodology San Francisco Jossey-Bass 1996

Sullivan MD LaCroix AZ Russo JE Walker EADepression and self-reported physical health in patientswith coronary disease mediating and moderatingfactors Psychosom Med 2001 63 248ndash56

Tanur JM Questions about Questions Inquiries into theCognitive Bases of Surveys New York Russell SageFoundation 1992

Tsang HW Tam PK Chan F Chang WM Sources ofburdens on families of individuals with mental illnessInt J Rehabil Res 2003 26 123ndash30

Turner C Martin E Surveying Subjective PhenomenaNew York Russell Sage Foundation 1985

Tweed DL Depression-related impairment estimatingconcurrent and lingering effects Psychol Med 1993 23373ndash86

Uumlstuumln TB Chatterji S and Rehm J Limitations of diag-nostic paradigm it doesnrsquot explain lsquoneedrsquo Arch GenPsychiatry 1998 55 1145ndash8

van den Bos GA The burden of chronic disease in termsof disability use of health care and healthy lifeexpectancies Eur J Public Health 1995 5 29ndash34

Verbrugge LM and Patrick DL Seven chronic conditionstheir impact on US adultsrsquo activity levels and use ofmedical services Am J Public Health 1995 85 173ndash82

Vinokur A Oksenberg L and Cannell CF Effects of feed-back and reinforcement on the report of health infor-mation In CF Cannell L Oksenberg and JM Converseeds Experiments in Interviewing Techniques Ann

Arbor MI Survey Research Center University ofMichigan 1979

Wells KB Stewart A Hays RD Burnam M Rogers WDaniels M Berry S Greenfield S and Ware J The func-tioning and well-being of depressed patients resultsfrom the Medical Outcomes Study JAMA 1989 262914ndash19

WHO International Consortium in PsychiatricEpidemiology Cross-national comparisons of the preva-lences and correlates of mental disorders Bull WorldHealth Organ 2000 78 413ndash26

Wittchen H-U Reliability and validity studies of theWHO Composite International Diagnostic Interview(CIDI) a critical review J Psychiatr Res 1994 2857ndash84

Wittchen H-U Uumlstuumln TB and Kessler RC Diagnosingmental disorders in the community A difference thatmatters Psychol Med 1999 29 1021ndash7

World Health Organization Composite InternationalDiagnostic Interview Geneva Switzerland WorldHealth Organization 1990

World Health Organization WHO DisablementsAssessment Schedule II (WHO-DAS II) GenevaSwitzerland World Health Organization 1998

World Health Organization International Classificationof Functioning Disability and Health GenevaSwitzerland World Health Organization 2001

Zeiss A Lewinsohn P Enduring deficits after remission ofdepression a test of the lsquoscarrsquo hypotheses Behav ResTher 1988 26 151ndash8

Correspondence RC Kessler Department of HealthCare Policy Harvard Medical School 180 LongwoodAvenue Boston MA USA 02115Telephone (+1) 617-432-3587Fax (+1) 617-432-3588Email kesslerhcpmedharvardedu

Also

T Bedirhan Uumlstuumln Global Programme on Evidence forHealth Policy World Health Organization AvenueAppia 20 1211 Geneva 27 SwitzerlandTelephone (+41) 22-791-3609Fax (+41) 22-791-4885Email ustunbwhoch

Appendix examples of WMH-CIDI modifications

Question comprehension

Breaking down complex questions into less complex sub-questions

A good example of a complex question is the stem ques-tion for dysphoria in the depression section This is a very

IJMPR 132 3rd v4 25604 1026 am Page 117


important question because failure to endorse it or aparallel question about anhedonia leads to the respondentbeing skipped out of further questions about depressionThe question in the standard CIDI is as follows lsquoIn yourlifetime have you ever had two weeks or longer whennearly every day you felt sad empty or depressed for mostof the dayrsquo This is an exceedingly complex question as itasks about a cluster of emotions (sad empty or depressed)over a duration of time (2 weeks or longer) which itselfcan have internal variation in duration both across days(nearly every day) and within days (most of the day)Debriefing studies show that most respondents miss at leastone of these core components Therefore we decomposethe question in the WMH-CIDI We begin by asking abouta shorter period of time and omit mention of between-dayvariation in duration lsquoHave you ever in your life had aperiod lasting several days or longer when most of the dayyou felt sad empty or depressedrsquo Positive responses arethen followed by a duration question lsquoDid you ever have aperiod of this sort that lasted most of the day nearly everyday for two weeks or longerrsquo (The underlines are an indi-cation to interviewers to emphasize these words) Thisquestion is then followed by a within-day duration ques-tion lsquoThink of times lasting two weeks or longer whenthese problems with your mood were more severe andfrequent During those times did these feelings usually lastless than one hour between one and three hours betweenthree and five hours or more than five hoursrsquo The readermight think that this last question is unnecessary as thequestion before it asked about dysphoria lasting lsquomost ofthe dayrsquo However pilot studies of the WMH-CIDI showedclearly that this aspect of the question was the least likelyto be heard by respondents leading to quite a few respon-dents who endorsed this question reporting in a follow-upquestion that their low mood lasted only for an hour ortwo Based on this result the follow-up question aboutwithin-day duration was retained in the final WMH-CIDIA separate follow-up question about between-day duration(whether the dysphoria lasted every day nearly ever daymost days half the days or less than half the days over the2 weeks) was found not to be necessary as all pilot respon-dents responded lsquoeveryrsquo or lsquonearly every dayrsquo

Clarifying vaguely defined terms

A good example of a vaguely defined term is the singlestandard dichotomous CIDI question about role impair-ment that is used in every diagnostic section of the CIDIlsquoDid (SYNDROME) interfere with your life and activitiesa lotrsquo This is a critical question as it is used to opera-tionalize the impairment component of the DSM-IVrequirement that a syndrome cause clinically significantdistress or impairment to qualify as a disorder Yet both theword lsquointerferersquo and the words lsquoa lotrsquo are ambiguous

Furthermore the fact that the question is dichotomousand that it requires an intensity qualifier to be answeredpositively (it is not enough for the syndrome to interfere itmust interfere a lot) creates a source of confusion Thiswas clearly visible in pilot studies carried out by Kessler etal (2000) who found that respondents who endorsed thestandard CIDI question often responded lsquosomersquo or lsquoa littlersquoto a follow-up question that asked lsquoHow much did it inter-fere with these activities ndash would you say a lot some oronly a littlersquo

It might seem strange that a respondent who has justresponded lsquoyesrsquo to a question about whether there was lsquoalotrsquo of interference would characterize the interference asless intense in the very next question but the fact that thisfrequently occurs illustrates that survey respondents oftenfail to listen carefully to secondary clauses in dichoto-mously worded questions Recognizing this problem theWMH-CIDI converts the dichotomous yes-no questionabout a lot of interference into a dimensional questionthat focuses the respondentrsquos attention on the intensity ofinterference by asking lsquoHow much did [SYNDROME]interfere ndash not at all a little some a lot or extremelyrsquoA small split ballot experiment in a pilot study carried outby Kessler et al (2000) found that this wording changeresulted in a significantly lower proportion of peoplereporting that the interference was severe enough to becharacterized as lsquoa lotrsquo than in the standard dichotomousCIDI question

In addition to turning the dichotomous responsescale into a polychotomous scale the WMH-CIDI alsoexpands the standard CIDI interference question to makesure respondents broadly review all major areas in their lifebefore answering The fully modified question is lsquoHowmuch did [SYNDROME] interfere with either your workyour social life or your personal relationships ndash not at all alittle some a lot or extremelyrsquo In order to clarify theintensity level implied by the various response categoriesa follow-up question is then asked in the WMH-CIDI toall respondents who report any interference lsquoHow oftenwere you unable to carry out your daily activities becauseof [SYNDROME] ndash often sometimes rarely or neverrsquo

Clarifying questions about odd experiences that could benormalized

The standard version of CIDI introduces the psychosisquestions with the statement lsquoNow I want to ask youabout some ideas you might have had about other peoplersquoThe questioning then begins by asking about delusionsrather than about hallucinations The first question islsquoHave you ever believed people were spying on yoursquo Thisis followed by questions about believing that people arespying on you and talking about you behind your backEach time a positive response is given the interviewer

IJMPR 132 3rd v4 25604 1026 am Page 118


asks lsquoHow did you know this was happeningrsquo An open-ended response is recorded and rated for plausibility Agreat many people answer these questions positively thevast majority of whom give plausible answers This is notsurprising as the experiences asked about are all quitecommon The first genuinely odd experience isnrsquot askedabout until the fourth question in the series lsquoHave you everbelieved that you were being secretly tested or experi-mented onrsquo This is followed by two remaining questionsabout delusions that could be interpreted in plausibleterms whether someone ever lsquoplotted against yoursquo or lsquotriedto hurt you or poison yoursquo whether the respondent everthought that lsquosomeone you never met was in love with yoursquoand a question about whether the respondent ever wasconvinced that his or her spouse or partner was beingunfaithful even though they said this was not true

These are followed by questions about more bizarredelusions such as mind reading (for example lsquoHave youever believed that someone was reading your mindrsquo)thought control (for example lsquoHave you ever beenconvinced that you were under the control of some poweror force so that your actions and thoughts were not yourownrsquo) and being able to receive special messages throughthe mass media Finally the symptom assessment finisheswith questions about visual auditory olfactory and tactilehallucinations Most of these questions are purposefullyworded in a way that is designed to normalize them Forexample the auditory hallucinations question asks lsquoHaveyou more than once heard things other people couldnrsquothear such as a voicersquo This normalized phrasing leads toconfusion as a great many respondents in general popula-tion samples respond positively and then in response tothe follow-up question lsquoHow do you explain hearing thingsother people couldnrsquot hearrsquo respond by saying lsquoI have verygood hearingrsquo

The philosophy behind this approach to assessingpsychosis is that psychotics will be more willing to admittheir symptoms if these symptoms are normalized A greatdifficulty with this approach is that it generates an enor-mous number of false positives This substantially compli-cates the process of screening for psychosis and alsointroduces the strong possibility of errors in classifying falsepositives as cases based on misleading open-endedresponses In addition a question can be raised whethertrue psychotics particularly those with paranoid tenden-cies will be motivated positively or negatively to respondpositively to normalized questions that they might perceiveas trying to trick them into reporting their clearly abnormalexperiences The philosophy behind the WMH-CIDIapproach is the opposite to make it clear to respondentsthat we are asking about odd experiences to motivatereporting with an introduction that validates the experi-ences and points to the importance of learning more aboutthem and beginning the questioning with hallucinations

rather than delusions in order to reinforce the introductoryremarks about the questions being about odd behaviours

The WMH-CIDI introduction is as follows lsquoThenext questions are about unusual things like seeing visionsor hearing voices We believe that these things may bequite common but we donrsquot know for sure because previousresearch has not done a good job of asking about them Soplease take your time and think carefully before answeringrsquoOne can see in this introduction a number of the method-ological refinements discussed in the body of the paperclarification of the nature of the questions validation ofthe experiences motivation for honest reporting and facil-itation of serious memory search by legitimating therespondent not answering immediately in order to thinkThe questions themselves are worded in such a way as toavoid confusion Compare for example the WMH-CIDIquestion about auditory hallucinations with the standardCIDI question about the same symptom lsquoThe next ques-tion is about hearing voices that other people could nothear I donrsquot mean having good hearing but rather hearingthings that other people said did not exist like strangevoices coming from inside your head talking to you orabout you or voices coming out of the air when there wasno one around Did you ever hear voices in this wayrsquo

Contextual misunderstanding

Contextual misunderstanding is a type of misunder-standing that derives from the position of the question inthe flow of the interview An example was given thereabout confusion in a question regarding the duration ofphobic fears The standard question order in the CIDI wasmodified to correct this problem A number of similarsmall but important cases of a related sort were found andcorrected in developing the WMH-CIDI One additionalexample in asking about number of lifetime manicepisodes the standard CIDI comes directly after a questionabout the duration of the respondentrsquos longest lifetimemanic episode The number-of-episodes question thenfollows asking about the number of lsquothesersquo lifetimeepisodes Debriefing shows that the word lsquothesersquo in thecontext of this placement leads a number of respondentsincorrectly to believe that the interviewer is asking aboutepisodes that were equally as long as the longest lifetimeepisode resulting in an underestimate of the number oflifetime manic episodes A person who had say 30 manicepisodes only four of which went on as long as the longestepisode might respond lsquothreersquo rather than lsquo30rsquo to this ques-tion about number of episodes This problem was correctedin the WMH-CIDI by a change in question placement

Task comprehension

Survey respondents are often unaware that interviewerswant them to engage in active memory search in

IJMPR 132 3rd v4 25604 1026 am Page 119


answering complex questions Indeed it often occurs thatinvestigators who write survey questions without cognitivedebriefing pilot studies themselves fail to appreciate thecognitive complexity involved in answering some surveyquestions This is all the more true for interviewers whoin the absence of special training in the use of feedbackmethods aimed at stimulating thoughtful responding willbe more concerned with the accuracy of recording answersthan in the accuracy of the answers being given TheWMH-CIDI includes clarifications in a number of placesto make this task clear to respondents Earlier the paperdiscussed one of the most important places where this isdone in the life review section of the interview See thissection of the paper for the example

There are numerous other places in the instrumentwhere the same principle is used to remind the respondentto think carefully One of the most consistent of these is inthe important question about age of disorder onset whichappears in each diagnostic section of the interview Thestandard CIDI question asks lsquoHow old were you the firsttime you [HAD THE SYNDROME]rsquo We know fromdebriefing interviews that many respondents have a verydifficult time remembering their age of onset especially foraccretion disorders Therefore it is important both tomake it clear that we are looking for a serious memorysearch and to deal with the possibility that we may beasking for more than the respondent knows As describedin the body of the paper we deal with the first of theseissues in the WMH-CIDI by asking what is known in thesurvey methodology literature as a lsquoprequestrsquo a questiondesigned to create a clarifying context for the substantivequestion that follows it lsquoCan you remember your exact agethe very first time you [HAD THE SYNDROME]rsquo Eventhough a great many respondents answer lsquonorsquo this questionis extremely important in making the task clear to respon-dents We are interested in a precise answer which meansthat serious memory search will be required

It is also noteworthy that we try to be equally clearwhen we want estimates rather than precise informationThis is important because it is not possible to motivaterespondents to engage in active memory search throughoutan entire long interview Instead we need to pick andchoose the especially important questions where extraeffort is thought to be needed as well as to be clear both toourselves and to respondents when we are willing to settlefor approximations So for example questions concerninglifetime course of illness (for example lsquoAbout how manydifferent years in your life did you have [SYNDROME]rsquo)explicitly ask for approximations by using the word lsquoaboutrsquoIn addition interviewers are trained both to acceptapproximations as answers to these questions and to probefor rough estimates if respondents say they are unable toprovide exact responses In a similar way we use structuredresponse categories with prespecified ranges rather than

open-ended responses in cases where we ask estimationquestions that will be difficult to answer and we recognizethe inability to obtain accurate fine-grained responsesfrom most respondents

Motivation

Motivational instructions and commitment questions

We developed a statement at the beginning of the lifereview section to emphasize the importance of carefulresponse in order to encourage complete and accurateanswers As noted earlier a commitment question is used inconjunction with the motivational instructions to makesure that the respondent is aware of and acknowledges theimportance of responding carefully The statement andcommitment question are as follows lsquo[READ SLOWLY]The next questions are about health problems you may havehad at any time in your life It is important for the researchthat you think carefully before answering Are you willing todo thisrsquo In cases where the respondent does not answer lsquoyesrsquoto the commitment question the interviewer is instructedto offer to reschedule the interview for a time when therespondent is more able to give serious thought to the ques-tions If respondents persist in saying that they are unwillingto think carefully before answering the interviewer isinstructed to terminate the interview and to code therespondent as a refusal

Contingent reinforcement

Contingent reinforcement is an interviewer feedbackstrategy rather than a question-wording strategy Theinterviewer training manual for the WMH-CIDI (which isavailable only to researchers who participate in the officialWMH-CIDI training described earlier in the paper) focusesconsiderable attention on the use of this directive feedbackstrategy A variety of probes are provided to interviewers toreinforce apparent effort in providing thoughtful answersThese are very simple feedback responses like lsquothanksrsquo andlsquothatrsquos usefulrsquo which are administered whether the responseis positive or negative so long as effort appears to beinvested in providing a thoughtful answer Responses thatmight be thought to imply a value judgment like lsquogoodrsquo orlsquoexcellentrsquo are not used

A variety of probes are also provided to give nega-tive feedback for apparently superficial responsesSometimes these take the form of a follow-up question to aseemingly superficial negative response (for example lsquoareyou sure there was nothingrsquo) At other times they take theform of an injunction (for example lsquoplease take your timeand think carefullyrsquo) At still other times the probe cantake the form of an observation followed by a request (forexample lsquoYou answered that one awfully quickly Could

IJMPR 132 3rd v4 25604 1026 am Page 120


you take a minute to think hard and make sure you didnrsquotforget anythingrsquo) Our experience is that the long versionsof the negative probes are most useful early in the inter-view when respondents who give superficial answers arebeing trained to be more thoughtful Periodic short posi-tive feedback probes with the rare use of a short negativeprobe at the first signs of the recurrence of superficialresponding generally are used throughout the remainderof the interview


We use two broad strategies to deal with the realizationthat autobiographical memory has limits The main one isto accept these limits and revise our questions so as tosettle for less detail than we would ideally like to haveThe other is to push the limits of memory by decomposingquestions in ways that mimic successful memory searchstrategies and bound uncertainty We already gave exam-ples earlier in the appendix of the first of these two strate-gies As illustrated in these examples we always signal torespondents when we want approximations rather thanexact responses This is done either by building into thequestion an explicit reference to wanting an approxima-tion or by providing structured response categories inrough groupings that indicate the level of approximationwe are looking for in responses Perhaps the best exampleof this approach is the question series described earlier inthe appendix about age of disorder onset As noted in thatearlier description we begin this series with the prequestlsquoCan you remember your exact age the very first time you[HAD THE SYNDROME]rsquo Respondents who answer lsquoyesrsquoare dealt with easily by asking them to report this exactage Respondents who say lsquonorsquo in comparison are askedfor an estimate lsquoAbout how old were you the first timersquo

This same question series also illustrates the use ofthe second strategy used to deal with the limits of autobio-graphical memory decomposing questions and boundinguncertainty for respondents who vary in accuracy of recallAs information about age of onset is of great importancefor a variety of substantive research questions specialeffort was invested in pushing for the limits of memoryamong respondents who reported that they could notrecall the exact age when the syndrome first began This isdone using special probes to ask respondents to go back-

wards in time sequentially This requires flexibility on thepart of interviewers as a good many respondents whoanswer lsquonorsquo to the question about exact recall volunteer inconjunction with that answer that the syndrome has beengoing on lsquomy whole lifersquo or lsquoas long as I can rememberrsquo

This kind of inexact information is useful because itimplies a very early age of onset To confirm this thoughsuch responses are probed for clarification of an upperbound by asking lsquoWas it before you first started schoolrsquo Ifthe respondent answers lsquoyesrsquo the question sequence endswith a fairly narrow bounding of uncertainty in an earlyage range If however the respondent does not answerlsquoyesrsquo to this first probe (for example if the respondent sayslsquoI canrsquot really rememberrsquo or lsquoI donrsquot think sorsquo or somethingsimilar) interviewers then move up the age range incre-mentally until they find an interval of time at which therespondent feels secure in saying that the syndrome wasdefinitely in existence as of that point in the life course(for example lsquoWas it before you were a teenagerrsquo If notlsquoWas it before you turned 20rsquo and so forth) Our experi-ence in pilot studies which has been confirmed in WMHsurveys is that most uncertainty among respondents whobegin by saying that their inability to provide an exact ageis due to the syndrome going on lsquoas long as I canrememberrsquo can be bounded before the teenage years

Among respondents who do not volunteer acomment to the effect that the syndrome has been goingon lsquoas long as I can rememberrsquo in response to the exact agequestion in comparison the interviewer begins by probingfor such a response (lsquoDid it go on as long as you canrememberrsquo) If so then probing proceeds as if thisresponse was volunteered If respondents say that thesyndrome did not go on as long as they can remember incomparison debriefing studies show that recall is quitecertain for a fairly specific time in the life course but notfor a particular age Therefore the same series of probesmoving up the age range is used as described in the lastparagraph with a final request for a best estimate of theonset age once an upper bound age range is reachedResponses that are given in ranges (for example lsquoIt startedduring my teens but I canrsquot remember the exact agersquo) arerecorded initially as ranges and then analysed as the upperend of the range in order to provide a consistently conserv-ative lower bound estimate

IJMPR 132 3rd v4 25604 1026 am Page 121


(Robins and Regier 1991) The ECA was a landmarkcommunity-based survey of mental disorders carriedout in selected neighbourhoods in five US communi-ties The wide dissemination of ECA results inhigh-profile publications led to replications in othercountries as well as to the development of other struc-tured diagnostic interviews The most widely used ofthese instruments is the WHO CIDI (World HealthOrganization 1990) The CIDI is an expansion of theDIS that was developed under the auspices of WHOby an international task force under the supervisionof Lee Robins to address the problem that DIS diag-noses are exclusively based on the definitions andcriteria of the American Psychiatric Associationrsquos(APA) Diagnostic and Statistical Manual (DSM) ofMental Disorders (Robins Wing Wittchen HelzerBabor Burke Farmer Jablenski Pickens RegierSartorius and Towle 1988) The WHO was keen toexpand the DIS to generate diagnoses based on thedefinitions and criteria of the WHO InternationalClassification of Disease (ICD) This was especiallyimportant for cross-national comparative research asthe ICD system is the international standard diag-nostic system

The CIDI was designed to encourage communityepidemiological surveys in many countries around theworld To this end a multinational CIDI editorialcommittee translated and field-tested the instrumentin many different countries (Wittchen 1994) whileWHO encouraged researchers around the world tocarry out CIDI surveys beginning in 1990 when theCIDI was first made available These efforts weresuccessful as over a dozen large-scale CIDI surveys inas many countries were completed during the firsthalf of the 1990s The WHO created theInternational Consortium in Psychiatric Epi-demiology (ICPE) in 1997 to bring together andcompare results across these surveys (Kessler 1999)The ICPE has subsequently published a number ofuseful descriptive studies of cross-national similaritiesand differences in prevalence and socio-demographiccorrelates of mental disorders (for example Aguilar-Gaxiola Alegria Andrade Bijl Caraveo-AnduagaDeWit Kolody Kessler Uumlstuumln Vega and Wittchen2000 Alegria Kessler Bijl Lin Heeringa Takeuchiand Kolody 2000 Bijl de Graaf Hiripi KesslerKohn Offord Uumlstuumln Vicente Vollebergh Waltersand Wittchen 2003 WHO InternationalConsortium in Psychiatric Epidemiology 2000)

However the work of the ICPE with this first genera-tion of CIDI surveys was hampered by the fact thatcomparability among the surveys was limited to theassessment of mental disorders Measures of riskfactors consequences patterns and correlates oftreatment and treatment adequacy none of whichwere included in the CIDI were not assessed in aconsistent manner across the surveys

Recognizing the value of coordinating themeasurement of these broader areas of assessmentthe ICPE launched an initiative in 1997 to bringtogether the senior scientists in planned CIDIsurveys prior to the time their surveys were carriedout in order to coordinate measurement Within ashort period of time research groups in over a dozencountries joined this initiative The World HealthOrganization officially established the WHO WMHSurvey Initiative to coordinate this undertaking in1998 Since that time the number of participatingWMH countries has expanded to 28 with an antici-pated combined sample size of over 200000interviews The authors of the current paper are theco-directors of both the ICPE and the WMH SurveyInitiative as well as the principal developers of theWMH-CIDI the expanded version of the WHOCIDI that was created for use in the WMH surveys

An overview of the WMH-CIDIIn the course of expanding the CIDI to includebroader areas of assessment we also took the oppor-tunity to make the diagnostic sections of the CIDImore operational We expanded questions to breakdown critical criteria including the clinical signifi-cance criteria required in the DSM-IV system Weexpanded the diagnostic sections to include dimen-sional information along with the categoricalinformation that existed in previous CIDI versionsWe also expanded the number of disorders includedin the CIDI

The 41 sections in the WMH-CIDI are listed inTable 1 These are not in their order of assessmentThe first section is an introductory screening andlifetime review section the logic of which isdiscussed later in this article There are also 22 diag-nostic sections that assess mood disorders (twosections) anxiety disorders (seven sections)substance-use disorders (two sections) childhooddisorders (four sections) and other disorders (sevensections) Four additional sections assess various

IJMPR 132 3rd v4 25604 1026 am Page 94









II Disorders











IJMPR 132 3rd v4 25604 1026 am Page 95







IJMPR 132 3rd v4 25604 1026 am Page 96








IJMPR 132 3rd v4 25604 1026 am Page 97







IJMPR 132 3rd v4 25604 1026 am Page 98







IJMPR 132 3rd v4 25604 1026 am Page 99









IJMPR 132 3rd v4 25604 1026 am Page 100









IJMPR 132 3rd v4 25604 1026 am Page 101







IJMPR 132 3rd v4 25604 1026 am Page 102








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121









II Disorders











IJMPR 132 3rd v4 25604 1026 am Page 95







IJMPR 132 3rd v4 25604 1026 am Page 96








IJMPR 132 3rd v4 25604 1026 am Page 97







IJMPR 132 3rd v4 25604 1026 am Page 98







IJMPR 132 3rd v4 25604 1026 am Page 99









IJMPR 132 3rd v4 25604 1026 am Page 100









IJMPR 132 3rd v4 25604 1026 am Page 101







IJMPR 132 3rd v4 25604 1026 am Page 102








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121







IJMPR 132 3rd v4 25604 1026 am Page 96








IJMPR 132 3rd v4 25604 1026 am Page 97







IJMPR 132 3rd v4 25604 1026 am Page 98







IJMPR 132 3rd v4 25604 1026 am Page 99









IJMPR 132 3rd v4 25604 1026 am Page 100









IJMPR 132 3rd v4 25604 1026 am Page 101







IJMPR 132 3rd v4 25604 1026 am Page 102








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121








IJMPR 132 3rd v4 25604 1026 am Page 97







IJMPR 132 3rd v4 25604 1026 am Page 98







IJMPR 132 3rd v4 25604 1026 am Page 99









IJMPR 132 3rd v4 25604 1026 am Page 100









IJMPR 132 3rd v4 25604 1026 am Page 101







IJMPR 132 3rd v4 25604 1026 am Page 102








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121







IJMPR 132 3rd v4 25604 1026 am Page 98







IJMPR 132 3rd v4 25604 1026 am Page 99









IJMPR 132 3rd v4 25604 1026 am Page 100









IJMPR 132 3rd v4 25604 1026 am Page 101







IJMPR 132 3rd v4 25604 1026 am Page 102








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121







IJMPR 132 3rd v4 25604 1026 am Page 99









IJMPR 132 3rd v4 25604 1026 am Page 100









IJMPR 132 3rd v4 25604 1026 am Page 101







IJMPR 132 3rd v4 25604 1026 am Page 102








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121









IJMPR 132 3rd v4 25604 1026 am Page 100









IJMPR 132 3rd v4 25604 1026 am Page 101







IJMPR 132 3rd v4 25604 1026 am Page 102








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121









IJMPR 132 3rd v4 25604 1026 am Page 101







IJMPR 132 3rd v4 25604 1026 am Page 102








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121







IJMPR 132 3rd v4 25604 1026 am 2








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121








IJMPR 132 3rd v4 25604 1026 am Page 103








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am 0







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121








IJMPR 132 3rd v4 25604 1026 am Page 104







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am 0









IJMPR 132 3rd v4 25604 1026 am Page 121







IJMPR 132 3rd v4 25604 1026 am Page 105








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121








IJMPR 132 3rd v4 25604 1026 am Page 106








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121








IJMPR 132 3rd v4 25604 1026 am Page 107








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121








IJMPR 132 3rd v4 25604 1026 am Page 108









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121









IJMPR 132 3rd v4 25604 1026 am Page 109








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121








IJMPR 132 3rd v4 25604 1026 am Page 110







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121







IJMPR 132 3rd v4 25604 1026 am Page 111








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121








IJMPR 132 3rd v4 25604 1026 am Page 112



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121



















IJMPR 132 3rd v4 25604 1026 am Page 113






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121






























IJMPR 132 3rd v4 25604 1026 am Page 114





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121





























IJMPR 132 3rd v4 25604 1026 am Page 115






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121






























IJMPR 132 3rd v4 25604 1026 am Page 116






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121






























Also






IJMPR 132 3rd v4 25604 1026 am Page 117










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121










IJMPR 132 3rd v4 25604 1026 am Page 118









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121









Task comprehension


IJMPR 132 3rd v4 25604 1026 am Page 119






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121






Motivation






IJMPR 132 3rd v4 25604 1026 am Page 120









IJMPR 132 3rd v4 25604 1026 am Page 121









IJMPR 132 3rd v4 25604 1026 am Page 121

department of health care policy | health care policy - … · 2005-05-13 · international journal...

Documents