This article was downloaded by: [Eindhoven Technical University]On: 23 November 2014, At: 03:57Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Clinical and ExperimentalNeuropsychologyPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/ncen20

A Preliminary Psychometric Analysis ofa Computer-Assisted Administration ofthe Telephone Interview of CognitiveStatus-ModifiedJ. Galen Buckwalter , Valerie C. Crooks & Diana B. PetittiPublished online: 09 Aug 2010.

To cite this article: J. Galen Buckwalter , Valerie C. Crooks & Diana B. Petitti (2002) A PreliminaryPsychometric Analysis of a Computer-Assisted Administration of the Telephone Interview ofCognitive Status-Modified, Journal of Clinical and Experimental Neuropsychology, 24:2, 168-175

To link to this article: http://dx.doi.org/10.1076/jcen.24.2.168.994

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to orarising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

A Preliminary Psychometric Analysis of a Computer-AssistedAdministration of the Telephone Interview of Cognitive

Status-Modi®ed

J. Galen Buckwalter, Valerie C. Crooks, and Diana B. PetittiDepartment of Research and Evaluation, Southern California Permanente Medical Group, Pasadena, CA, USA

ABSTRACT

Most screening tests of cognitive functioning require face to face administration by trained examiners. Thislimits their utility in epidemiology and in primary care settings. Further, existing screening tests have notbeen developed using established psychometric principles. We adapted the Telephone Interview of CognitiveStatus-modi®ed (TICSm) for administration as a computer-assisted telephone interview (CATI). Wescreened 3,681 elderly women with the CATI version of the TICSm, using lay staff as part of a longitudinalstudy. A preliminary analysis of the psychometric properties of the TICSm indicated good internalconsistency. Test-retest reliability is needed to con®rm reliability. Further work remains to adequately judgethe validity of the TICSm including comparisons with well-standardized tests and assessment of itspredictive properties in identifying dementia. However, the CATI version of the TICSm appears to havepotential as a cost-effective means of testing cognitive performance.

The goal of neuropsychological assessment is to

quantify behavioral expressions of brain dysfunc-

tion (Lezak, 1995). Perhaps the most challenging

context within which neuropsychology attempts

to distinguish between pathological and non-

pathological conditions is among aging adults.

Given `normal' age-related declines in cognition

and accelerated rates of decline associated with

dementia, this distinction can be subtle. Compre-

hensive neuropsychological batteries, in conjunc-

tion with neurological evaluation, laboratory tests

and brain imaging, are the accepted methods for

establishing a diagnosis of dementia. However,

the costs of these procedures preclude their use in

large-scale studies of the epidemiology of demen-

tia and, increasingly, in primary care settings.

The most widely used strategy in both situa-

tions is to employ brief neuropsychological tests,

or cognitive screens, as a method of determining

who should receive a further diagnostic evalua-

tion. The Mini-Mental State examination (MMSE;

Folstein, Folstein, & McHugh, 1975) is the most

widely used instrument. However, the MMSE has

failed to demonstrate consistent predictive ability

among heterogeneous populations (Tombaugh &

McIntyre, 1992). The poor performance of

screening instruments relates, to some extent, to

the failure to develop such tests in accordance

with established psychometric criteria (Gifford &

Cummings, 1999). Further, the in-person format,

frequently with a physician administrator, com-

bined with the length of time required to

administer tests such as the MMSE limits their

usefulness in primary care settings (Tangalos

et al., 1996) and in epidemiological studies.

The Telephone Interview for Cognitive Status

(TICS) and the Telephone Interview for Cognitive

Status-modi®ed (TICSm) show high correlations

Address correspondence to: J. Galen Buckwalter, Ph.D., Southern California Kaiser Permanente Medical Group,100 S. Los Robles Avenue (2nd Floor), Pasadena, CA 91101, USA. Tel.: �1 626 564 5535. Fax: �1 626 564 3430.E-mail: galen.x.buckwalter@kp.orgAccepted for publication: July 10, 2001.

Journal of Clinical and Experimental Neuropsychology 1380-3395/02/2402-168$16.002002, Vol. 24, No. 2, pp. 168±175 # Swets & Zeitlinger

Dow

nloa

ded

by [

Ein

dhov

en T

echn

ical

Uni

vers

ity]

at 0

3:57

23

Nov

embe

r 20

14

with the MMSE (Brandt, Spencer, & Folstein,

1988; Plassman, Newman, Welsh, Helms, &

Breitner, 1994). We administered the TICSm in

a computer-assisted telephone interview (CATI)

format. While other groups have utilized some

aspects of CATI technology (Breitner et al., 1995;

Plassman et al., 2000), we speci®cally examined

if cognitive screening could be conducted reliably

over the telephone by lay staff. Further, we

provide a preliminary evaluation of the psycho-

metric properties of the CATI administered

TICS.

METHODS

InstrumentThe TICS was originally comprised of 11 items witha maximum score of 41. The instrument wasmodeled after the MMSE and included itemsfrequently used in bedside mental status examina-tions (Brandt et al., 1988). It assesses a variety ofcognitive domains including orientation, compre-hension, attention, naming, working memory, verbalabstraction, and immediate verbal memory. Amodi®cation of the TICS, the TICSm, eliminateditems dif®cult to verify in a survey format and addeda delayed verbal recall measure to increase sensi-tivity to early dementia (Welsh, Breitner, &Magruder-Habib, 1993). This modi®ed scale has23 questions, scored as 12 items, for a total of 50points. The TICSm served as the basis for the presentstudy.

Pilot testing using non-CATI administration wasconducted with 80 volunteers to gain experiencewith administration issues. The initial step in admin-istering the TICSm in a CATI format was to develop acomprehensive set of guidelines for administrationand scoring. Using the scoring system provided for theTICSm (Welsh et al., 1993), we developed compre-hensive procedures for administration. After elabora-tion of the procedures, a computer program wasdeveloped which displays each question, and allow-able prompts, on the screen. When the interviewerenters an acceptable response, the next question isdisplayed. The program requires all responses, corrector incorrect, be entered. While immediate scoring iscompleted using the standard algorithm, the digitalstorage of all responses allows for the development ofalternative scoring procedures.

Interviewers were trained to provide a standardi-zation code after completion of the test. The fol-lowing criteria were used: (1) Test was administered

according to protocol; (2) Minor breaks in stand-ardization requiring slight alteration in protocol suchas repetition of an item (e.g., distracting noiseoccurred outside, doorbell rang but was answered bysomeone else). This is when the participant'sattention is interrupted but the testing sessioncontinues. A code of 2 was also used for examiners'errors such as misstating a word; (3) Major breaks inprotocol where testing had to be interrupted for aperiod of time (e.g., participant answered thedoorbell or another phone call), indicating thetesting session is interrupted but the test iscompleted after the interruption; (4) Test began butcould not be completed due to an interruption. Asecond testing session was scheduled to administerthe entire test.

After completion of the test, the examiner alsoanswered the questions: `̀ Did the respondent appearto have any dif®culty hearing the questions?'' and`̀ Did the respondent appear to have any dif®cultyunderstanding the questions?'' with one of thefollowing codes: (1) Yes, great dif®culty; (2) Yes,some dif®culty; (3) Yes, a little dif®culty; (4) No,none at all. The participant's attitude during testingwas rated as (1) enthusiastic, (2) interested, (3)indifferent, (4) somewhat reluctant, or (5) veryreluctant. Reluctance included poor motivation aswell as such undesirable test taking behaviors ashostility, impulsivity and irritability. Examinerswere encouraged to provide narrative descriptionsof any unusual occurrences during testing, includingany indications that the participant received help orused any means other than pure cognitive strategiesto answer the questions, for example, using a paperand pencil. All narrative responses were reviewed.Dichotomous variables were created re¯ecting thepresence or absence of the following: receiving helpor cheating; language dif®culty; or a bad telephoneconnection.

IndividualsAll enrollees were participants in The Women'sMemory Study conducted by the Southern CaliforniaPermanente Medical Group and the University ofSouthern California. This study examines theassociation between the use of hormone replacementtherapy (HRT) and dementia status and cognitivefunctioning. The original sample of 6,918 women,all 75 years of age or older on July 31, 1998 andmembers of Kaiser Permanente for at least 7 years,were randomly selected from Kaiser Permanente'sdatabase based on whether they had used HRTduring the past 7 years. Of these women, 277 weredeceased prior to the time they could be contacted,99 were unable to speak English, and 243 weretoo debilitated to complete a phone interview. Five

COMPUTER-ASSISTED COGNITIVE SCREEN 169

Dow

nloa

ded

by [

Ein

dhov

en T

echn

ical

Uni

vers

ity]

at 0

3:57

23

Nov

embe

r 20

14

hundred twenty-four women were not reached due toincorrect phone numbers or because no one wascontacted after 10 attempts. An additional 2,094declined to participate. The sample for this studywas 3,681.

The average age of participants was 78.8 (SD�3.3) years. Self-reported education showed 489(13.3%) did not ®nish high school, 1,012 (27.6%)were high school graduates, 1,350 (36.7%) hadcompleted some college or trade school, and 822(22.4%) had completed at least a baccalaureatedegree. Eight women refused to provide thisinformation.

ProceduresProcedures were reviewed and approved by theInstitutional Review Boards at the Southern Cali-fornia Permanente Medical Group and the Univer-sity of Southern California. All women in theoriginal sample were sent a package that includeda letter informing them of the Women's MemoryStudy, an informational brochure and a postcard theycould return if they did not wish to participate. Ifthey did not return the postcard they received aphone call and were asked to complete a telephoneinterview. The interview included information ontheir medical, reproductive and lifestyle history. The®nal section was the TICSm.

Telephone marketing personnel, with no previousexperience in cognitive evaluation, conducted allinterviews. The interviewers received 8 hr of trainingfor the purpose of the study, the administration ofstructured interviews and standardized tests, andhow to interact effectively with elderly individuals.All interviewers were observed conducting mockinterviews prior to working with enrollees by agerontologist or a psychologist. All were randomlymonitored by the same professionals throughoutthe project. A marketing manager also monitoredinterviewers.

Data AnalysisThe preliminary psychometric properties of theCATI administered TICSm were evaluated byestimating reliability and considering limitedaspects of both construct and concurrent validity.This is part of an ongoing study of the TICSm withthe long-term goal to re®ne the TICSm as a brief testof cognition and as a screening test for dementia. Asadditional data becomes available we will exploretest-retest reliability, predictive validity in identify-ing demented individuals as well as individuals whobecome demented and construct validity as deter-mined by administration of other neuropsycholog-ical tests. The goal of this analysis was to determine

if the CATI administered version of the TICSmmeets the basic psychometric criteria needed towarrant further development and to provide a solidpsychometric underpinning for any future work.

Descriptive information for all items andresponse characteristics was generated. For eachitem we provide the average of each individual scoredivided by the maximum possible score. This indexallows for a relative comparison of item dif®culty.We also provide the correlation between each itemand the total TICSm score that has been correctedfor the item. This widely used discrimination indexindicates the variance an item shares with what allitems have in common (Nunnally & Bernstein, 1994,p. 305).

We used a modi®cation of split-half reliabilitydeveloped by Gorsuch (personal communication,2000). In this re®nement of traditional split-half reli-ability, we ranked items based on dif®culty and thenformed pairs of items of similar dif®culty. One itemfrom each pair was randomly assigned to each half toassure that the halves were of equal dif®culty. Aftercalculating the split-half reliability coef®cient, theSpearman±Brown formula was used to estimate thefull-length reliability from the split-half (Nunnally &Bernstein, 1994, p. 234). We also provide the widelyused coef®cient alpha.

Factor analysis was conducted to evaluate anaspect of construct validity. This is based on theassumption that cognition does operate in a unitarymanner but is comprised of distinct domains.Principal component factors were extracted, usingthe roots greater than one criterion to determine thenumber of factors. Factors were then orthogonallyrotated with the Varimax procedure. Factor loadingsof .4 and above were considered relevant in inter-preting the factor. Although it is recommended thatloadings be considered relevant if they are two timesthe value required for a correlation coef®cient to besigni®cant with the given sample size (Gorsuch,1983, p. 209), we chose a much higher level as aconservative approach given our large sample. Thefactor structure was interpreted based on theoreticalformulations of the cognitive domains underlyingthe items that loaded on each factor. As a measure ofconcurrent validity, total scores and factor scoreswere associated with age and education. Both areknown to have associations, to varying degrees, withmost cognitive domains, and valid measures ofglobal cognitive functioning can be expected tocorrelate with both.

We also planned two post hoc analyses to explorethe association between information unique to theCATI administered TICSm and cognitive perfor-mance. First we evaluated how requests for repeti-tion were associated with performance by creating a

170 J. GALEN BUCKWALTER ET AL.

Dow

nloa

ded

by [

Ein

dhov

en T

echn

ical

Uni

vers

ity]

at 0

3:57

23

Nov

embe

r 20

14

variable re¯ecting the total number of such requests.This variable was then correlated with the TICSmscore. We also explored the ef®cacy of using theTICSm to identify primacy (enhanced learning andrecall for material presented initially) and recencyeffects (enhanced learning and recall for materialpresented last) on the immediate and delayed wordlist by developing descriptive statistics for all wordsbased on order of presentation.

RESULTS

Descriptive InformationAnalysis of the narrative responses found that there

were language dif®culties noted for 2% of inter-

views, and poor telephone connections for 0.3%

of interviews. It was suspected that participants

received help or cheated in 0.7% of interviews.

The TICSm was administered according to

protocol with 3,506 participants (95.2%). Table 1

shows the performance on the test and the

percentages of participants with hearing and

understanding dif®culties and their attitudes about

testing. We report 95% con®dence intervals (CI)

for each mean.

The overall mean for the TICSm was 29.4

(SD�6.2: range 4±48). Mean item scores are

provided in Table 2. The most dif®cult items, as

identi®ed by the index of dif®culty, were Word

List Delayed Recall, Word List Immediate Recall

Table 1. Response Characteristics and TICSmPerformance.

Adherence to Protocol.

Group n (%) TICSm (95% CI)M

Protocol followed 3506 (95.2) 29.5 (29.3±29.7)Minor break 159 (4.3)Major break 15 (0.4) 27.6 (26.6±28.6)*Discontinued 1 (0.0)

HearingNo dif®culty 2699 (73.3) 30.2 (30.0±30.4)Little dif®culty 631 (17.1) 28.1 (27.6±28.6)Some dif®culty 56 (7.0) 26.1 (25.3±26.9)Great dif®culty 92 (2.6) 23.2 (21.8±24.6)

UnderstandingNo dif®culty 2843 (77.2) 30.7 (30.5±30.9)Little dif®culty 578 (15.7) 26.7 (26.2±27.1)Some dif®culty 91 (5.2) 22.1 (21.3±22.9)Great dif®culty 69 (1.9) 18.6 (17.0±20.3)

AttitudeEnthusiastic 2522 (68.5) 30.0 (29.8±30.2)Interested 098 (29.8) 28.2 (27.8±28.6)Indifferent 26 (0.7)Somewhat 34 (0.9) 25.6 (23.8±27.4)^

reluctantVery reluctant 0 (0.0)

Note. *Mean and CI for minor break, major break anddiscontinued combined.^ Mean and CI for indifferent and somewhat reluctantcombined.

Table 2. Descriptive Information.Telephone Interview of Cognitive Status.

Item Possible % Requesting M (SD) Dif®culty Item-Totalpoints repeat* index? correlation

Full name 2 1.3 1.99 (.06) 1.0 .02Date (5 items) 5 3.9 3.29 (.95) .66 .36Age and phone number 2 1.4 1.90 (.32) .95 .19Count backward from 20 to 1 2 2.3 1.80 (.49) .90 .2110 word immediate recall 10 9.9 3.89 (1.91) .39 .61Serial 7 subtractions 5 15.3 2.08 (1.81) .42 .37Naming (4 items) 4 25.8 3.73 (.56) .93 .30Repetition (2 items) 2 13.7 1.48 (.69) .74 .27President and Vice President 4 1.7 3.02 (1.12) .76 .35Tap the phone 5 times 2 9.2 1.89 (.37) .95 .20Identify opposites (2 items) 2 19.0 1.78 (.47) .89 .3410 word delayed recall 10 5.1 2.52 (2.04) .25 .58

Note. *% includes any request for one or more questions in a multiple part item.?Obtained score/possible score.

COMPUTER-ASSISTED COGNITIVE SCREEN 171

Dow

nloa

ded

by [

Ein

dhov

en T

echn

ical

Uni

vers

ity]

at 0

3:57

23

Nov

embe

r 20

14

and Serial 7's subtraction. The probability was

less than .5 for all three items. The next most

dif®cult items were Date, Repetition, President/

Vice President, and Opposites. Five items were

answered correctly by at least 90% of partic-

ipants: Count Backward, Age/Phone, Naming,

Tapping, and Full Name.

The item for which requests for repetition were

made most frequently was Naming, with Word

Opposites, Serial 7's and Repetition all requested

to be repeated by over 10% of participants.

When considering Item-Total correlations,

Immediate Recall and Delayed Recall of the

word list had coef®cients above .55. Full Name,

Age/Phone, Count Backward, Repetition and

Tapping had correlations below .3. Note that

these items also tended to be of least dif®culty and

had the smallest standard deviations.

Psychometric Properties of the CATIAdministered TICSmSplit-half reliability was .82, which when cor-

rected to estimate full-length reliability with the

Spearman±Brown formula was .90. Coef®cient

alpha was .70.

Three factors met the criterion for extraction

and accounted for 42% of the total variance. Items

that loaded on Factor 1 were Naming, Repetition,

Tapping and Opposites. Factor 2 was comprised

of Immediate and Delayed Recall and Factor 3 of

Date, Age/Phone, Serial 7's, and President/Vice

President with Full Name loading negatively. The

factors were labeled: (1) Language Processing

and Output; (2) Memory; and (3) Orientation and

Concentration (see Table 3 for all factor scores).

The TICSm was correlated with age at ÿ.18

and with education at .33. Given the limited range

of age and education, we provide means for age

and education groups in Table 4. Both show a

clear `dose response' association with the TICSm,

and with factor scores in expected directions.

Post Hoc AnalysesThere was a negative association between the

total number of requests for questions to be

Table 3. Construct Validity.Factor Loadings for the TICSm.

Item Language Memory Orientation/Processing Concentration

Name ± ± ÿ.40Date ± ± .56Age/Phone ± ± .49Count ± ± ±

BackwardImmediate ± 93 ±

RecallSerial 7's ± .50Naming .55 ± ±Repetition .57 ± ±President/ ± ± .59

Vice PresidentTapping .60 ± ±Opposites .53 ± ±Delayed ± .93 ±

RecallPercent of 13% 16% 13%

variance

Table 4. Concurrent Validity.Associations of TICSm Total Score and Factor Scores with Age and Education.

n TICSm Language Memory OrientationM (SD) Processing M (SD) and Concentration

M (SD) M (SD)

Age75±79 2475 30.0 (6.0) .02 (.96) .09 (.99) .04 (.98)80±84 952 28.6 (6.4) ÿ.03 (1.05) ÿ.12 (1.00) ÿ.03 (1.0)85� 254 26.3 (6.4) ÿ.11 (1.19) ÿ.45 (.86) ÿ.27 (1.16)

Education<HS Degree 489 24.6 (6.1) ÿ.51 (1.37) ÿ.42 (.83) ÿ.49 (1.13)HS graduate 1012 28.6 (5.8) ÿ.01 (.97) ÿ.10 (.96) ÿ.06 (.94)Some college 1350 30.5 (5.7) .11 (.87) .13 (1.03) .08 (.94)College graduate 822 31.4 (7.2) .14 (.87) .16 (1.09) .24 (.90)

172 J. GALEN BUCKWALTER ET AL.

Dow

nloa

ded

by [

Ein

dhov

en T

echn

ical

Uni

vers

ity]

at 0

3:57

23

Nov

embe

r 20

14

repeated and the TICSm (r�ÿ.30), that is more

requests to have items repeated were associated

with worse TICS performance.

Table 5 provides a description of response

characteristics to word list items. For the immediate

recall trial, the words recalled with the highest

frequency were numbers one, three, and nine

while on the delayed trial those recalled most

often were those presented in the ®rst, third, and

®fth position.

DISCUSSION

This study suggests that cognition can be assessed

reliably over the telephone using a computer-

assisted test administered by non-psychometrists.

Although there are encouraging aspects of both

the instrument's reliability and validity, this is a

work in progress and further work needs to be

done before it can be accepted as a psychome-

trically sound test of cognition or a screening

instrument.

The reliability of the instrument appears to be

good as measured by internal consistency proce-

dures. An argument can be made that the modi®ed

split-half procedure developed by Gorsuch is a

more accurate estimate of reliability than coef®-

cient alpha. Coef®cient alpha underestimates

reliability in all situations unless items are

selected based solely on item correlations. As

such, Cronbach's �, although widely used, is a

lower bound to reliability and not a reliability

coef®cient (Lord, 1980, p. 8). While under-

estimation is widely accepted as a conservative,

thus preferable stance, it is not preferable to ac-

curacy. However, either procedure only assesses

a limited aspect of reliability; internal consistency

and test-retest reliability will ultimately be

needed to assure reliability. Inter-examiner relia-

bility also remains to be evaluated.

The factor structure of the TICSm was

generally conceptually appealing, although the

negative loading of Full Name on Orientation and

Concentration highlights a compelling problem

with this item, notably its ease (99.7% of

participants received 2 points). The factors

identi®ed in our analysis do not replicate previous

factor solutions. The only factor clearly replicated

from a previous factor study, which used 19 item

scoring rather than the standard 12 items, is the

memory factor (Brandt et al., 1993). A further

concern with our factor solution is the relatively

small proportion of variance for which it accounts.

When the majority of the matrix variance is not

accounted for by factors that meet criteria for

extraction, a poor ®t is suggested that questions the

validity of the TICSm. This contention is further

supported by the fact that when a common factor

solution was employed (using multiple R2's as

estimates of communalities), only a single factor

emerged with an eigenvalue above .7 (unreported

data). Thus, the ability of the TICSm to assess

distinct cognitive domains is questionable. This

may indicate that distinct cognitive domains can

only be validly measured with comprehensive

neuropsychological batteries.

There is modest support for concurrent validity,

with clear associations found with age and edu-

cation. Both age and education are unequivocally

Table 5. Word List Characteristics.

Word Learning trial % Perseveration(s) Recall trial % Perseveration (s)% correct % correct

Cabin 64.7 1.8 51.8 0.5Pipe 31.7 0.7 21.6 0.5Elephant 58.3 1.0 47.6 0.8Chest 26.7 0.8 21.2 0.4Silk 29.7 0.8 24.7 0.3Theater 26.7 0.5 19.2 0.4Watch 28.4 0.8 18.9 0.2Whip 26.6 0.5 13.2 0.2Pillow 51.2 1.4 23.2 0.2Giant 45.4 1.0 10.2 0.8

COMPUTER-ASSISTED COGNITIVE SCREEN 173

Dow

nloa

ded

by [

Ein

dhov

en T

echn

ical

Uni

vers

ity]

at 0

3:57

23

Nov

embe

r 20

14

associated with screening tests of global cognitive

performance (see Tombaugh & McIntyre, 1992, for

a review of such associations with the MMSE) and

failure to ®nd such associations would question the

validity of the TICSm. The associations between

interruptions, ratings of hearing, understanding and

attitude may also be indicative of validity. However,

it is possible that the interviewers' ratings were

in¯uenced by poor performance.

Considering all currently available data, sup-

port for the validity of the TICSm is tentative.

Based on results of the factor analysis, it is

unclear if the TICSm can assess cognition in

predictable domains. This may be an inevitable

consequence of brie¯y testing cognitive function,

however the utility of validly measuring multiple

domains in the study of cognitive aging/dementia

is apparent. The TICSm does appear to be asso-

ciated with variables such as age and education in

predictable patterns. Given our current evidence

of good reliability as demonstrated by internal

consistency, further work assessing both relia-

bility and validity is warranted and will include

comparisons with established neuropsychological

tests and its ability to identify individuals with

con®rmed dementia.

While the correlation with education supports

the validity of the TICSm, it also highlights a key

methodological issue in the development of screen-

ing tests. Education has been consistently found to

correlate with scores on the MMSE (Crum,

Anthony, Bassett, & Folstein, 1993; Katzman

et al., 1988). Therefore, adjustment for education

is needed in effective screening tests. Norms

speci®c for education groups, and other relevant

demographic variables such as age and ethnicity,

remain to be developed for the CATI TICSm.

The item-total analyses of the CATI TICSm

suggests it may be possible to reduce the number

of items while maintaining, or improving, the

psychometric properties of the instrument. Item-

total correlations below .3 are not considered to

add substantially to the discrimination ability of a

scale (Nunnally & Bernstein, 1994, p. 305). Thus,

it may be possible to drop Full Name, Age/Phone,

Count Backward, Repetition and Tapping from

the instrument. However, given that item deletion

may alter the psychometric properties of other

items, this will need to be tested with participants.

We suggest that consideration should also be

given to scoring additional information that is

readily available utilizing the CATI administra-

tion, for example, primacy and recency on the

immediate and delayed word lists. Demented and

non-demented individuals may differ in their

ability to bene®t from the serial order of

presentation (Burkart, Heun, & Benkert, 1998).

Table 5 suggests that participants in this study

bene®ted from the primacy effect on both the

immediate and delayed recall while the recency

effect was apparent on the immediate trial only.

The current factor structure of the TICSm

contains a single factor that includes immediate

and delayed memory, as does the factor solution

provided by Brandt et al. (1993). An instrument

that separated learning from delayed recall would

have greater utility in the assessment of incipient

dementia, given the importance of identifying

de®cits in delayed memory for the diagnosis of

dementia (APA, 1994). Given different effects of

recency in the immediate and delayed trials,

inclusion of this level of information in the

scoring system may allow for the identi®cation of

more discrete cognitive constructs.

Our ®ndings regarding interruptions, ratings of

hearing, understanding and attitude all support the

use of CATI technology and lay examiners. These

variables related to cognitive performance in ex-

pected patterns. Given effects on performance of

interruptions and hearing dif®culty, which likely

do not re¯ect cognitive de®cits, objective criteria

for identifying situations and individuals where

the TICSm is not appropriate is needed.

Associations between requests for repetition

and performance are also revealing. Either

hearing dif®culties or comprehension dif®culties

could explain the relatively strong association

between the number of requests for repetition and

the TICSm score. We found that the number of

requests for repetition was similar based on

ratings of hearing and understanding problems

(data not shown). Distinguishing between hearing

and comprehension problems is an unresolved

issue in the use of the TICSm.

This study supports the incorporation of

computer technology into the administration

and scoring of telephone screens of cognition.

Nonprofessional staff can collect internally

174 J. GALEN BUCKWALTER ET AL.

Dow

nloa

ded

by [

Ein

dhov

en T

echn

ical

Uni

vers

ity]

at 0

3:57

23

Nov

embe

r 20

14

consistent data using this technique. This has

notable implications for the inclusion of cognitive

testing in large-scale epidemiological studies. The

validity of the existing test is not yet suf®cient nor

is our current assessment of validity. The TICS

needs to be used in conjunction with established

tests to further evaluate its construct validity.

Further re®nements following standard psycho-

metric procedures for test development may

enhance construct validity. There is information

collected from CATI administration of the TICSm

that may allow for the evaluation of a broader

range of relevant cognitive domains. Exploration

of these options for the CATI TICSm is ongoing,

as is evaluation of the instrument's ability to

detect change over time and its predictive values

as a screen for dementia.

ACKNOWLEDGMENTS

This study greatly bene®ted from the contribu-

tions of Sean B. Robins and Rohina Furmuly.

Jason Kerns and all the staff involved in this

project from Davis Research, also provided

invaluable support.

REFERENCES

American Psychiatric Association. (1994). Thediagnostic and statistical manual-IV. Washington,DC.

Brandt, J., Spencer, M., & Folstein, M. (1988). TheTelephone Interview for Cognitive Status. Neuro-psychiatry, Neuropsychology, & Behavioral Neurol-ogy, 1, 111±117.

Brandt, J., Welsh, K.A., Breitner, J.C.S., Folstein, M.F.,Helms, M., & Christian, J.C. (1993). Heriditaryin¯uences on cognitive functioning in older men: Astudy of 4000 twin pairs. Archives of Neurology, 50,599±603.

Breitner, J.C., Welsh, K.A., Gau, B.A., McDonald,W.M., Steffens, D.C., Saunders, A.M., Magruder,K.M., Helms, M.J., Plassman, B.L., & Folstein,M.F. (1995). Alzheimer's disease in the NationalAcademy of Sciences-National Research CouncilRegistry of Aging Twin Veterans. III. Detection of

cases, longitudinal results, and observations on twinconcordance. Archives of Neurology, 52, 763±771.

Burkart, M., Heun, R., & Benkert, O. (1998). Serialposition effects in dementia of the Alzheimer type.Dementia & Geriatric Cognitive Disorders, 9, 130±136.

Crum, R.M., Anthony, J.C., Bassett, S.S., & Folstein,M.F. (1993). Population-based norms for the Mini-Mental State Examination by age and educationlevel. Journal of the American Medical Association,269, 2386±2391.

Folstein, M.F., Folstein, S.E., & McHugh, P.R. (1975).Mini-Mental State. A practical method for gradingthe cognitive state of patients for the clinician.Journal of Psychiatric Research, 12, 189±198.

Gifford, D.R., & Cummings, J.L. (1999) Evaluatingdementia screening tests: Methodologic standards torate their performance. Neurology, 52, 224±227.

Gorsuch, R.L. (1983). Factor analysis (2nd ed.).Hillsdale, NJ: Lawrence Erlbaum.

Lezak, M. (1995). Neuropsychological Assessment (3rded.). New York, NY: Oxford University Press.

Lord, F.M. (1980). Applications of Item ResponseTheory to Practical Testing Problems. Hillsdale, NJ:Lawrence Erlbaum.

Nunnally, J.C., & Bernstein, I.H. (1994). Psychometrictheory (3rd ed.). New York, NY: McGraw-Hill.

Plassman, B.L., Havlik, R.J., Steffens, D.C., Helms,M.J., Newman, T.N., Drosdick, D., Phillips, C.,Gau, B.A., Welsh-Bohmer, K.A., Burke, J.R.,Guralnik, J.M., & Breitner, J.C. (2000). Document-ed head injury in early adulthood and risk ofAlzheimer's disease and other dementias. Neurol-ogy, 55, 1158±1166.

Plassman, B.L., Newman, T.T., Welsh, K.A., Helms,M., & Breitner, J.C.S. (1994). Properties of theTelephone Interview for Cognitive Status: Applica-tion in epidemiological and longitudinal studies.Neuropsychiatry, Neuropsychology, & BehavioralNeurology, 7, 235±241.

Tangalos, E.G., Smith, G.E., Ivnik, R.J., Petersen, R.C.,Kokmen, E., Kurland, L.T., Offord, K.P., & Parisi,J.E. (1996). The Mini-Mental State Examinationin general medical practice: Clinical utility andacceptance. Mayo Clinic Proceedings, 71, 829±837.

Tombaugh, T.N., & McIntyre, N.J. (1992). The Mini-Mental State Examination: A comprehensivereview. Journal of the American Geriatric Society,40, 922±935.

Welsh, K.A., Breitner, J.C.S., & Magruder-Habib, K.M.(1993). Detection of dementia in the elderly usingtelephone screening of cognitive status. Neuropsy-chiatry, Neuropsychology, & Behavioral Neurology,6, 103±110.

COMPUTER-ASSISTED COGNITIVE SCREEN 175

Dow

nloa

ded

by [

Ein

dhov

en T

echn

ical

Uni

vers

ity]

at 0

3:57

23

Nov

embe

r 20

14