Plenary Session 2 Psychometric Assessment

Ralph H B Benedict, PhD, ABPP-CN

Professor of Neurology and Psychiatry

SUNY Buffalo

• Reliability

• Validity – Group Discrimination, Sensitivity

• Validity – Association with ADLs

• Validity – Ability to Detect Clinically Meaningful Change

Cognitive Performance ≠ PRO

Benedict RHB, et al (2004). Reliable Screening for Neuropsychological Impairment

in MS. Multiple Sclerosis, 10:675-678

Patient report

Informant Report

Composite MACFIMS

-.45 -.64

CES-D-10 Depression

.61 .37

100 MS Patient Volunteers from Community

100 NCs

Most Sensitive Tests:

Selective Reminding Test

7/24 Spatial Recall Test

Paced Auditory Serial Addition Test

Brief Repeatable Battery of Neuropsychological Tests for MS Selective Reminding Test 10/36 Spatial Recall Test Symbol Digit Modalities Test Paced Auditory Serial Addition Test Controlled Oral Word Association aka Verbal Fluency

Rao et al. 1991.

Language Spatial

Abilities Attention

Span

Processing

Speed Memory

Executive

30

25

20

15

10

5

0

8-9%

12-19%

7-8%

22-25%

22-31%

13-19%

Recommended Rao Battery: Problems

• No assessment of visual/spatial or executive

function

• Psychometrics of adapted versions of SRT and

7/24 Spatial Recall Test unknown

• Validity of adapted version of SDMT unknown

• Lack of information on reliability:

– test-retest

– alternate form

SDMT test 1 vs test 2 r = 0.84 PASAT test1 vs test3 r = 0.70

Acceptable > 0.80

Costs of low reliability : - less confidence in accuracy in clinical settings

- error variance reduces statistical power

Minimal Neuropsychological Assessment of MS Patients: A Consensus Approach

Benedict Fischer et al. Clinical Neuropsychologist, 2002

Standardized Manual, commercially available

Normative Data Published, large sample, age 20-55

Adequate Range No ceiling or floor effects.

Reliability Moderate/high test-retest reliability.

Criterion Validity Discriminates MS and controls, good sensitivity & specificity.

Alternate Forms Alternate forms are equivalent and reliable.

Practical Brief, minimal equipment, minimal confound of neurol signs.

Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS) Benedict et al, 2002

Test Cognitive Domain Min

Paced Auditory Serial Addition Test Processing Speed/Working Memory 10

Symbol Digit Modalities Test Processing Speed/Working Memory 5

California Verbal Learning Test Learning and Memory 25

Brief Visuospatial Memory Test Learning and Memory 10

D-KEFS Sorting Test Executive Function 25

Judgment of Line Orientation Test Visual-Spatial Ability 10

Controlled Oral Word Association Test Language and Other Domains 5

TestxTrial F=4.7 p<.05

T1 p<.001

T5 p=.05

Delay p<.01

Trial F=49.7 p<.001

TestxTrial ns

T1 ns

T5 ns

Delay ns

ns

p=.07

Test-Retest r

CVLT2 Total Learning 0.78

CVLT2 Delayed Recall 0.89

BVMTR Total Learning 0.91

BVMTR Delayed Recall 0.85

PASAT Total Score 0.94

SDMT 0.97

COWAT 0.84

• 291 MS pts

• 56 controls

Benedict RHB, et al (2006). J Inter Neuropsychol Soc 12:549-558.

TEST % z< -1.5

COWAT 13.1

JLO 22.3

CVLT-II Learning 29.6

CVLT-II Delayed Recall 34.4

BVMT-R Learning 54.3

BVMT-R Delayed Recall 56.0

PASAT 3.0 ISI 27.4

PASAT 2.0 ISI 27.5

Symbol Digit Modalities Test 51.9

DKEFS Sorting Test Categories 15.8

DKEFS Sorting Test Description 25.8

59.5% impaired

on 2 or more

measures

Validity – Visual Processing Speed and

Memory Tests Most Sensitive in MS

Benedict et al MACFIMS data from 291 pts, superimposed on data adapted

from Rao et al. 1991.

Language Spatial

Abilities Attention

Span

Processing

Speed Memory

Executive

30

25

20

15

10

5

0

8-9%

12-19%

7-8%

22-25%

22-31%

13-19%

Model Predicting disabled/employed status,

conservatively defined (n=102)

Model Predicting disabled/employed status, liberally defined (n=162)

Odds* Ratio

95% Confidence

Interval p Odds* Ratio

95% Confidence

Interval p

Depression† 2.30 0.95- 5.54 0.065 2.70 1.45 - 5.04 0.002

Disease Course†† 5.79 1.88 – 17.8 0.002 5.00 2.20 – 11.4 <0.001

CVLT2-DR 3.20 1.19 - 8.63 0.022 2.82 1.41 – 5.66 0.003

BVMTR-DR ns ns

PASAT 4.34 1.63 - 11.6 0.003 ns

SDMT ns ns

DKEFS ns 2.27 1.03 - 5.01 0.042

Parmenter B, Testa SM, Schretlen DJ, Benedict RHB. (2010). Utility of Regression-based Norms in Interpreting the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS). J Int Neuropsychol Soc, 16:6-16.

Measure Predictor B Standard Error B T Standardized B Total

R square

SDMT (constant) 5.953 5.457 1.091

age 0.249 0.235 1.057 0.768

age2 0.005 0.003 1.757 -1.271

sex 0.601 0.706 0.852 0.08

education 0.157 0.154 1.017 0.095 0.276

Sample: 65 MS 46 NC 72% RR Mean EDSS = 3.3 + 1.7 Memory Tests in Counterbalanced Order

Strober et al 2009: Comparing Effect Sizes Among Gold Standard Cognitive Outcomes

MACFIMS 90 min testing Rao BRNB 45 min testing Brief Assessment ??

New Learning and Recall Memory Processing Speed and Working Memory Retention of Learned Material

MRI Partial r

Bermel et al, 2003 BCR +0.43

Benedict et al, 2004 3VW -0.71

Amato et al, 2004 Cortical Vol +0.65

Christodoulou et al, 2004 Vent Vol -0.70

Benedict et al, 2005 Temp Lobe Vol +0.64

Benedict et al, 2006 BPF +0.58

Benedict et al, 2006 3VW -0.62

Houtchens, Benedict, et al, 2007 Thalamic Vol +0.66

Tekok-Kilic, Benedict, et al, 2007 R Sup Frnt GM +0.64

Benedict, et al, 2009 Deep GM +0.62

Concurrent Validity – SDMT most robust correlate with

structural imaging

Discriminative Validity - SDMT is most sensitive

• Camp et al. Brain. 1999;122:1341-1348

• Benedict et al. JINS; 12:549-558

• Benedict et al. Arch Neurol; 63:1301-1306

• Deloire et al. JNNP. 2005;76:519-526

• Deloire et al. MS. 2007;13:52-57

• Huijbregts et al. Neurol. 2004; 63:335-339

• Nocentini et al. MS. 2006;12:77-87

• Parmenter et al. MS. 2007;13:52-57

• Strober et al. MS. 2009; in press

SDMT in Longitudinal Work: Amato et al. Mult Scler, 2010

SDMT in Longitudinal Work: Amato et al. Mult Scler, 2010

660 pts, 109 sites in 21 countries

Translated into 14 languages

Morrow SA, O’Connor PW, Polman CH, Goodman AD, Kappos L, Lublin FD, Rudick RA, Jurgensen S, Paes D, Forrestal F, Panzara MA, Benedict RHB (2010). Evaluation of the Symbol Digit Modalities Test (SDMT) and MS Neuropsychological Screening Questionnaire (MSNQ) in Natalizumab treated MS patients over 48 weeks. Multiple Sclerosis, 16:1385-1392

•Morrow SA, Jurgensen S, Forrestal , Munchauer FE, Benedict RHB. (2011). Effects of acute relapses on neuropsychological status in multiple sclerosis patients. Journal of Neurology, in press.

5% Drop

• 65 MS cases with confirmed relapses: age 42.9±7.8, 66.2% female, median EDSS 3.0 (range 1.0-7.5).

• No significant difference on demographics or EDSS at baseline.

Adaptation of SDMT to the MSFC: Drake et al. Mult Scler, 2010

Symbol Digit Modalities Test in MSFC

Test-Retest over 2 years

MSFCPjnims r = 0.84; MSFCSjnims r = 0.83

MSFCPjninc r = 0.80; MSFCSjninc r = 0.81

BICAMS Brief International Cognitive Assessment for MS

BICAMS Member Meeting September 11, 2010

Chairs: Dawn Langdon and Ralph Benedict

1. The committee will recommend a brief cognitive assessment that is optimised for small centres, with perhaps one or few staff members, who may not have neuropsychological training.

2. The recommended assessment will be constructed to maximise international use, although in the first instance suitability may only be established for certain areas of the world.

3. The committee will review relevant scientific papers and evaluate possible scales for psychometric qualities, international application, ease of administration, feasibility in the specified context and acceptability to patients.

4. A consensus opinion article will be submitted by the committee, which will comprise the objectives, a literature review, and recommendations

Language and Spatial Processing

Controlled Oral Word

Association Test

Judgment of Line

Orientation Test

PSYCHOMETRIC AVE 2.6 2.4

PRAGMATIC AVE 2.6 2.4

Memory Wechsler

Memory Scale

Rey Auditory Verbal

Learning Test

California Verbal

Learning Test

Selective Reminding

Test

Brief Visuospatial Memory Test

Revised

10/36 or 7/24 Spatial Recall

Test

PSYCHOMETRIC AVE 1.3 1.8 2.9 2.5 3.0 2.3

PRAGMATIC AVE 2.1 2.3 2.6 2.2 2.2 2.6

Attention, Processing Speed, Working Memory Digit Span Spatial Span

Wechsler Digit Symbol

or Coding Test

Symbol Digit Modalities

Test

Paced Auditory

Serial Addition Test

Stroop Conflict Tasks N-Back Tasks

PSYCHOMETRIC AVE 2.5 2.3 2.2 2.8 2.6 2.5 2.5

PRAGMATIC AVE 2.7 2.5 2.9 3.0 1.9 2.5 2.0

Higher Executive Function

Wisconsin Card Sorting

Test

Tower Tasks, eg Tower of

London DKEFS Sorting

Test

Raven's Progressive

Matrices

PSYCHOMETRIC AVE 1.6 1.7 2.4 2.3

PRAGMATIC AVE 2.1 2.3 2.3 2.3

SDMT CVLT2 Learning BVMTR Learning

Standardization Smith 1982 Teat Manual Delis 2000 Test Manual Benedict 1997 Test Manual

Normalization Parmenter 2010. Peer review journal article

Delis 2000 Test Manual Benedict 2005 Test Manual

Reliability I: Test-Retest Benedict 2005 r = 0.91 Benedict 2005 r = 0.80 Benedict 2005 r = 0.91

Reliability II: Alternate Form

Rao 1991. No or Little Data Delis 2000. Fair Benedict 1996. Good

Validity I: Criterion Related

Many Studies. Good Many Studies. Good Many Studies. Good

Validity II: Clinically Meaningful Change

Morrow 2010. Fair No or Little Data No or Little Data

Table 1 Manner in which BICAMS measures meet psychometric criteria in samples with English as a first language.

Multiple Sclerosis (MS) CDE Working Group

Chairman: [Ralph HB Benedict]

Subgroup members: [Dawn Langdon, Lauren Krupp, Nicholas LaRocca, Steve Rao]

Which psychometric tests should be recommended as a

brief battery – ie ‘supplemental’ CDE for studies focusing on cognition in MS?

Should any test be recommended for the overall ‘core’ MS CDE?

Test Domain Time

CVLT2 Learning Trials 1-5 auditory/verbal memory 10 min

BVMTR Learning Trials 1-3 visual/spatial memory 05 min

Rao PASAT 3.0 ISI processing speed and working memory 05 min

Rao SDMT processing speed and working memory 05 min

DKEFS Sorting Test executive function 10 min

CVLT2 Delayed Recall auditory/verbal memory 10 min

BVMTR Delayed Recall visual/spatial memory 05 min

COWAT executive function 05 min

Symbol Digit Modalities Test (SDMT) recommended for MS CDE

1. Excellent reliability

2. Excellent discriminative validity

3. Can replace the PASAT in MSFC

4. Clinically meaningful change associated with discrete raw score decline

Conclusions

• Self Report is Invalid, performance measures are

and will be needed

• SDMT is the most cost effective and sensitive test

available

• Should be complemented by verbal and visual

memory tests

• Psychometric standards will be increasingly

emphasized in clinical trial work