Test-Retest Reliability and Convergent Validity of Measures of Executive FunctionOliver Sawi & Kenneth Paap

Department of Psychology, San Francisco State University, San Francisco, CA

• The construct of executive function (ef) often viewed as a set of interrelated processes all involving the prefrontal cortex, with each component recruiting other constellations of cortical function.

• Smith and Kosslyn (2007) partition executive processing into five sub-processes: executive attention, conflict monitoring, switching, sequencing, and response inhibition.

• Miyake, et al. (2000) have proposed a similar family of interrelated executive processes. • This framework, in contrast to the unitary view of the central executive (Baddeley, 1986), allows for the

possibility that the sub-processes have some degree of anatomical and functional independence. • Consequently, one could be good at interference tasks that require only executive attention (inhibitory control)

and poor at tasks that require only switching.

Our principle argument is that an investigation of individual differences in executive function should be grounded in a specific conceptual framework that elucidates the nature of executive processes and guides operational definitions for manipulating and measuring them. Furthermore, there needs to be more converging evidence that corresponding indicators from different tasks correlate with one another and, therefore, are valid measures of a specific type of executive function.

Participants

The 53 participants were psychology majors at SFSU who volunteered to participate as one alternative for satisfying a class requirement. Participants have diverse backgrounds, for example, 26 were monolingual and 27 billingual.

Method

Introduction

No Convergent Validity for Executive Attention

All participants completed four computer-controlled tasks commonly assumed to require executive control and a demographic survey. Participants then returned a week later, at the same time, and completed the four computer-controlled tasks again.

Antisaccade Task. Task on each trial was to identify the target stimulus (“B”, “P”, “R) by pressing the key with the corresponding label with three fingers of the right hand. (“<“,”>”,”?”) Briefly presented target followed by visually similar mask (“8”). Antisaccade condition: a distractor stimulus is always blinked before on the opposite side from the target stimulus.

Flanker task. On each trial either a left or right pointing arrow appeared at fixation and participants were required to press the corresponding key. Half of the trials were incongruent - the central arrow was flanked by a pair of arrows pointing in the opposite direction.

No evidence for a Bilingual Advantage in Executive Function

Cognitive/Neural Components of Executive Function

References

Color-shape switching task. On each trial a single target appeared that was either a circle or a triangle and colored either blue or red.

Block 1: participants always made color decisions with two fingers of the left hand. Block 2: they always made shape decisions with two fingers of the right hand. These two “pure” blocks were combined to provide a baseline for assessing mixing costs and switching costs.Block 3: the target stimulus was always preceded by a precueHalf of the trials were “repeat” trials, half were “switch” trials.

Mixing Costs: difference between the mean of the pure trials and the mean of the repeat trialsSwitching Costs: difference between the mean of the “repeat” and and “switch” trials

Baddeley, A. (1986) Working memory. Oxford. Clarendon Press. Fan, J. I., McCandless, B. D., Thomas, K. B., & Posnr, M. I. (2003). Cognitive and brain consequences of

Keye, D., Wilhelm, O., Oberauer, K., van Ravenzwaaij, D. (2009). Individual differences in conflict-monitoring:

Miyake, A. Friedman, N. P., Emerson, M. Witzki, A. H., Howerter, A., & Wagner, T. D. (200). The unity and

Smith, E. E., & Kosslyn, S. M. (2007). Cognitive psychology: mind and brain. Upper Saddle River, NJ: Pearson

conflict. NeuroImage, 18, 42-57.

testing means and covariance hypotheses about the Simon and Eriksen flanker task. Psychological Research, 73, 762-776.

diversity of executive functions and their contribution to complex “frontal lobe” tasks. CognitivePsychology, 41, 49-100.

Prentice Hall.

Lack of Convergent Validity Across Measures of EF

Task Simon Flanker Mixing Switching AntisaccadeSimon Effect 0.185 0.021 0.153 0.214

Flanker Effect 0.185 0.035 0.13 0.22

Mixing Costs 0.021 0.035 0.282 0.57

Switching Costs 0.153 0.13 0.282 0.202

Antisaccade 0.214 0.22 0.57 0.202

Table 5. Correlations between measures of EF

Group Main Effect: F(1, 56) = 5.08, p = .028 Group x Congruency Interaction: F(1,56) = 5.56, p = .022

Group Main Effect: F(1, 51) = 3.56, p = .065

Group Main Effect: F(1, 56) = 4.34, p = .042 Group x Congruency Interaction: F(1,56) = 0.811, p = .372

Group Main Effect: F(1, 55) = 8.52, p = .005Group x Congruency Interaction: F(1,55) = 3.48, p = .068

_ n Eng. Prof. Eng AOA Other Prof Other AOA % Eng Switching Frequency PEDBilinguals 26 6.0 4.8 6.0 4.5 61% 3.3 3.8

Monolinguals 25 6.7 0.0 1.3 11.5 98% 0.4 4.6

Table 6. Language Group Demographics

DiscussionThere were no main effects of group nor were there any GroupxCongruency interactions. In fact, the only significant difference between the groups was in the form of a bilingual disadvantage in the Simon Effect (replicating a finding from a previous study). In addition, the antisaccade and flanker effects also trended towards a bilingual disadvantage. Further, the low test-retest reliability and lack of convergent reliability may contribute to inconsistent findings for the bilingual advantage in EF.

Test-Retest Reliability of Measures of Executive Function (EF)

Task r n Exact pAntisaccade Effect 0.774 41 0.000Flanker Effect 0.520 41 0.000Simon Effect 0.137 44 0.376*Switching Costs 0.635 47 0.000Mixing Costs 0.472 47 0.001

r df exact p0.946 56 0.0000.556 58 0.0000.473 60 0.000*0.546 59 0.0000.815 59 0.000

Task r df exact pAntisaccade Effect

Flanker Effect 0.554 58 0.000

Simon Effect

Switching Costs 0.533 59 0.000

Mixing Costs 0.774 59 0.000

r df exact p

0.484 58 0.000

0.395 59 0.000

0.714 59 0.000

Table 1. Session 1 vs. Session 2 Table 2. Block 1 to Block 2 (Session 1)

Table 3. Block 2 to Block 3 (Session 1) Table 4. Block 1 to Block 3 (Session 1)

Discussion

The measures of EF produce moderate to low reliability 1. between sessions and 2. between blocks within the same session. For example, while significant, mixing cost in session 1 can only account for 22.3% of the variance in session 2. However, the Simon Effect shows significant block to block reliability, but not session to session.

Taken together, this suggests that the Simon Effect is more vulnerable to day to day variation than the other purported measures of EF, more specifically, Executive Attention (antisaccade, flanker) and undermines the idea that these tasks are measuring the same construct.

Simon task. On each trial either a Z or / appeared to the left or right of fixation and participants responded by pushing the corresponding key. Half of the trials were incongruent in that the spatial location of the target was opposite the location of the correct response key.

Results

Discussion

The absence of an association between interference effects has important implications for the standard practice of using these tasks as indicators of individual differences in the ability to control executive attention.

If there is no association, then only one of the tasks could require executive attention or the inhibitory control exercised in the tasks are completely task dependent. This lack of convergence undermines the confidence that individual or group differences in the magnitude of the flanker, antisaccade or Simon effect signals differences in executive attention.

Eng. Prof. Scale level 6 is "as fluent as a typical native speaker", level 7 is "better than a typical native speaker" AOA Age of Acquisition (native language coded as 0) PED Parent's Educational Level t(49)=1.82, p = .075

Executive Attention (Inhibitory Control). The difference between the congruent and incongruent trials in both the Simon and flanker task is used as an indicator of executive attention. The RT for the antisaccade task is also used as an indicator.

*All three tasks yield significant interference effects that are typical in magnitude for each task.