Jorge Mndez Seijas

Individual Differences in Reactivity Research:

Working Memory and Extraversion/Introversion

Abstract

Verbalizations, also known as verbal protocols or think-alouds, have become a common

methodological tool in second language acquisition (SLA) research studies. This procedure is

used to tap into participants mental processes, and therefore provides researchers with

information about the thought processes participants engage in as they complete a task. A

challenge for researchers using verbalizations is that having participants think aloud while they

complete a task may be reactive, that is, it may enhance or hinder participants performance.

Although having a control group that does not think aloud as a point of comparison was a

solution proposed in the very first study about reactivity in SLA (Leow & Morgan, 2004), many

subsequent studies have attempted to determine whether verbalizations are reactive, and if so,

with which tasks, under what conditions, and so forth.

Relevant to the question of reactivity in SLA research is that verbalizations have been

examined in absolute terms, with the conditions being with or without verbal protocols. A deeper

look into this method of data elicitation may reveal a much more complicated picture, as not all

think-alouds are the same (Ericsson & Simon, 1993). For instance, does the amount of

verbalizations (measured in words, syllables, etc.) result in differences within the participants in

the think-aloud conditions? Do individual differences play a role in the reported reactivity, or

lack thereof, in some of the tasks? Does a personality trait such as introversion/extroversion have

an impact on how reactive or non-reactive a task may be? This paper will attempt to answer this

question by adding to independent variables, working memory and introversion/extroversion,

and slightly modifying the research design of Bowles & Leow (2005). It will therefore be both a

partial replication and an extension

Introduction

Individual differences play an important role in second language acquisition (Dornyei,

2009) and second language acquisition research. In terms of think-aloud protocols, however,

Goo (2010) is in the only SLA study that has investigated if an individual difference

(specifically, working memory capacity) should be taken into consideration when employing

think-aloud protocols. Besides working memory, given the characteristics of verbalizations, the

introversion/extraversion dimension of personality may be an interesting addition to this strand

of research.

Working memory capacity (WMC) and introversion/extroversion may be relevant

because they show two dimensions that may define how participants interact with the concurrent

measures: examining WMC could allow researchers to determine to what extent the innate and

varying capacity that participants have to hold information in memory while completing a task

may induce positive or negative reactivity. Some of these questions regarding WMC have been

dealt with in Goo (2010), but limitations will be pointed out that may cast doubt on the reliability

of his results.

To have an even better picture of what reactivity is and what tasks it impacts, given the

conditions in which the data is collected, an individual difference like introversion/extroversion

would introduce a personality trait that may determine, to some extent, whether or not a task is

reactive. It is not unlikely that verbalizations may impose additional pressure on introverted

participants, as compared to extroverted ones. Furthermore, introverted participants may simply

verbalize less than extroverted ones, which could, in turn, affect whether or not the think-alouds

are reactive. By introducing these two individual difference variables, we could further explore

reactivity with regards to task type, and also the role that innate and social individual differences

play.

Verbal Protocols and Reactivity

Verbal protocols can be broadly divided into two types: concurrent and retrospective.

Concurrent verbal protocols, which will be the subject of my discussion in this paper, occur

when participants verbalize their thoughts out loud while they are completing a task.

Retrospective verbal protocols, on the other hand, occur when participants verbalize their

thoughts a posteriori, that is, after they have completed a given task. Verbal protocols can be

further divided into two different types: metacognitive (also known as metalinguistic) and non-

metacognitive (non-metalinguistic). In the first type, participants are asked for explanations and

justifications for their verbalizations, in the second, they are simply asked to voice their thoughts

without any explanations or justifications.

Concurrent protocols pose a potential threat to studies internal validity, though, as it is

possible that the imposition of a secondary task (i.e., verbalizing ones thoughts) may somehow

affect the primary process(es) required for the task(s) under investigation. Such a threat is known

as reactivity. Reactivity, when present, can affect performance on the primary task in two

different ways: it can potentially enhance performance (positive reactivity) or hinder it (negative

reactivity). The way researchers generally test for reactivity is by comparing the performance of

an experimental group (i.e., those who think aloud) to that of a silent control group. If the groups

perform equally well (or poorly), then the researcher assumes that the verbal protocols are non-

reactive. Currently, the results on reactivity have been mixed. That is, in the SLA literature, it is

possible to find studies that report positive reactivity in recognition and production tasks

(Rossomondo, 2007); negative reactivity in a written reformulation task (Sachs & Polio, 2007

(first experiment)), text completion (Sachs & Suh, 2007), reading comprehension (Bowles &

Leow, 2005; Goo, 2010; Morgan-Short, Heil, Botero-Moriarty & Ebert, 2012), L2 problem-

solving (Bowles, 2008), aural posttest accuracy (Sanz et al., 2009), and latency (Bowles & Leow,

2005; Sanz, Lin, Lado, Bowden & Stafford; 2009: Yoshida, 2008). It is also possible to find

studies that report no reactivity in reading comprehension tasks (Bowles & Leow, 2005; Leow &

Morgan-Short, 2004; Rossomondo, 2007; Yoshida, 2008), posttest production (Bowles & Leow,

2005; Leow & Morgan-Short, 2004; Sachs & Suh, 2007), system learning (Bowles, 2008),

writing (Sachs & Polio, 2007 (second experiment)), posttest accuracy (Sanz et al., 2009), written

reformulation tasks (Sachs & Polio, 2007), and latency (Sachs & Suh, 2007; Sanz et al., 2009).

The previously cited studies differ in many crucial aspects, such as the proficiency level

of the participant, the primary task under investigation (reading, writing, etc.), the type of

secondary task (metacognitive vs. non-metacognitive think-alouds), the dependent variables

(comprehension tasks, form recognition, etc.), and so forth. Having so many differences among

studies makes it difficult to make comparisons and generalizations. I will illustrate this by

organizing some of the studies according to two tasks: reading comprehension and writing

production, as only these tasks will be part of my experimental design.

Reactivity in Reading Comprehension and Writing Production

Let us first take a look at studies whose primary task was reading comprehension. Leow

& Morgan-Short (2004) compared a non-metalinguistic think-aloud group and a silent control

group. Analyses comparing the think-aloud and control group showed think-alouds were non-

reactive. That is, no significant differences between the groups were found. Similar results were

found in other studies that also used reading as a primary task, such as Rossomodo (2007), and

Yoshida (2008). Goo (2010), conversely, found that think-alouds were indeed reactive for

reading comprehension.

As for writing as a primary task, to my knowledge, two studies deal with the issue of

reactivity in writing production: Sachs and Polio (2007), and Yanguas & Lado (2012). Sachs &

Polio (2007)s found reactivity for one experiment and non-reactivity for the other experiment.

In this study, they investigated the effects of different types of written feedback (error correction

and reformulations) and the role of reactivity in L2 development of writing skills. Their results

for the first experiment (within-subjects) indicated that think-alouds were negatively reactive in

the reformulation group. In their second experiment (between-subjects), no reactivity was

reported. A look into this design and the comparisons it offers complicates matters further, as

they compare results from a within-subject design with results from a between-subject design.

Generalizations in writing can hardly be drawn by comparing such disparate designs. If we

include Yanguas & Lado (2012) to try to disentangle the problem, more differences are found:

the participants in their study are heritage speakers, while those in Sachs and Polio (2007) are

intermediate L2 English learners. Also, while the think-alouds in Yanguas & Lado could be

performed in either language, in Sachs & Polio (2007) participants were asked to think aloud in

the target language. Once again, as it was the case with reading as a primary task, generalizations

about reactivity cannot be easily made and further studies addressing the writing process are

clearly warranted.

Besides the internal and external validity issues, in all these studies, with the exception of

Goo (2010), verbalizations have been treated as if they were a uniform construct. That is, as

though it is enough to report that they took place, without further exploring their characteristics

or the individual differences in the participants that produce them. Some of these individual

differences may prove to have an impact in the quality and quantity of the verbalization, which

may, in turn, influence whether or not the verbalizations are reactive. Understanding these

variables will allow for a better understanding of verbal protocols and the type of reactivity they

may or may not generate. For the purposes of this study, I propose that two relevant individual

differences be taken into consideration: working memory and introversion/extroversion.

Working Memory

In the field of SLA, the interaction between working memory and learning has been

widely studied (Kormos & Sfr, 2008; Linck, Osthus, Koeth & Bunting, 2013, for a meta-

analysis; Mackey, Adams, Stafford, & Winke, 2008). This interaction is expected and of great

interest in the field given the cognitive load involved in speaking a second language (L2),

especially at low levels of proficiency. The concept of working memory is relatively recent, and

was introduced by Baddeley & Hitch (1974). WM has been gaining track in psychology and

SLA studies. WM is fundamentally a form of memory, but it is more than memory, for it is

memory at work (Conway, Jarrold, Kane, Miyake, & Towse, 2008, p. 3). Therefore, WM

involves cognitive multitasking, as information must be kept active while performing an

additional task. Such multitasking is especially relevant in think-aloud protocols, as participants

must perform a task while at the same time speaking aloud. The impact WM may have in terms

of reactivity (positive and negative) may very well depend of the type of task.

Goo (2010), for instance, examined working memory capacity (WMC) as an independent

variable. In his study, the think-aloud and non-think-aloud groups were further subdivided into

two groups, according to the working memory capacity levels of participants. His results

indicated that there was reactivity for participants with high memory capacity in the think-aloud

condition, and no reactivity for low memory capacity participants. Although having this variable

could be an important development in the reactivity strand, some internal validity issues hamper

this studys overall reliability. One major limitation is the sample size, which only included 5-8

participants per condition. The tasks in this study were intended to investigate the role of WMC

in reading comprehension and rule learning. One significant problem with this design is that how

rule learning is operationalized is not easily comprehensible, as it is not clear how a fill-in-the-

blank test serves this purpose. Furthermore, the structure under investigation (vamos + infinitive)

requires no rule learning whatsoever, for participants are given an infinitive form and they only

need to reproduce it. Another internal validity issue with this study is that it only presented

results for the relationship between WMC and performance in the think-aloud groups. The

aforementioned shortcomings make it necessary to further investigate the role WM plays in

reactivity, so as to make sure that the results in this study in fact hold in a more robustly designed

investigation.

Introversion-extroversion

Another important factor that should be taken into consideration in reactivity research is

the introversion-extroversion dichotomy. This personality trait may positively or negatively

impact the act of thinking aloud, thus potentially generating positive or negative reactivity. In the

previous section, I discussed that an innate trait such as WM may differentiate participants to

such degree that a concurrent measure might be more or less taxing. The same may prove to be

true about introversion and extroversion. The introvert/extrovert dimension of personality could

be of great interest in reactivity research, because performing better or poorer on a test may have

to do, to a certain extent, with how personality affects such performance. Even more so when the

activities in the experiment include speaking out loud, which could impose different levels of

pressure depending on whether or not a participant is an introvert or an extrovert. Revelle,

Amaral, and Turriff (1976), for instance, compared verbal intelligence scores of introverts and

extroverts in three different conditions: a) untimed, b) timed + placebo caffeine, and c) timed +

caffeine. The results indicated that extroverts performed well in all three conditions, while

introverts performance dramatically dropped from the untimed condition to those of timed +

placebo caffeine and timed + caffeine. Bates and Rock (2004) compared introverts and

extroverts performance in different conditions according to different levels of noise, ranging

from silent environments to very noisy ones, and his result indicated that introverts performance

peaked in the silence condition, and decreased performance was reported in all other conditions.

Extrovert outperformed both introverts and ambiverts (the neither-nor group) in all other

conditions. Introversion/extroversion with regards to SLA has been studied in terms of learning

(for a review, see Zafar & Meenakshi, 2013). To my knowledge, however, this dimension of

personality has not been studied in the second language acquisition field in terms of test taking

conditions or verbalization protocols.

As Goo (2010) suggested, an individual difference such as working memory capacity

might play a role in the reported reactivity, or lack thereof, in studies that involve verbalizations.

Given the nature of verbalizations, the dichotomy introvert/extrovert may also have an effect. In

order to address the role these individual differences play in the issue of reactivity, this study will

introduce to the robust design of Bowles & Leow (2005) two individual differences: working

memory capacity and the introvert/extrovert dimension. This study will be guided by the

following research questions:

1. Are verbalizations reactive during:

a. a reading comprehension task?

b. a writing production task in which students produce old exemplars of the

Spanish pluperfect subjective?

c. a writing production task in which students produce new exemplars of the

Spanish pluperfect subjective?

2. Does working memory capacity mediate the possible effects of reactivity?

3. Does introversion and extroversion mediate the possible effects of reactivity?

Method

The current study will be a partial replication and expansion of Bowles and Leow (2005).

Most of the materials and procedures of Bowles and Leow (2005) will be maintained, and small

methodological changes will be implemented. Two tests will be added to the design of Bowles

and Leow (2005): a working memory test and an introversion/extroversion test.

Participants

A total of 80 Spanish advanced students will participate in the current study: 40 will be

assigned to a control group, and 40 will be assigned to a think-aloud group. As in Bowles and

Leow (2005), participants will be given a pretest to make sure they only have minimal

knowledge of the Spanish pluperfect subjunctive. Any participant that scores over 2 out of 10 on

the pretest will be eliminated from the analyses.

Reading Materials and Tests

The target structure for the current study will be the Spanish pluperfect subjective. As in

Bowles and Leow (2005), the participants will read an 861-word article titled Si Bush hubiera

sido porteo. This article contains 28 instances of the Spanish pluperfect subjective. After the

students read the article, they will take one comprehension test and two written production tests.

The comprehension test and the written production tests will be the same tests used in Bowles

and Leow (2005). The comprehension test will be done in English, and it will consist of 10

multiple-choice items. There will be two written production tests: one will test the participants

ability to produce old exemplars of the Spanish pluperfect subjective and one will test their

ability to produce new exemplars. These tests are both fill-in-the-blank tests. The first test

consists of 20 sentences directly from the text that the participants read. Ten instances will test

their knowledge of the Spanish pluperfect subjective, and ten instances will be distractor items.

The second test will be similar, but it will consist of sentences that were not presented in the

original test so as to determine if the participants are able to create new exemplars of the Spanish

pluperfect subjective. As in the previous test, 10 items will test the Spanish pluperfect subjective

and 10 will be distractors.

Working Memory and Introversion/Extroversion Test

Working Memory Span Tests, as described in Conway, Kane, Bunting, Hambrick, Wilhelm, &

Engle (2005), will be used in order to determine the participants levels of WMC. Following Goo

(2010), two tasks will be used to test working memory capacity: a listening span task, as

developed by Mackey et at (2008), and a operation span task that is a variant of Turners and

Eagles, developed specifically for the said study by Goo (2010). The introversion/extroversion

dimension will be measured by means of the Myers-Briggs Type Indicator, developed in the

1940s and 1950s by Isabel Briggs-Myers.

Procedure

Participants will first take the pretest on the Spanish pluperfect subjective in class. Only

those who do not answer more than 2 out of 10 correctly will be asked to participate in the

primary data collection. Unlike Bowles and Leow (2005), the data for the primary data collection

will be collected on an individual basis, and not in a language laboratory. The reason why it will

be done this way is the following: if reactivity depends on the introversion/extroversion level of

the participants, then it is likely that this method of data collection would have an effect when

participants are seated in a room with the researcher. That is, having the participants complete

the tasks in this way might have a greater effect on a personality trait such as

introversion/extroversion, making it possible for the researcher to better study the differences

between participants.

The introversion/extroversion test and the working memory test will be carried out at the

beginning of the first session. Depending on the results, participants will be assigned to different

groups: think-aloud, low WMC; think-aloud, high WMC; think-aloud, introvert; think-aloud,

extrovert; non-think-aloud, low WMC; non-think-aloud, high WMC; non-think-aloud, introvert;

non-think-aloud, extrovert.

Participants assigned to the think-aloud groups will be given the same non-metalinguistic

instruction as that in Bowles and Leow (2005). The data for the control groups will still be

collected individually with the researcher, but they will not be asked to think aloud while reading

or completing the production task.

Once participants read the text they will then complete the comprehension test and the

written production tests (old and then new exemplars).

Scoring Procedure

The same procedures will be used as in Bowles and Leow (2005) for scoring the

comprehension test and the two written production tests: participants will receive 1 point for

each correct answer for a total of 10 possible points per test. Participants will receive a numerical

score for both the introversion/extroversion and working memory tests.

Analyses

To determine whether think-alouds are reactive (research question 1) a series of three

independent t-tests will compare the silent control groups performance on each of the three

dependent variables.

To determine if working memory (research question 2) mediates the effects of reactivity,

separate tests will be run for each of the three dependent variables (comprehension, production

of old exemplars, production of new exemplars). The possible mediating effects of working

memory will be investigated in two ways: (1) by dividing the participants into high and low

working memory groups 1and, (2) by entering working memory as a covariate. For (1),

participants working memory scores will first be ranked. The bottom one-third of the group will

be considered to have low working memory capacity and the top one-third high working 1Unlike Goo, I think its better to eliminate participants with MID WMC. That way, comparisons will be clearer.

memory capacity.

To determine if introversion/extroversion (research question 3) affects reactivity, similar

tests will be conducted as with the working memory scores. Separate tests will be conducted for

each of the three dependent variables, and the tests will be run in 2 ways: (1) by dividing the

participants into introversion and extroversion groups and, (2) by entering

introversion/extroversion as a covariate. For (1), participants introversion/extroversion scores

will first be ranked. Then, the bottom one-third of the introversion/extroversion class will be

considered introverted and the top one-third will be considered extroverted. Finally, 2 ANOVAS

will be conducted for each of the three dependent variables.

References

Bates, T. C., & Rock, A. (2004). Personality and information processing speed: Independent

influences on intelligent performance. Intelligence, 32, 3346.

Bowles, M. A. (2008). Task type and reactivity of verbal reports in SLA: A first look at a L2 task

other than reading. Studies in Second Language Acquisition, 30, 359-387.

Bowles, Melissa A. The Think-aloud Controversy in Second Language Research. New York:

Routledge, 2010. Print.

Bowles, M. A., & Leow, R. P. (2005). Reactivity and type of verbal report in SLA research

methodology: Expanding the scope of investigation. Studies in Second Language

Acquisition, 27, 415-440.

Baddeley, A. D., & Hitch, G. J. (1974). Working memory. In S. Dornic (Ed.), Recent advances

in learning and motivation Vol. VIII (pp. 47-89). New York: Academic Press.

Conway, A. R. A., Kane, M. J., Bunting, M. F., Hambrick, D. Z., Wilhelm, O., & Engle, R. W.

(2005). Working memory span tasks: A methodological review and users guide.

Psychonomic Bulletin & Review, 12, 769-786.

Conway, A. R. A., Jarrold, C., Kane, M. J., Miyake, A., & Towse, J. N. (Eds.). (2007).

Variation in working memory. Oxford, UK: Oxford University Press.

Horwitz, E. K. (2001). Language Anxiety and Achievment. Annual Review of Applied

Linguistics, 21, 112-126.

Ericsson, K.A. and Simon, H.A. (1993), Protocol Analysis; Verbal Reports As Data (revised

edition; original edition published 1984) (Cambridge, MA: Bradford Books/MIT Press).

Goo, J. (2010). Working memory and reactivity. Language Learning, 43, 245254.

Horwitz, E. K., Horwitz, M. B., & Cope, J. (1986). Foreign language classroom anxiety. Modern

Language Journal, 70 , 125-132.

Kormos, J. and A. Sfr (2008) Phonological short term-memory, working memory and

foreign language performance in intensive language learning. Bilingualism: Language

and Cognition 11.2: 26171.

Linck, J.A., Osthus, P., Koeth, J.T., & Bunting, M.F. (2013). Working memory and second

language comprehension and production: A meta-analysis Psychonomic Bulletin &

Review

Leow, R. P., & Morgan-Short, K. (2004). To think aloud or not to think aloud: The issue of

reactivity in SLA research methodology. Studies in Second Language Acquisition, 26,

3557.

Mackey, A. R. Adams, C. Stafford and P. Winke (2010) Exploring the relationship between

modified output and working memory capacity. Language Learning 60.3: 50133.

Morgan-Short, K., Heil, J., Botero-Moriarty, A., Ebert, S. (2012). Issues of Think-Alouds and

Depth of Processing. Studies in Second Language Acquision, 34, 659-685.

Revelle, W., Amaral, P., & Turriff, S. (1976). Introversion-extraversion, time stress, and

caffeine: effect on verbal performance. Science, 192, 149-150.

Rossomondo, A. E. (2007). The role of lexical temporal indicators and text interaction format in

the incidental acquisition of the Spanish future tense. Studies in Second Language

Acquisition, 29, 39-66.

Sachs, R., & Polio, C. (2007). Learners uses of two types of written feedback on a L2 writing

revision task. Studies in Second Language Acquisition, 29, 67100.

Sachs, R., & Suh, B. R. (2007). Textually enhanced recasts, learner awareness, and L2 outcomes

in synchronous computer-mediated interaction. In A. Mackey (Ed.), Conversational

interaction in second language acquisition: A collection of empirical studies (pp.197

227). Oxford: Oxford University Press.

Sanz, C., Lin, H.-J., Lado, B., Bowden, H. W., & Stafford, C. A. (2009). Concurrent

verbalizations, pedagogical conditions, and reactivity: Two CALL studies. Language

Learning, 59(1), 3371.

VanPatten, B. & J. Rothman (2014). Against rules. In A. Benati, C. Laval, & M. J.

Arche (Eds.), The grammar dimension in instructed second language acquisition:

theory, research, and practice (pp. 15-35). London: Bloomsbury.AILA Applied

linguistics series, 9 (9). John Benjamins, Amsterdam, pp. 243-256.

Yanguas, I., & Lado, B. (2012). Is Thinking Aloud Reactive When Writing in the Heritage

Language. Foreign Language Annals 45, 380-399.

Yoshida, M. (2008). Think-aloud protocols and type of reading task: The issue of reactivity in L2

reading research. In M. Bowles, R. Foote, S. Perpin, & R. Bhatt (Eds.), Selected

proceedings of the 2007 Second Language Research Forum (pp. 199-209). Somerville,

MA: Cascadilla Proceedings Project.