validation of the c-test amongst hungarian efl learners · 192 validation of the c-test amongst...

Validation of the C-test amongstHungarian EFL learnersZoltan Dornyei and Lucy Katona Eotvos University, Budapest

This article reports on the results of a research programme carried out tovalidate the C-test amongst Hungarian EFL learners. One hundred and twouniversity English majors were administered four different language tests(including an oral interview) to form a General Language Proficiencymeasure against which the C-test was evaluated. Various analyses weremade, partly to replicate the results of the earlier studies to see to whatextent these could be generalized, and partly to shed light on controversialissues. The same C-test was then administered to four groups of secondaryschool pupils (N=53) to examine whether the findings amongst universitystudents were also true in another proficiency range. The results of theprogramme confirmed that the C-test is a reliable and valid instrument, anddetailed information was obtained about issues such as text difficulty and textappropriateness, the role of content and structure words, and the use ofdifferent scoring methods.

I IntroductionThe popularity of the cloze test has proved that language examinersand testers are very much in need of a written measuring instrumentwhich is easy to design and administer, and which assesses overalllanguage proficiency reliably. This popularity is all the more signi-ficant as cloze tests are known to be rather difficult and problematicto score objectively unless 'exact word scoring' is followed, which isnot easy to justify and makes the test very difficult. There have alsobeen more theoretical criticisms of cloze testing, concerning therepresentatives of the deleted words, the fact that the choice of aparticular text, deletion rate, starting point for deletion, and scoringmethod considerably affects the score, and that even native speak-ers can rarely obtain a maximum score, which is rather surprisingabout a test which is supposed to measure language abilities (for anoverview see Alderson, 1979; Klein-Braley and Raatz, 1984).

In 1981, Christine Klein-Braley and Ulrich Raatz introduced anew deletion technique, 'the rule of 2', which was believed toremedy most of the shortcomings of the classical cloze procedure.According to 'the rule of 2', the second half of every second wordshould be deleted in a text, starting and ending with an intact

© Edward Arnold 1992

188 Validation of the C-test amongst Hungarian EFL learners

sentence. This resulted in what the two researchers termed the'C-test' (for an overview see Klein-Braley and Raatz, 1984; Raatzand Klein-Braley, 1985; Raatz, 1985; Klein-Braley, 1985).

The C-test is easy both to design and to score and several differenttexts can be used to make up a complete test, which is shorter andcontains more deletions than a cloze test. Native speakers can scoreup to 100%, and Raatz (1985) presents evidence that the truncatedwords tend to be representative of the texts they occur in. What ismore, students also find C-tests less frustrating than cloze tests.Almost too good to be true, and yet the empirical results all attest tothe fact that the C-test is a reliable and valid instrument, doing'everything that the cloze test promised' (Klein-Braley and Raatz,1984:145).

One shadow on the positive features of the C-test has been cast bythe fact that no one seems to be quite sure what it actually measures,but this ambiguity exists with regard to the cloze test as well and,after all, in the history of psychometrics there is a surprisingly largenumber of tests which are known to work reliably whereas nobodyknows exactly why and how (see, for example, aptitude tests ingeneral, including language aptitude batteries).

The great potential of C-testing led us to launch a researchprogramme at Eotvos University, Budapest aiming at validating thetest amongst Hungarian EFL learners. We intended to achieve twothings: 1) to replicate the very positive results reported in theliterature to see to what extent they could be generalized, and 2) tolook into some uncharted or controversial areas in C-testing. Wewere especially interested in the following issues:a) the concurrent validity of the C-test and the aspects of language

proficiency it measures;b) a comparison of the cloze with a C-test;c) the relationship between text difficulty, reliability and measure-

ment accuracy;d) the role of content and structure words;e) measurement accuracy in samples of different homogeneity and

proficiency level;f) the effect of different scoring methods.

In order to investigate these issues, we needed a very reliable andvalid criterion measure to serve as a reference point and we tookevery care to produce one. Through the co-operation of 24 col-leagues and graduate students, we formed a composite criterionscore from the results of four different language tests/test batteries:our department's regular test battery (three subtests), a standar-dized American test battery (two subtests), an oral interview and a

Zoltdn Dornyei and Lucy Katona 189

cloze test. After we had completed the main study with universitystudents, we re-administered the C-test to secondary school pupilsto check whether our results still held true for a different samplewith a lower proficiency level.

II Method

1 SubjectsIn the first (and main) study the subjects were 102 first-year Englishmajors at the Department of English, Eotvos University, Budapest.They represented more than two-thirds of the whole first year. Asthey had all had to take an entrance exam to be accepted at theDepartment, they formed a rather homogeneous sample with anaverage language proficiency of above the intermediate level. Thefollow-up study involved four groups of secondary pupils (N=53) intwo different schools; their proficiency level was well under that ofthe first sample.

2 The language tests comprising the criterion measureFour different language tests were administered to all the subjects inthe first study to form a composite criterion measure; for thesecondary school sample only the second battery in the list belowwas administered:

Department Proficiency Test: which is developed every year forthe purpose of testing first year students in order to screen out thosewho are not proficient enough to enter the second year. This batterytakes about 90 minutes to complete and consists of three parts:a) vocabulary test (20 multiple-choice items);b) grammar test - sentence-transformation tasks, similar to the

ones in the Cambridge First Certificate test (20 items);c) listening comprehension test (two dialogues, 20 multiple-choice

and true-or-false items).

Test of English for International Communication (TOEIC): whichis a standardized multiple-choice test offered by the EducationalTesting Service, Princeton as a nonacademic counterpart ofTOEFL; it is considered to be a 'highly reliable and valid measure ofEnglish' (Perkins, 1987: 82) and it has been validated for HungarianEFL learners (Dornyei, 1990). TOEIC consists of two sections,


each containing 100 items, and students have two and a half hours tocomplete it. These sections are:a) listening comprehension;b) reading, which contains varied items focusing also on language

accuracy.

Oral interview, conducted by two examiners, one of whom isalways a native speaker of English, lasting about 10-15 minutes.The students' achievement was rated according to accuracy, voca-bulary and fluency, on five-point scales where each point could bemodified by '+' and '—' marks, and these are then summarized toform a composite score.

Cloze test: (39 gaps with a deletion rate of five - see Appendix).

3 The development of the C-testThe development of the C-test was carried out as part of aspecialization course for graduate students. Here we had the sameexperience as Grotjahn (1987), who pointed out that it could provedifficult to find texts which could discriminate amongst advancedlearners, especially if we are aiming at the ideal 50% solution rate.We considered more than 15 texts, and finally four were included inthe final C-test, containing 24, 17, 21 and 19 gaps respectively (seeAppendix). The internal consistency reliability of the four extractsregarded as superitems, assessed by means of Cronbach a was .75 inthe university student sample and .77 in the secondary pupil sample.

4 ProceduresThe Department Proficiency Test, TOEIC and the Oral Interviewwere administered centrally as part of the university students'official end-of-term exams. The cloze and the C-test were adminis-tered six weeks later, at the beginning of the following term, duringthe students' language practice classes. In the secondary schoolsample, the C-test and TOEIC were administered in the pupils'regular English language classes. The data were processed using theStatistical Package for the Social Sciences (SPSS). The criterionmeasure was created by factor analysing the four different languagetests and taking the regression-method factor score generated bySPSS from the one-factor solution (see below for more details). Thevarious data procession analyses (described below in detail) in-volved correlation and factor analyses, and Mest procedures forvarious subgroups.


III Results and discussion

1 Concurrent validityUsing a minimum-eigenvalue criterion of 1.0, the principal compo-nent factor analysis of the four different language tests yielded aone-factor solution accounting for over 50% of the variance (Table1, first column). This was indeed expected, as the subjects in thesurvey were all preselected university English majors whose diffe-rent language skills were fairly evenly developed and thus could besufficiently represented by a unitary factor. As can be seen in thetable, the various language measures loaded on this factor evenly,thus it was termed General Language Proficiency.

Table 2 (first two columns) presents the correlations of GeneralLanguage Proficiency and the administered language tests with theC-test in the university and secondary school samples. Two compo-site scores, Department Proficiency Test Total and TOEIC Total,were computed by adding up the standardized component scores ofthe batteries in question.

As can be seen in Table 2, the C-test has highly sufficient positivecorrelations in all the composite language proficiency measures(General Language Proficiency, Department Proficiency Test Totaland TOEIC Total), and has significant positive correlations with allthe language tests. This indicates that the C-test is a highlyintegrative language test which measures global language profi-ciency. The only area in our study where the C-test appeared to beless efficient is in the testing of grammar; this can be explained bythe fact that deletions in the C-test are not performed at sentencelevel but at word level. Little and Singleton had similar findings:

There is some evidence that in filling C-test slots our subjects tended to givepriority to a ready lexical solution over morpho-syntactic and more generalsemantic issues. Intuitively this is in line with the priorities that we exercise inspontaneous communication (Little and Singleton, 1990:14).

Table 1 Factor analyses of the language tests administered (Analysis 1: all thelanguage tests except the C-test have been entered; Analysis 2: all the language testsexcept the C-test and the cloze test have been entered)

Type of testDepartment Proficiency Test

vocabulary sectiongrammar sectionlistening section

TOEIC Listening SectionReading Section

Oral InterviewCloze Test

Analysis 1Factor 1

.72

.67

.63

.80

.73

.75

.66

Analysis 2Factor 1

.75

.70

.63

.80

.73

.75


Table 2 Correlations of General Language Proficiency and the administered languagetests with the C-test in both samples, and with the cloze test in the university sampleType of test

General Language ProficiencyGeneral Language Proficiency (B)1

Department Proficiency TestVocabulary SectionGrammar SectionListening SectionTotal2

TOEIC Listening SectionReading SectionTotal2

Oral InterviewCloze Test

C-testUniv.

sample(N=102).57***.56***

.38***

.25*

.33***

.43***

.51***

.54***

.62***

.43***

.38***

Sec.sample(N=53)

——

————

.53***

.58***

.62***——

Cloze testUniv.

sample(N=102)

—.53***

.33*

.26*

.38*

.43*

.45*

.44*

.52*

.46*

#

—

Notes: 1 General Language Proficiency (B) has been computed in the same way asGeneral Language Proficiency except that the cloze scores have not been entered inthe factor analysis (see Table 1, Column 2).2The Total scores were computed by adding the standardized component scores.*p<.05; **p<.01; ***p<.001

On the other hand, as confirmed by the high correlations with themore general proficiency measures, the C-test does call the testee'sexpectancy grammar into action in a similar way to the cloze test.Indeed, based on the highly significant correlations with the twomost general measures - General Language Proficiency and TOEICTotal (.57 and .62 respectively) - we may conclude that the C-test isa very efficient integrative language test. Furthermore (as will bedescribed later) after piloting the C-test and considering only themost efficient extracts (1,2 and 3), these correlations became evenhigher (.61 and .65 - see Table 5).

2 Comparing the results of the cloze and the C-testTable 2 (third column) presents correlations between the cloze testand the other language measures. In this case, however, thecomposite measure termed General Language Proficiency could notbe used as it actually contained the cloze results and therefore thecorrelation would have been disproportionately high. In order to beable to compare the cloze and the C-test we computed a secondgeneral language proficiency measure, termed General ProficiencyLanguage (B), in the same way as we computed General LanguageProficiency, but this time the cloze results were not entered into thefactor analysis (see Table 1, Column 2). The correlations in Table 2


Table 3 Mean scores, standard deviations and the difficulty rates of the C-test and itsfour extracts in the two samples1______________________________Extracts University sample Secondary school sample

C-testExtract 1Extract 2Extract 3Extract 4

X

58.519.211.914.013.3

SO

8.42.22.93.42.6

Diff. rate.72.80.70.67.70

X

29.913.42.68.25.8

SO

7.83.41.92.82.0

Diff. rate

.37

.56

.15

.39

.31

Note: 'As could be expected all the intersample differences in the mean scores andthe difficulty rates are highly significant.

suggest that the C-test provides as good estimates of languageproficiency as the cloze, if not better. In fact, the differences in thecorrelations with TOEIC imply superior discriminatory power onthe part of the C-test. (This will be confirmed later when analysingresults obtained in more homogeneous subgroups.)

3 Text difficulty, reliability and measurement accuracyKlein-Braley and Raatz state that 'it has been repeatedly shown thateven tests which are far too difficult or far too easy for the targetgroups will still produce acceptable reliability and validity coef-ficients' (1984:140). Our results support this claim. Table 3 presentsthe mean difficulty rates of the C-test in the two samples (asobtained by dividing the mean scores by the number of gaps). Theseshow that the test proved to be fairly easy for the university sampleand rather difficult for the secondary school sample. However, ashas been mentioned before, the internal consistency coefficientsfound in the two samples were very consistent, .75 and .77 respec-tively.

Let us consider now the difficulty of the extracts. Klein-Braley(1985) reports attempts to define objective predictor measures fortext difficulty: The two measures which emerged as the bestpredictors were the type-token ratio (number of different words/total number of words) and average sentence length. In a review ofthese findings, Carroll states: 'it is not clear, however, that theseestimates are really generalizable over varied samples of examineesand texts; one wonders whether the extensive investigation thatmight be required would be worth the effort' (1987:102). Ourfindings support Carroll's doubts. Table 4 presents Mest statisticstesting the significance of the differences between the extractdifficulties in the two samples. For the university students, Extract 1proved to be significantly easier than the other three texts, Extract 3


Table 4 Paired samples f-test statistics comparing the difficulty rates of the fourextractsExtractpairsExtract 1Extract 2Extract 1Extract 3Extract 1Extract 4Extract 2ExtractsExtract 2Extract 4Extract 3Extract 4

X

UnS SecS.80.70.80.67.80•70.70.67

.70

.70

.67

.70

.56

.15

.56

.39

.56

.31

.15

.39

.15

.31

.39

.31

SDUnS SecS.09.17.09.16

.09

.14

.17

.16

.17

.14

.16

.14

.14

.11

.14

.14

.14

.11

.11

.14

.11

.11

.14

.11

t-valueUnS SecS

6.61

10.33

7.58

1.93

.01

-1.98

24.74

8.82

12.07

-13.88

-9.91

4.59

ProbabilityUnS SecS

.000

.000

.000

.057

.992

.051

.000

.000

.000

.000

.000

.000

Notes: UnS = University student sample; SecS = Secondary pupil sample

was the most difficult, and Extracts 2 and 4 were equally difficult.This is, however, not the case for the secondary school sample:there, Extract 2 proved to be by far the most difficult and Extract 3,which the university students found most difficult, turned out hereto be the second easiest. This seems to be at odds with Klein-Braley's conclusion, namely that 'the group of texts used in anyC-test remains more or less constant in terms of relative difficulty'(1985: 88).

Another and perhaps more important question is whether themeasuring capacity/discrimination index of the texts remains con-stant in different samples. Table 5 (first column) presents thecorrelations of the scores on the extracts and their various combina-tions with General Language Proficiency, as obtained in the uni-versity student sample. In the secondary school sample only the twoTOEIC subtests were administered besides the C-test and thereforetheir combination (TOEIC Total) will be treated as the languagecriterion measure. The fifth column of Table 5 presents the correla-tions between TOEIC Total and the various extract-combinations;in order to be able to compare the results obtained in the twosamples, the fourth column of Table 5 contains the same extract-TOEIC Total correlations for the university sample.

As can be seen in Table 5, there is a very consistent pattern ofextract-criterion measure correlations across the two samples andthe two criterion measures. Of the four extracts, Extract 1 appearsto measure language proficiency best, followed by Extract 2 andExtract 3; adding Extracts 2 and 3 to Extract 1 increases theaccuracy of the measurement. Extract 4, however, seems to be a


Table 5 Correlations of the four extracts and their various combinations a) withGeneral Language Proficiency in the whole university sample and in the high and lowability subsamples, and b) with TOEIC Total (the sum of the standardized scores of thetwo TOEIC subtests) in both samplesExtracts General Language Proficiency

(University student sample)TOEIC Total

Extract 1Extract 2Extract 3Extract 4Extracts 1 + 2Extracts 1 + 3Extracts 2 + 3Extracts 3 + 4Extracts 1 +4Extracts 2 + 4Extracts 1+2 + 3Extracts 1+3 + 4Extracts 1+2 + 4Extracts 2 + 3 + 4Extracts 1+2 + 3 + 4(complete test)

Whole Topsample half(N=102) (N=51).58*.46*.46*.25*.59*.57*.55*.46*.48*.42*.61*.54*.53*.52*.57*

.36**

.22

.49***

.02

.33*

.50***

.46***

.37**

.18

.12

.47***

.39**

.22

.35*

.38**

Bottomhalf

(N=51).53***.32*.20.16.49***.39**.33*.23.45***.30*44»»*.38**.44***.32*.42**

Univ. Secondarysample sample(N=102) (N=53).61*.54*.46*.27*.66*.58*.60*.47*.51*.48*.65*.56*.59*.56*.62*

.62***

.52***

.44**

.24

.65***

.62* *

.55* *

.43*

.59* *

.44*

.64* *

.59* *

.61* *

.51***

.62***

Notes: *p<.05; **p<.01; ***p<.001

bad predictor of language proficiency, and its inclusion actuallyreduces the measurement accuracy of the instrument (thus ourC-test would be more efficient with only the first three extracts).These results point to the fact that even though the relative difficultyof an extract varies according to the proficiency level of the testees,the measuring ability of an extract is fairly stable. This implies thatonce an extract has proved to be efficient with one sample, it islikely to work well with different samples too; the results alsoconfirm Klein-Braley and Raatz's (1984) claim that even if a C-testis too easy or difficult for a particular target group, it will produceacceptable validity.

How can differences in measuring capacity be explained? Table 6summarizes the main parameters of each of the four extracts: themean sentence length (both for the extract as a whole and for themutilated sentences only), the type-token ratio, the difficulty rate(for the two samples), and finally the rank order of measuringcapacity (for the two samples) as defined by the extract-criterionmeasure correlations. We have already seen that the difficulty rate isnot directly related to the measuring capacity and, as the tableshows, neither is the type-token ratio. The two sentence lengthmeasures, however, produce a pattern which is very consistent withthe measuring capacity: Extract 1, which measures language pro-

2sS?&•».i^1

Table6 Summary of different parameters of the four extracts ("̂Extract

Extract 1Extract 2Extract 3Extract 4

Rank order of ^Mean sentence length Type-token Difficulty rate meas. capacity «•»

ratio §Whole Mutilated Univ. Secondary Univ. Secondary §extracts sentences sample sample sample sample ^3

9.8 10.0 .79 .80 .56 1 1 S3.11.8 17.5 .73 .70 .15 2-3 2 £c13.3 11.3 .78 .67 .39 2-3 3 K18.3 19.5 .73 .70 .31 4 4 £J

S'3

r5


ficiency best, contains the shortest sentences, whereas Extract 4, theleast efficient of the four texts, contains the longest sentences. Thetwo interim texts do not show a clear pattern with respect to the twosentence length measures: even though Extract 3 has longer sen-tences as a whole, the actual mutilated sentences are shorter thanthose in Extract 2; this may be the reason why the two texts turnedout to be more or less equal measures of language proficiency in theuniversity student sample.

Thus we may conclude that in our two samples mean sentencelength was the best predictor of the efficiency of a text as a measureof language proficiency: the shorter the sentences were, the betterthe text measured general language proficiency. Klein-Braley (1985)gives a detailed overview of studies which concluded that averagesentence length was a good index of syntactical complexity and alsoof the text's cognitive loading. This is certainly true about ourextracts. In trying to find a fourth, that is, most difficult, text for ourC-test, we selected a highly complex, abstract passage which wasdifficult to understand even without any mutilations - and, as itturned out, the text did not work very well. As for the otherextreme, the first and easiest text was a straightforward narrative inthe simple past - and this simplicity worked beautifully. All thispoints to the fact that texts which have a relatively simple structureboth in terms of syntax and content complexity are best suited tobeing used for C-testing. This does not mean, however, that theycannot be difficult, but the difficulty of a text should be a function ofthe language and especially of the vocabulary load the text contains.This issue will be further investigated below with regard to thestructure and content words in the extracts.

4 Content words and structure wordsIt has been shown repeatedly that truncated structure words aresignificantly easier to restore in a C-test than truncated contentwords (Klein-Braley, 1985). This was true in our investigation too,as demonstrated by f-test statistics (t = 23.35, p< .001). The differ-ence in word type difficulty was assumed to have important bearingson the measuring capacity of the C-test and the individual extracts,and therefore we analysed to what extent the two word types relateto the accuracy of measurement in the two samples; separateanalyses were also computed for the high and low ability halves ofthe university student sample based on the General LanguageProficiency measure. Table 7 contains the correlations of the scoreson the truncated structure and content words (broken down by the


extracts) with the language criterion measures and the total C-testscore.

The correlations presented in the table help to shed light on whythe extracts behaved as they did (i.e., which were their weak partsand which proficiency range in particular was affected by these), butthe most important feature of the pattern is that content andstructure words behave strikingly differently in the two samples.Amongst the university students, for whom the C-test was fairlyeasy, the content words were a better measure of language pro-ficiency than structure words. In fact, the latter actually reduced themeasuring capacity of the test (i.e., if we ignore structure words, weget higher correlations with General Language Proficiency). On theother hand, amongst the secondary school pupils, for whom theC-test was too difficult, we find the opposite tendency: the structurewords separately give a better estimate of the testees' languageproficiency than the whole test, and here it is the content wordswhich reduce the measurement accuracy of the test. In the lowability half of the university sample, which is somewhere in betweenthe university and the secondary school samples in terms of averageproficiency level, the increasing importance of the structure wordscan already be noticed, especially in Extract 2. The last fourcolumns of Table 7 present correlations between the whole C-testand its parts. Here again we find that in the secondary school samplethe structure words contribute more to the total C-test score thanthe content words.

We believe that the tendency described above is a typical exampleof the remarkably versatile character of the C-test: we could say thatit has 'something to offer to everybody', whether university lan-guage majors or average secondary school learners. The specialdeletion technique in C-test appears to allow for many differentapproaches to achieving item solutions at various testee proficiencylevels, tapping on a wide range of language areas and cognitiveprocessing skills. A good example of this was provided by Cohen,Segal and Bar-Siman-Tov, who found that 'because half the wordwas given, students who did not understand the macro-contextcould still mobilize their vocabulary skills adequately to fill in theappropriate discourse connector without indulging in higher-levelprocessing' (1984: 225). In a study using thinking-aloud and re-trospective methods, Feldmann and Stemmer (1987) also foundvarious problem-solving student behaviours and succeeded in de-fining several different bottom-up and top-down processingstrategies which are called into action by subjects doing C-tests. Thisrange of possible strategies to be applied explains why the test worksin samples of various proficiency levels, and this is also the reason

Table 7 Correlations of scores on the truncated structure and content words (broken down by the four extracts) with the language criterionmeasures and the total C-test scores as obtained in the two samples

Gen. Lang. Proficiency(University sample)

Whole Top Bottomsample half half(N=102) (N=51) (N=51)

Whole C-testContent Words(N=44)Structure words(N=37)

Extract 1Content Words(N=10)Structure words(N=14)

Extract 2Content Words

Structure words(N=6)

ExtractsContent Words(N=12)Structure words(N=9)

Extract 4Content Words

Structure words(N=8)

.61***

.34***

.62***

.34***

.48***

.23*

.53***

.20*

.27**

.01

.47**

.11

.40**

.17

.26

-.05

.56***

.20

.04

-.18

.42**

.30*

.48***

.49***

.40**

.05

.22

.11

.21

-.04

TOEIC Total(sec. sch.sample)(N=53)

.44**

.67***

.25

.66***

.27*

.53***

.40**

.36**

.30*

.04

UnWh

.97***

.82***

.72***

.56***

.77***

.54***

.78***

.51***

.67***

.31**

C-test

UnTop UnBot

.96***

.83***

.64***

.62***

.74***

.38**

.71***

.63***

.64***

.40**

.97***

.78***

.66***

.48***

.74***

.56***

.75***

.38**

.66***

.18

SecS

.88***

.91***

.64***

.69***

.52***

.71***

.70***

.66***

.62***

.32*

N2.ax

t)O:

••<

1

t-C

>!Sv|

&

. \ONotes: *p<.05; **p<.01; ***p<.001; UnWh = Whole university sample; UnTop= High ability university subsample; UnBot = Low abilityuniversity subsample; SecS = Secondary school pupil sample.


why the test is more 'user-friendly' than, for example, the cloze test,which is known to be consistently difficult and frustrating (Scott andMadsen, 1983).

5 Student proficiency range and sample homogeneityThe proficiency level of the university student sample ranged fromintermediate to advanced. To examine how well the C-test discri-minated in both tails of the distribution, the sample was divided intoa high and a low ability group, based on the General LanguageProficiency measure. This also allowed us to see how the C-testworked in extremely homogeneous samples: by dividing the uni-versity students into two proficiency level groups we further reducedthe within-group variation in a sample which was already pre-selected to start with.

The second and the third columns of Table 5 present thecorrelations of the four extracts and their various combinations withGeneral Language Proficiency found in the two subsamples. As canbe seen in the table, Extract 4 again does not work very well witheither subsample. Extract 1, which is the easiest text, appears to bethe most efficient with the lower level group, and Extract 3, themost difficult text, works best with the high proficiency group. Thevarious combinations of the extracts produce a wide range ofcorrelations, with considerable differences between the correlationsin the two groups. The only combination which has correlationsover .40 in both groups is the sum of Extracts 1, 2 and 3, whichindicates that in order to cater for variations amongst languagetestees, and thus provide equal measurement accuracy in both tailsof a sample distribution, we should select extracts of variousdifficulties for a C-test.

The correlations in the two subsamples are depressed by thereduced variance, but it is worth noting how much more efficientlythe C-test measures language proficiency in the two subgroups thanthe cloze test, as shown in Table 8. In fact, the rather low reliabilityand validity of the classical cloze test in homogeneous samples wasone criticism put forward by Klein-Braley and Raatz (1984). TheC-test does indeed remedy this shortcoming to a certain extent.

6 Scoring methodsThere are three obvious ways of scoring the C-test: 1) only the exactsolutions are accepted; 2) every appropriate word is accepted; and3) either of the methods above combined with spelling mistakesbeing or not being tolerated. A rationale for allowing spelling


mistakes would be partly the wish to maintain face validity (studentswho fill a gap but commit a minor spelling mistake will feel that theirattempt is worth more than that of others who leave the gapunfilled), and partly the fact that integrative tests are trying tomeasure the ability to cope with reduced language redundancy andto activate one's expectancy grammar, and spelling is only marginal-ly related to these complex skills.

We believe that the first option, 'exact word scoring' is hard tojustify theoretically and therefore we decided to apply the secondmethod. However, we were soon forced to realize that the role ofspelling mistakes was more significant than we had originallyassumed. Table 9 contains the frequencies of the spelling errors inthe two samples: Almost 60% of the university students did commitspelling mistakes and more than 22% made more than one mistake.The secondary school pupils committed fewer spelling errors, buteven in this sample 32.1% of the students made at least one spellingmistake.

Table 10 presents the correlations between the language criterionmeasures (General Language Proficiency and TOEIC Total) andthe C-test score with and without the spelling mistakes. As can beseen, whether or not we tolerate spelling mistakes does not signi-ficantly affect the discriminatory power of the C-test, although someminor improvement can be achieved by tolerating spelling mistakes(in the university student sample there was an improvement of.0053, which disappeared when the coefficients were rounded). It is

Table 8 Correlations of General Language Proficiency (B) with the C-test and the clozetest in the top and bottom proficiency halves of the university sample

Top half (N=51) Bottom half (N=51)C-test Cloze C-test Cloze

General Language Proficiency (B) .44*** .25 .34* .25

/Votes: *p<.05;**p<.01;***p<.001

Table 9 Frequencies of spelling mistakes in the two samples_________No. of spelling mistakes Frequency of students Percentmade by one student

Univ.sample

0 411 382 183 34 15 1

Sec.sample

361412

Univ.sample40.2%37.3%17.6%2.9%0.9%0.9%

Sec.sample67.9%26.4%

1.9%3.8%


Table 10 Correlation between the language criterion measures and the C-test scorewith and without the spelling mistakes__________________________

C-testLanguage criterion measures Spelling mistakes Spelling mistakes

tolerated marked wrongUniv. Sec. Univ. Sec.

__ samp.___samp.___samp.___samp.General Language Proficiency .57 — .57 —TOEIC Total________________—_____.62_____—_____.61

Note: All the coefficients are highly significant

therefore up to the examiner to decide which scoring method ismore appropriate for the sample and purpose in question. In theanalyses reported in this study we have marked every appropriateword with a minor spelling mistake correct.

IV ConclusionThe various analyses reported in this study produced the followingresults:

1) The C-test proved to be a reliable and valid instrument amongstHungarian EFL learners. The high concurrent validity coef-ficients were especially noteworthy for two reasons: a) theC-test we used was previously untried, and by submitting it tostatistical evaluation and selecting only the most efficientextracts, the coefficients could be further improved; b) thecriterion measure we used was very carefully developed andinvolved several language proficiency measures including anoral measure.

2) Satisfactory reliability and validity coefficients were obtainedeven when the test was too easy or too difficult for the targetgroup, which confirmed earlier results in other countries.

3) The C-test proved to be a highly integrative language testinginstrument, assessing general language proficiency veryefficiently. This again is consistent with previous findings.

4) The C-test appeared to be a somewhat better measure ofgeneral language proficiency than the cloze test; this superioritybecame especially featured in more homogeneous samples.

5) In our study, the relative difficulty of the individual extractscomprising the C-test varied according to the target group; themeasuring ability/discrimination index of the extracts, however,appeared to be fairly stable across samples.

6) The best predictor of the measuring ability of a text in our studywas mean sentence length; it was argued that tests which have a


relatively simple structure both in terms of syntax and contentcomplexity are best suited to being used for C-testing.

7) Similarly to other studies, our results also suggested that aC-test should include extracts of different difficulties to providean accurate measurement in both tails of the sample distribu-tion.

8) Truncated structure words were confirmed to be easier toreconstruct than truncated content words.

9) We found that with samples of higher proficiency levels themeasurement of language proficiency was primarily due to therole played by content words, and structure words actuallyreduced measurement accuracy. In lower level samples, howev-er, structure words gained an increasing importance to a pointwhen the role of the two word types was completely reversed.

10) The tolerance of spelling errors, which increases the C-test'sface validity, did not reduce the concurrent validity of the test;in fact, if anything, a slight improvement in measurementaccuracy was found when the spelling mistakes were notmarked wrong.

In sum, the results we obtained, exceeded our original expecta-tions. The C-test proved to be a highly integrative and versatilemeasuring instrument, working well in samples of various difficultyand homogeneity levels. We have argued that this is due to thespecial deletion technique, which samples a representative set oflanguage items and allows for several different approaches toachieving item solutions at various testee proficiency levels, thustapping on a wide range of language areas and cognitive skills.

With respect to more practical questions, it was difficult to believethat a test which is significantly easier to design and score than mostother testing instruments, did as good a job - in some ways better -than the latter. A major objective of research on language testing isto increase the cost-effectiveness of the assessment; our conclusionabout the C-test is that not only is it a reliable and valid measure ofgeneral language proficiency, but it is also one of the most efficientlanguage testing instruments in terms of the ratio between resourcesinvested and measurement accuracy obtained.

V ReferencesAlderson, J.C. 1979: The cloze procedure and proficiency in English as a

foreign language. TESOL Quarterly 13, 219-26.Carroll, J.B. 1987: Review of 'Klein-Braley, C. and Raatz, U., editors,

C-Tests in der Praxis. Fremdsprachen und Hochschule, AKS-Rundbrief 13/14, Bochum: Arbeitskreis Sprachenzentrum [AKS]'.Language Testing 4, 99-106.


Cohen, A.D., Segal, M. and Bar-Siman-Tov, R. 1984: The C-Test inHebrew. Language Testing 1, 221-25.

Ddrnyei, Z. 1990: Local validation of TOEIC in Hungary. The Reporter,(TOEIC News International) 4, Princeton: Educational Testing Ser-vice, 4.

Feldmann, U. and Stemmer, B. 1987: Thin__ aloud a_ retrospectiveda_ in C-te_ taking: diffe languages - diff__ learners - sa—approaches? In Faerch, C. and Kasper, G., editors, Introspection insecond language research, Clevedon: Multilingual Matters, 251-67.

Grotjahn, R. 1987: How to construct and evaluate a C-Test: a discussion ofsome problems and some statistical analyses. In Grotjahn, R.,Klein-Braley, C. and Stevenson, D.K., editors, Taking their measure:the validity and validation of language tests, Bochum: Brockmeyer,219-53.

Klein-Braley, C. 1985: A cloze-up on the C-Test: a study in the constructvalidation of authentic tests. Language Testing 2, 76-104.

Klein-Braley, C. and Raatz, U. 1984: A survey of research on the C-Test.Language Testing 1, 134-46.

Little, D. and Singleton, D. 1990: The C-test as an elicitation instrument insecond language research. Paper presented at AILA 1990, Thessalo-niki, Greece, 16-22 April.

Perkins, K. 1987: Review of the 'Test of English for InternationalCommunication'. In Alderson, J.C., Krahnke, K.J. and Stansfield,C.W., editors, Reviews of English language proficiency tests,Washington, DC: TESOL, 81-83.

Raatz, U. 1985: Tests of reduced redundancy - The C-Test, a practicalexample. In Klein-Braley, C. and Raatz, U., editors, C-Tests in derPraxis. Fremdsprachen und Hochschule, AKS-Rundbrief 13114,Bochum: Arbeitskreis Sprachenzentrum [AKS], 14-19.

Raatz, U. and Klein-Braley, C. 1985: How to develop a C-test. InKlein-Braley, C. and Raatz, U., editors, C-Tests in der Praxis.Fremdsprachen und Hochschule, AKS-Rundbrief 13114, Bochum:Arbeitskreis Sprachenzentrum [AKS], 20-22.

Scott, M.L. and Madsen, H.S. 1983: The influence of retesting on testaffect. In Oiler, J.W. Jr., editor, Issues in language testing research,Rowley, MA: Newbury House, 270-79.

VI Appendix

The cloze test used in the studyIn olden days when a glimpse of stocking was looked upon assomething far to shocking to distract the serious work of an office,secretaries were men. Then came the first . . . . . War and the male. . . . . were replaced by women. . . . . . man's secretary became his. . . . . servant, charged with remembering . . . . . wife's birthday andbuying . . . . . presents; taking his suits . . . . . the dry-cleaners;


telling lies . . . . . the telephone to keep . . . . . he did not wish . . . . .speak to at bay; . . . . . . of course, typing and . . . . . and takingshorthand. Now . . . . . this may be changing . . . . . . The microchipand high . . . . . is sweeping the British . . . . . taking with it much. . . . . the routine clerical work . . . . . secretaries did. 'Once office. . . . . takes over generally, the . . . . . of the job will . . . . . againbecause it will . . . . . only the high-powered work-. . . . . then menwill w a n t . . . . . do it again'. That . . . . . said by one of . . . . . maleexecutives of one . . . . . the biggest secretarial agencies . . . . . thiscountry. What he . . . . . predicted is already under . . . . . in theU.S. One . . . . . described to me a . . . . . temporary job placing men. . . . . secretarial jobs in San . . . . . . she noted that all . . . . . menshe dealt with . . . . . to be gay so . . . . . that is just a . . . . . twist tothe old . . . . . . Over here, though, there are men coming onto thejob market as secretaries.

The C-test used in the studyExtract 1 One cool autumn evening, Bob L., a young professional,returned home from a trip to the supermarket to find his computergone. Gone! all sorts of crazy thoughts raced through his mind: Hadit been stolen? Had it been kidnapped? He searched hte house for aclue until he noticed a small piece of printout paper stuck under amagnet on his refrigerator door. His heart sank as; he read thissim£le message: CAN'T CONTINUE, FILE CLOSED, BYE.

Extract 2 There is a third factor besides farming and herding in thespread of man-made deserts: deforestation. The progressive des-truction of the Third World's stock of trees is damaging not onlj indry^ regions: everywhere it occurs it can accelerate the decay of thesoil and reduce its capacity to feed people. It can reduce rainfall andlead to drought.

Extract 3 There are certain things which no student can do withoutand others which may not be as necessary as you thought. It may beworth1 considering some small hints. You may find yourself in needof electrical appliances such as light bulbs, adaptors or plugs. Thesecan be obtained from many places. GILL is a good hardware shopand trying to find it is a challenge. It is hidden in a little alley leadingoff HigFStreet called WheatsHeaf Yard.

'Due to a copying fault the letter 'o' was not legible on some copies and therefore therestoration of this word was not included in the final score.


Extract 4 The private conscience of the leader - father than hispublic responsibilities - becomes the focal point of politics. Internalcriteria - possession of, devotion to, and standing up for private

judgetuents disappear, and we are lett with a political leader determiningpolicy on the basis of compatibility with his private principles. Fromthis perspective we can better understand why Goldwater votedagainst the Civil Rights Act of 1964.

___——————— • ___' ——— ' 1~S i ______ «.

principles - become the standards of political judgement. Consti-tuents disappear, and we are left with a political leader determining

validation of the c-test amongst hungarian efl learners · 192 validation of the c-test amongst...

Documents