do curriculum-based external exit exam systems enhance student

Do Curriculum-Based External Exit Exam Systems Enhance Student Achievement?

John Bishop CPRE Research Report Series RR-40 Consortium for Policy Research in Education University of Pennsylvania Graduate School of Education

8 Copyright 1998 by the Consortium for Policy Research in Education

Do Curriculum-Based External Exit Exam Systems Enhance Student Achievement? Bishop

CPRE Research Report Series, RR-40

Contents

Biography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Author’s Note. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

What’s So Different About a Curriculum-Based External Exit Exam System?. . . . . . . . . . . 1

How are CBEEES Hypothesized to Increase Achievement?. . . . . . . . . . . . . . . . . . 3

Do CBEEES Increase Achievement?: A Look at the Evidence. . . . . . . . . . . . . . . . . 6

Third International Mathematics and Science Study. . . . . . . . . . . . . . . . . . 6Analysis of the 1991 International Assessment of Educational Progress. . . . . . . . . . 11The Impact of New York State Regents Examinations . . . . . . . . . . . . . . . . 16

Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Appendix A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

End Notes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32


CPRE Research Report Series, RR-40 i

The preparation of this paper was made possible by support from the Center for Advanced HumanResource Studies and the Consortium for Policy Research in Education (CPRE), funded by a grant (No.OERI-R308A60003) from the National Institute on Educational Governance, Finance, Policymaking andManagement, Office of Educational Research and Improvement, U.S. Department of Education. Thispaper has not undergone formal review or approval of the faculty of the New York State School ofIndustrial and Labor Relations (ILR). It is intended to make results of Center research, conferences, andprojects available to others interested in human resource management in preliminary form to encouragediscussion and suggestions. The findings and opinions expressed in this report are those of the authorand do not reflect the position or policies of the U.S. Department of Education, ILR, CPRE, or its insti-tutional partners.

Biography

Author’s Note

John Bishop is an Associate Professor and Chair of the Department of Human Resources at CornellUniversity. His areas of research include education reform and work force preparedness, employerbehavior and personnel, program evaluation and government training programs, income maintenancepolicy, and comparative human resource institutions and policies. He is a member of the Safety NetStudy Group advising New York State’s Commissioner of Education on how to minimize the number ofstudents who fail to pass the New Regents examinations.


CPRE Research Report Series, RR-40 1

Introduction

Two presidents, the National GovernorsAssociation and numerous blue ribbonpanels have called for the developmentof state content standards for core

subjects and examinations that assess the achieve-ment of these standards. The CompetitivenessPolicy Council, for example, advocates that“external assessments be given to individualstudents at the secondary level and that the resultsshould be a major but not exclusive factor qualify-ing for college and better jobs at better wages(1993, p. 30).” The American Federation ofTeachers advocates a system in which:

Students are periodically tested on whetherthey’re reaching the standards, and if they arenot, the system responds with appropriateassistance and intervention. Until they meetthe standards, they won’t be able to graduatefrom high school or enter college (AFT 1995p. 1-2).

It is claimed that curriculum-based external exitexam systems (CBEEES), based on world classcontent standards will improve teaching andlearning of core subjects. What evidence is therefor this claim? New York’s Regents Exams are anexample of such a system. Do New York studentsoutperform students with similar socio-economicbackgrounds from other states? Outside theUnited States such systems are the rule, not theexception. What impacts have such systems hadon school policies, teaching and student learning?

What’s So Different About aCurriculum-Based External ExitExam System?

Skeptics point out that American students alreadytake lots of standardized tests. They ask, “Whyshould a curriculum-based external exit examina-

tion system significantly improve incentives andlearning?” Advocates claim that CBEEES haveuniquely powerful incentive effects because theyhave the following six characteristics.

1. CBEEES produce signals of student accom-plishment that have real consequences forthe student.

2. CBEEES define achievement relative to anexternal standard, not relative to otherstudents in the classroom or the school.Fair comparisons of achievement acrossschools and across students at differentschools are possible. Costrell’s (1994) analy-sis of the optimal setting of educationalstandards concluded that more centralizedstandard-setting (state or national achievementexams) results in higher standards, higherachievement and higher social welfare thandecentralized standard setting (such as teachergrading or school graduation requirements).

3. CBEEES are organized by discipline andkeyed to the content of specific coursesequences. This focuses responsibility forpreparing students for particular exams on asmall group of teachers.

4. CBEEES signal multiple levels of achieve-ment in the subject. If an exam generatesonly a pass-fail signal, the standard will haveto be set low enough to allow almost everyoneto pass and this will not stimulate the greatbulk of students to greater effort (Kang 1985;Costrell 1994).

5. CBEEES cover almost all secondary schoolstudents. Exams for a set of elite schools,advanced courses or college applicants willinfluence standards at the top of the verticalcurriculum, but will probably have limitedeffects on the rest of the students. The schoolsystem as a whole must be made to acceptresponsibility for how students do on the


2 CPRE Research Report Series, RR-40

exams. A single exam taken by all is notessential. Many nations allow students tochoose the subjects in which they will beexamined and offer high and intermediatelevel exams in the same subject.

6. CBEEES assess a major portion of whatstudents studying a subject are expected toknow or be able to do. It is not essential,however, that the external exam assess everyinstructional objective. Teachers can be givenresponsibility for evaluating dimensions ofperformance that cannot be reliably assessedby external means.

Commercially prepared achievement tests such asthe CAT, CTBS, ITBS, and ITED are not curricu-lum-based external exit exams because they failrequirement number one noted above (as well asseveral others). Students have no stake in doingwell on these tests. Where stakes are attached toresults, it is teachers and school administratorswho experience the consequences, not individualstudents.

The minimum competency exams that manyAmerican states require students pass beforegraduation are not CBEEES because they failrequirements numbers three and four. Minimumcompetency exams are generally first taken inninth and tenth grade and most students pass onthe first sitting. High school transcripts indicateonly whether the student eventually passes thetest, not achievement levels above the minimum.For the great majority of students who pass on thefirst try, therefore, the tests no longer stimulatestudy. Incentive effects are focused on the smallminority who fail on the first try and must repeatthe test. Minimum competency exams can be auseful part of a CBEEES, but other more demand-ing curriculum-based exams that signal higherlevels of performance are essential.

The requirement (number four) that a CBEEEsignal different levels of achievement— not justwhether the student has achieved a minimum— isessential because it has major effects on theincentive effects of exams. Students differ dra-matically in their levels of achievement by age 13.On the National Assessment of EducationalProgress, seven to nine percent of 13 year-olds arefour or more grade-level equivalents behind theirage mates and 15 to 17 percent are four or moregrade-level equivalents ahead. When achieve-ment differentials among students are this large,incentives for effort are stronger for most studentsif the full range of achievement is signaled ratherthan whether the individual has just passed someabsolute standard. When a test generates only apass-fail signal, many students pass withoutexertion and are not stimulated to greater effort bythe reward for passing. Some of the least well-prepared students will judge the effort required toachieve the standard to be too great and thebenefits too small to warrant the effort. They giveup on the idea of meeting the standard. Fewstudents will find the reward for exceeding asingle absolute cutoff an incentive for greatereffort (Kang 1985). Costrell agrees: “The case forperfect information [making scores on externalexaminations available rather than just whetherthe individual passed or failed] would appear tobe strong, if not airtight: for most plausible de-grees of heterogeneity, egalitarianism, and poolingunder decentralization, perfect information notonly raises Gross Domestic Product, but alsosocial welfare.” (1994, p. 970)

The SAT-I reasoning tests are not curriculum-based external exit exams because they do notmeet requirements numbered five and six. SAT-Itests fail to assess most of the material— history,science, economics, civics, literature, foreignlanguages and the ability to write an essay— thathigh school students are expected to learn. Fromthe beginning the SAT was designed to minimizebackwash effects on teaching and student studyhabits. Indeed, Richard Gummere, Harvard



College’s admissions director when the machinescored multiple-choice Scholastic Aptitude Test(SAT) replaced the curriculum-based essay styleCollege Board Examinations, was very candidabout why the SAT had been adopted: “Learningin itself has ceased to be the main factor [incollege admissions]. The aptitude of the pupil isnow the leading consideration.” (Gummere, 1943p. 5)

The subject specific SAT-II achievement tests failrequirements number one and five. Stakes arevery low— few colleges consider SAT-II results inadmissions decisions, and few students take them.In 1982-83 only six percent of SAT-I test takerstook a science SAT-II and only three to fourpercent took a history or foreign language SAT-IItest. Schools do not assume responsibility forpreparing students for SAT-II tests.

The Advanced Placement (AP) examinations arethe single exception to the generalization that theU.S. lacks national curriculum-based external exitexaminations. Although growing rapidly, AP is

still a very small program. In 1995 only 3.2percent of juniors and seniors took AP english orAP history exams and only 2 percent took APcalculus or science exams (National EducationGoals Panel, 1995). Low participation means thatAP exams fail requirement number five and are,consequently, not a CBEEE system. They can,however, serve as a component of a larger system.

How are CBEEES Hypothesizedto Increase Achievement?

Curriculum-based external exit exam systems(CBEEES) fundamentally change the signaling ofstudent achievement, and by doing so they trans-form the incentives faced by students, parents,teachers and school administrators. Consequently,CBEEES hypothesized to influence the resourcesmade available to schools and the priorities ofschool administrators, teacher pedagogy, parentalencouragement and student effort. The manypaths by which CBEEES influence studentachievement are illustrated in Figure 1.

Figure 1: How CBEEES Influence Student Achievement

StudentAchievement

Student Effort

PeerPressureCurriculum-

Based ExternalExit ExamSystem

SchoolResources

Culture,Productivity,Parental SES

Administrator Priorities

Teacher Pedagogy & Standards



Impact on Students: CBEEES improve thesignaling of academic achievement. As a result,colleges and employers are likely to give greaterweight to academic achievement when they makeadmissions and hiring decisions, so the rewardsfor learning should grow and become more vis-ible. CBEEES also shift attention toward mea-sures of absolute achievement and away frommeasures of relative achievement such as classrank and teacher grades. By doing so, CBEEESameliorate the problem of peer pressure againststudying.

Interviews I conducted during 1996 and 1997 withmiddle school students in Collegeville, a smallcity dominated by two universities, indicate thatmost students (males especially) internalize anorm against “sucking up” to the teacher. Howdoes a student avoid being thought a “suck-up?”He avoids making eye contact with the teacher;does not hand in homework early for extra credit;does not raise his hand in class too frequently; andtalks or passes notes to friends during class (sig-naling that you value friends more than yourreputation with the teacher). Steinberg, Brown andDornbush similarly conclude, “The adolescentpeer culture in America demeans academic suc-cess and scorns students who try to do well inschool.” (1996, p.19).

Why are the studious called suck-ups, dorks andnerds or accused of “acting white”? In part,because exams are graded on a curve, their studyeffort make it more difficult for others to get topgrades. When exams are graded on a curve orwhen college admissions are based on class rank,joint welfare is maximized if no one puts in extraeffort. Side payments (friendship and respect) andpunishments (ridicule, harassment and ostracism)enforce the cooperative don’t study solution. If,by contrast, students are evaluated relative to anoutside standard, they no longer have a personalinterest in getting teachers off track or persuadingeach other to refrain from studying. Peer pressuredemeaning studiousness should diminish.

Impact on School Administrators: When there isno external assessment of academic achievement,students and their parents benefit little fromadministrative decisions that support higherstandards, more qualified teachers or a heavierstudent work load. The immediate consequencesof such decisions are all negative: higher taxes,more homework, having to repeat courses, lowergrade point averages, complaining parents, agreater risk of being denied a diploma.

When student learning is not assessed externally,the positive effects of choosing academic rigor arenegligible and postponed. If college admissiondecisions are based on class rank, GPA andaptitude tests, and not on externally assessedachievement in secondary school courses, up-graded standards will not improve the collegeadmission prospects of next year’s graduates.Graduates will probably do better in difficultcollege courses and be more likely to get a degree,but that benefit is uncertain and far in the future.Maybe, over time, the school’s reputation and thecollege admission prospects of graduates willimprove because the current graduates are moresuccessful at local colleges. That, however, iseven more uncertain and postponed. Publishingdata on proportions of students that meet targetson standardized tests probably speeds the processby which real improvements in a school’s perfor-mance influence its local reputation. However,other indicators— such as SAT test scores, propor-tions going to various types of colleges and thesocioeconomic background of the students— tendto be more prominent. As a result, school reputa-tions are largely determined by things that teach-ers and administrators have little control over: thesocio-economic status of the student body and theproportion of graduates going to college.

American employers have historically paid littleattention to student achievement in high school orto school reputation when selecting young work-ers (Bishop 1989, 1993; Hollenbeck and Smith1984). Employers that do pay attention to student



achievement use as hiring criteria indicators ofrelative performance such as GPA and class rankrather than results on an external exam. Becausehigher standards do not benefit students as agroup, parents as a group have little incentive tolobby strongly for higher teacher salaries, higherstandards and higher school taxes.

External exams in secondary school subjectstransform the signaling environment. Hiring betterteachers and improving the science laboratoriesnow yields a visible payoff— more studentspassing the external exams and being admitted totop colleges. This, in turn, is likely to lead to morespending on schools, more rigorous hiring stan-dards for secondary school teachers and a higherpriority assigned to student learning in the alloca-tion of school budgets.

Additionally, reform-minded administrators haveused results of CBEEES to shame and inspireteachers to raise standards for all students. Thesuperintendent of a suburban New York districtthat has been nationally recognized for raisingstudent achievement levels explained: “[Externalvalidators like Regents exams and InternationalBaccalaureate] were the best and only way inwhich we could get teachers and staff to seethemselves as others might see them and not justkeep looking in the mirror and seeing themselvesas they would like to see themselves.” (Interviewwith superintendent of an All-Regents Highschool, August 1997)

Impact on Teachers: Thirty percent of Americanteachers say they “feel pressure to give highergrades than students’ work deserves” and “feelpressure to reduce the difficulty and amount ofwork you assign” (Peter D. Hart Research Associ-ates, 1994). Under a system of external exams,teachers and local school administrators lose theoption of lowering standards to reduce failurerates and raise self-esteem. The available alterna-

tive is to demand more of their students in orderto maximize their chances of being successful onthe external exams.

An additional benefit of CBEEES is the profes-sional development that teachers receive whenthey came together at centralized locations tograde the extended-answer portions of examina-tions. In May of 1996 I interviewed some teacherunion activists about the examination system inthe Canadian province of Alberta. Even thoughthe union and these teachers opposed the exams,they universally reported that serving on gradingcommittees was “… a wonderful professionaldevelopment activity.” (Bob, 1996) Coming toagree on what constituted excellent, good, poor,and failing responses to essay questions or open-ended math problems elicited a sharing of per-spectives and teaching tips that most found veryhelpful.

Many, however, fear that external exams willnegatively effect teaching. Opponents argue that“preparation for high stakes tests often empha-sizes rote memorization and cramming of studentsand drill and practice teaching methods” and that“some kinds of teaching to the test permits stu-dents to do well in examinations without recourseto higher levels of cognitive activity.” (Madeus1991 p. 7-8)

CBEEES advocates challenge the assumptionimplicit in this argument that examinations devel-oped by the committees of teachers working forstate departments of education are or will beworse than tests developed by individual teachers.In fact, the tests teachers develop for themselvesare generally of low quality. The 1983 Flemingand Chambers study of tests developed by highschool teachers found that “over all grades, 80percent of the items on teachers’ tests were con-structed to tap the lowest of [Bloom’s] taxonomiccategories, knowledge (of terms, facts or prin-



ciples)” (Thomas 1991, p. 14). Rowher andThomas (1987) found that only 18 percent ofhistory test items developed by junior high teach-ers and 14 percent of items developed by seniorhigh teachers required the integration of ideas.College instructors, in contrast, required suchintegration in 99 percent of their test items.Secondary school teachers test low-level compe-tencies because that is what they teach.

Carefully designed external exams can induceimprovements in instructional practice. ShermanTinkelman, New York State’s Assistant Commis-sioner for Examinations and Scholarships, de-scribes one such instance:

For years our foreign language specialistswent up and down the State beating the drumsfor curriculum reform in modern languageteaching, for change in emphasis from formalgrammar to conversation skills and readingskills. There was not very great impact untilwe introduced, after notice and with numeroussample exercises, oral comprehension andreading comprehension into our Regentsexaminations. Promptly thereafter, mostschools adopted the new curricular objectives(Tinkelman, 1966 p. 12).

Do CBEEES IncreaseAchievement? A Look at theEvidence

The hypothesis that curriculum-based externalexit examination systems (CBEEES) improveachievement will be tested by comparing nations,states and provinces that do and do not have suchsystems. Four different data sets will be exam-ined: science and mathematics achievement ofseventh and eighth graders in the 40-nation ThirdInternational Math and Science Study; science andmath scores of 13-year-olds on the InternationalAssessment of Educational Progress (IAEP) for16 nations and nine Canadian provinces; and SAT

test and NAEP math scores for New York Stateversus the rest of the United States. The theorypredicts that CBEEES affect societal decisionsabout education spending, administrator decisionsabout school priorities, teachers decisions aboutstandards and pedagogy and student decisionsabout studying. Much of the ultimate impact ofCBEEES on student achievement derives from thechanges they induce in spending, priorities andpedagogy. Most of the components of the fullFigure 1 model have been estimated in data onCanadian schools and students in Bishop (1996).Educational systems are the units of observationin this paper, and in most analyses the objective isto assess the total effect of CBEEES on achieve-ment (the sum of all the paths leading fromCBEEES to student achievement in Figure 1).Total effects are estimated by a reduced formmodel that controls for parental socio-economicstatus (SES), productivity and national culture, butnot the endogenous administrator, teacher andparent behaviors.

Third International Mathematics andScience Study

The recently released Third International Math-ematics and Science Study (TIMSS) provides1994-95 data for seventh and eighth graders for 39countries. To determine which of the TIMSSnations have curriculum-based externally-set exitexaminations in secondary school, 1 we reviewedcomparative education studies, governmentdocuments and education encyclopedias andinterviewed education ministry officials, embassypersonnel and Cornell graduate students from thevarious countries. Twenty-two national schoolsystems were classified as having CBEEES forboth mathematics and science in all parts of thecountry: Austria, Bulgaria, Columbia, CzechRepublic, Denmark, England, Hong Kong, Hun-gary, Ireland, Iran, Israel, Japan, Korea, Lithuania,the Netherlands, New Zealand, Russia, Scotland,Singapore, Slovak Republic, Slovenia and Thai-



land. Three countries— France, Iceland andRomania— had CBEEES in mathematics but notin science. Five countries— Australia, Canada,Germany, Switzerland and the United States— hadCBEEES in some provinces but not in others.Norway has regular exit examinations in math-ematics, but examines science only every fewyears. Latvia had an external examination systemuntil very recently, so it was given a .5 on theCBEEES variable. The countries classified ashaving no CBEEES in either subject were Bel-gium (both Flemish and French speaking sys-tems), Cyprus, Greece, Philippines, Portugal,Spain and Sweden. Based on the work of Madeusand Kelleghan (1991), the university entranceexaminations in Greece, Portugal, Spain, andCyprus, and the ACT and SAT in the U.S. werenot considered to be CBEEES. University en-trance exams should have much smaller incentiveeffects because students headed into work do nottake them and teachers can avoid responsibilityfor their students’ exam results by arguing that noteveryone is college material or that examinershave set an unreasonably high standard to limitenrollment in higher education.

Figures 2 and 3 array the 40 TIMSS countriesaccording to the science and mathematics achieve-ment of their 13-year-olds. The U.S. ranks fif-teenth in science and thirty-first in mathematics.The gaps between the vertical grid lines representone U.S grade-level equivalent— the differencebetween seventh and eighth grade TIMSS testscore means for the U.S. Achievement differen-tials across nations are very large. In science,Singapore, Korea, Bulgaria and Flemish Belgiumare more than one grade-level equivalent ahead ofthe U.S.; Columbia, Phillipines, Lithuania, Roma-nia and Portugal are more than three grade-levelequivalents behind the U.S. In mathematics,Singapore, Korea, Japan and Hong Kong are fouror more grade-level equivalents ahead of the U.S.,while Columbia, Philippines and Iran are behindthe U.S. by more than three grade-level equiva-lents. The countries represented by a solid black

bar in the figures have a curriculum-based exter-nal exit exam in the subject; countries representedby white bars do not have CBEEES. The coun-tries with a CBEEES in the subject tend to havehigher TIMSS scores.

Regression Analysis: The mean seventh andeighth grade science and mathematics test scoreswere regressed on average per capita gross domes-tic product in 1987 and 1990 deflated by a pur-chasing power parity price index, a dummy forEast Asian nation and a dummy for CBEEES.The results presented in Table 1 indicate that testscores are significantly higher in more developednations, East Asian nations and in nations with aCBEEES in the subject.

The analysis of achievement at a particular gradelevel may be biased, however, by differing poli-cies regarding grade retention, age of school entryand the grade chosen for assessment. CBEEES,for example, might be associated with high ratesof grade retention. Therefore, a preferable depen-dent variable is a measure of student achievementat some fixed age. The third and fourth rows ofeach panel present estimated models predictingthe median test score for each nation’s 13-year-olds (Beaton et al., 1996a, b, Table 1.5). Forcountries not included in this table, the 13-year-old median was estimated by age adjusting theseventh and eighth grade means. 2 Switching tothe age constant achievement somewhat reducesthe estimated impact of the CBEEES but theeffects remain statistically significant. Usingtwo-tailed t tests, the CBEEES coefficient has a P= .08 in the mathematics model and a P = .01 inthe science model. The estimated impacts aresubstantively important: 1.3 U.S. grade-levelequivalents in science and 1.0 U.S. grade-levelequivalents in mathematics.

One of the ways CBEEES may improve achieve-ment is by inducing greater social investments ineducation. Row 4 presents results of regressionsthat add the share of GDP spent on education to



Figure 2. Math Achievement at Age 13



Figure 3. Science Achievement at Age 13



Table 1The Effects of Curriculum-Based External Exams on

Science and Mathematics Achievement

External ExitExam

LnGDP/Pop1987 & 90

East Asia $K-12/GDP AdjR2/RMSE

TIMSS Science-1994

Mean for 7th Graders 38.0***(2.93)

33.8***(3.44)

20.1(1.24)

.31735.4

Mean for 8th Graders 42.4***(3.40)

36.2***(3.80)

14.4(.92)

.36434.4

Median for 13 Yr Olds 34.9***(2.77)

45.0***(4.68)

21.5(1.35)

.40234.7

Median for 13 Yr Olds 32.0***(2.57)

38.0***(3.71)

33.7*(2.01)

13.6*(1.86)

.44233.9

Diff-13 minus 9 YrOlds

7.6(.54)

-32.3***(3.11)

5.5(0.39)

.25826.4

TIMSS Mathematics-1994

Mean for 7th Graders 29.6**(2.09)

46.6***(4.60)

66.0***(4.01)

.46936.2

Mean for 8th Graders 36.0**(2.54)

48.7***(4.81)

62.0***(3.75)

.47636.6

Median for 13 Yr Olds 24.7*(1.82)

56.0***(5.77)

9.4***(4.37)

.53735.1

Median for 13 Yr Olds 21.5(1.55)

53.9***(5.07)

75.9***(4.41)

5.7(.75)

.54535.1

Diff-13 minus 9 YrOlds

17.1**(2.28)

-3.4(.66)

22.5***(3.28)

.45013.2

IAEP-1991

Science % Correct(U.S. GLE = 6)

4.3(1.72)

1.7(.61)

9.6**(2.81)

.4364.0

Math % Correct(U.S. GLE = 8)

15.7***(3.85)

3.7(.25)

16.1**(2.81)

.6416.0

Note: Numbers in parentheses are t values. GLE = grade level equivalent.* p < .10 on a two-tailed test** p < .05 on a two-tailed test*** p < .01 on a two-tailed test



the standard model. Coefficients on this variableare positive for both outcomes and significantly sofor science. The estimated impact of spending ismodest, however. A one percentage point in-crease in the share of GDP devoted to educationincreases the science achievement of 13-year-oldsby one-half of a grade-level equivalent.

The bottom row of each panel assesses the impactof CBEEES on measures of science and mathlearning between ages nine and 13. 3 Coefficientson the CBEEES dummy are positive for bothmath and science, but statistically significant onlyfor mathematics. The exams are taken duringupper secondary school or at the end of lowersecondary school, so CBEEES may have largereffects on learning during secondary school thanduring primary school. This prediction is sup-ported for math but not for science. For math-ematics the coefficients suggest that about two-thirds of the effect of CBEEES on achievement atage 13 was generated in the previous four years.Since exams are also likely to affect learningduring upper secondary school, total effects at theend of twelfth grade are likely to be larger still.

Analysis of the 1991 InternationalAssessment of Educational Progress

The 1991 International Assessment of EducationalProgress (IAEP) is the second data set in whichCBEEE effects can be tested. Data on fifteennations are available for the analysis: England,France, Hungary, Ireland, Israel, Emilia Romagna/Northern Italy, Korea, Portugal, Scotland,Slovenia, Soviet Union, Spain, Switzerland,Taiwan and the United States.

The average percent correct (adjusted for guess-ing) for 13-year-old students was regressed on thesame set of variables used in the analysis of theTIMSS data. The results are presented in the

second panel of Table 1. For mathematics, theeffect of curriculum-based external exams ishighly significant and quite large. The U.S.standard deviation was 26.8 percentage points inmathematics, so the CBEEE effect on math wasmore than one-half of a U.S. standard deviation orabout two U.S. grade-level equivalents. CBEEEShad a smaller non-significant effect on scienceachievement. East Asian students scored signifi-cantly higher than students in Europe and NorthAmerica. Coefficients on per capita GDP werepositive but not statistically significant.

These results are consistent with the causal hy-potheses presented above. Causation is notproved, however, because other explanations canbe proposed. Other sources of variation in cur-riculum-based exams need to be analyzed. Bestof all would be studies that hold national cultureconstant. Two such studies follow: one compar-ing Canadian provinces, the other comparing U.S.states.

Comparing Canadian Provinces

In 1990-91, the year the IAEP data was collected,Alberta, British Columbia, Newfoundland, Que-bec and Francophone New Brunswick had cur-riculum-based provincial examinations in English,French, mathematics, biology, chemistry, andphysics during the senior year of high school.These exams accounted for 50 percent of thatyear’s final grade in Alberta, Newfoundland andQuebec and 40 percent in British Columbia. Theother provinces did not have curriculum-basedprovincial external exit examinations in 1990-91.Ontario eliminated them in 1967, Manitoba in1970 and Nova Scotia in 1972. Anglophone NewBrunswick had provincial exams in language artsand mathematics but exam grades were notreported on transcripts or counted in final coursegrades. Canadian provincial exams are medium-stakes, not high-stakes tests. They influencegrades but passing the examination is not essential



for graduation. Employers appear uninterested inexam scores. Job application forms do not requestthat applicants report exam scores or grades.

The principals of schools sampled by IAEPcompleted questionnaires describing schoolpolicies, school resources and the qualifications ofeighth grade mathematics and science teachers.Students were asked about books in the home;number of siblings; language spoken at home;hours spent watching television, doing homework,pleasure reading, and watching science programson television; parental oversight of school work;and teaching methods of teachers.

The effects of curriculum-based provincial exitexams taken by twelfth graders on achievementand the behavior of Canadian 13-year-olds, theirparents, teachers and school administrators wereexamined by estimating models predicting thesebehaviors using schools as observations. The dataset comprises 1,338 Canadian schools. The modelcontained 11 variables: logarithm of the meannumber of books in the home; the mean numberof siblings; the proportion of the school’s studentswhose home language was different from thelanguage of instruction; logarithm of the numberof students per grade in the school; and dummiesfor schools run by a locally elected Catholic (orProtestant) school board, independent secular andnon-secular schools, schools with primary grades,schools that include all grades in one building,French speaking schools, and a dummy forEXAM provinces.

Table 2 presents regression results predicting fourachievement outcomes, 12 measures of schooladministrator behavior, nine teacher behaviors and11 student/parent attitudes and behaviors. The firstcolumn presents the hypothesized sign of therelationship between CBEEES and that variable.The means and standard deviations across schoolsof each dependent variable are presented incolumns two and three. The R2 corrected for

degrees of freedom is reported in column 14. Thecoefficient for EXAM and its t statistic are pre-sented in columns four and five. Provincial exitexams had large effects on achievement: 19percent of a U.S. standard deviation (about four-fifths of a U.S. grade-level equivalent) in math-ematics and 13 percent of a standard deviation(about half of a grade-level equivalent) in science.

Effect of CBEEES on Behavior of Students,Teachers and Administrators: Exit exams alsoaffected the behavior of parents, teachers andschool administrators. Schools in exit-examprovinces scheduled significantly more hours ofmath and science instruction, assigned morehomework, had better science labs, were signifi-cantly more likely to use specialist teachers formath and science, and more likely to hire mathand science teachers who had studied the subjectin college. Eighth grade teachers in exam prov-inces gave tests and quizzes more frequently.Hours in the school year, library books per stu-dent, computers per student, class size and teacherpreparation time were not significantly affected byCBEEES.

Opponents of externally set curriculum-basedexaminations predict that they will cause studentsto avoid learning activities that do not enhanceexam scores. This hypothesis was operationalizedby testing whether exam systems were associatedwith less reading for pleasure and less watching ofscience programs like NOVA and Nature. Neitherof these hypotheses is supported. Indeed, studentsin exam provinces spent significantly more timereading for pleasure, more time watching scienceprograms on television, while watching signifi-cantly less television overall. Parents in theseprovinces were more likely to talk to their chil-dren about their math and science classes and theirchildren were more likely to report that theirparents “are interested in science” or “want me todo well in math.”



CBEEES do not seem to skew teaching in unde-sirable ways. Students did more (not fewer)experiments in science class; emphasis on compu-tation using whole numbers— a skill that shouldbe learned by the end of fifth grade— declinedsignificantly. Teachers subject to the subtlepressure of a provincial exam four years in thefuture apparently adopt strategies that are conven-tionally viewed as “best practice,” not strategies tomaximize scores on multiple-choice tests.

Students responded to the improved teaching bybecoming more likely to report that science was“useful in everyday life.” The data provided nosupport for our hypothesis that CBEEES wouldinduce employers to pay greater attention to highschool achievement. Students in exam provinceswere not more likely to believe that math wasimportant in getting a good job and were lesslikely to believe that science was important in jobhunting.

A skeptic might point out that the correlationbetween EXAM and other outcomes may not becausal. Perhaps, the people of Alberta, BritishColumbia, Newfoundland, Quebec andFrancophone New Brunswick— the provinceswith exam systems— just place higher priority oneducation than the rest of the nation. This traitmay also result in greater political support forexamination systems. If so, we would expect thatschools in the exam provinces should be betterthan schools in other provinces along other di-mensions, such as discipline and absenteeism, notjust by academic criteria. Bishop (1996) predicts,to the contrary, that exam systems induce studentsand schools to redirect resources and attention tolearning/teaching exam subjects and away fromthe achievement of other goals (such as lowabsenteeism, good discipline and lots of comput-ers). These competing hypotheses are evaluatedin the 3rd, 4th, and 12th rows of Table 2. Con-trary to the “provincial taste for education” hy-pothesis, principals in exam provinces had notpurchased additional computers, did not report

significantly fewer discipline problems, weresignificantly more likely to report absenteeismproblems.

The Impact on New York StateRegents Examinations

In the early 1990s, New York State was the onlystate with a CBEEE System. It has been adminis-tering curriculum-based Regents Examinations tohigh school students since June 1878. AsSherman Tinkelman, Assistant Commissioner forExaminations and Scholarships, described in a1966 report:

The Regents examinations are closely relatedto the curriculum in New York State. Theyare, as you can see, inseparably intertwined.One supports and reinforces the other.... Theseinstruments presuppose and define stan-dards.... They are a strong supervisory andinstructional tool— and deliberately so. Theyare effective in stimulating good teaching andgood learning practices (Tinkelman, 1966 p.12).

The Regents examinations are taken throughoutone’s high school career. A student taking a fullschedule of college preparatory Regents courseswould typically take Regents exams in mathemat-ics and earth science at the end of 9th grade;mathematics, biology and global studies exams atthe end of 10th grade; mathematics, chemistry,American history, English and foreign languageexams at the end of 11th grade and physics examsat the end of 12th grade.

In 1993, about 56 percent of ninth graders tookthe Mathematics Course 1 exam and, of these, 24percent scored below the 65 percent passinggrade. Similar proportions of tenth and eleventhgraders took the global studies, biology andEnglish exams. Failure rates were 20 percent inglobal studies, 18 percent in biology and 13



percent in English. Those not taking Regentsexams were typically in considerably less chal-lenging courses than Regents level courses. Asystem of minimum competency tests in specificsubjects set a minimum standard for those nottaking Regents courses but, as in other states, thepassing standard was low.

Impact on SAT Test Scores

New York students are more disadvantaged, moreheavily minority and more likely to be foreign-born than students in most other states. Amongnorthern states, only Maryland, Delaware andIllinois have a larger share of African-Americanpupils. Nationally, only California has a highershare of its population foreign-born; only Califor-nia, Texas, Arizona, New Mexico and Coloradohave larger Hispanic population shares. Literacylevels among adults in New York are substantiallybelow the national average (NEGP 1993, Vol 2).

Consequently, when one compares studentachievement levels, family background must betaken into account. Considering the high inci-dence of at-risk children, New York students doremarkably well. The proportions of studentstaking algebra, calculus, chemistry and physics isgenerally above national averages. A largerproportion of New York’s eleventh and twelfthgraders are taking and passing (9.4 percent) APexams in English, science, math or history thanany other state except Utah (NEGP 1993, Vol. 2).

Graham and Husted’s (1993) analysis of SAT testscores in the 37 states with reasonably large test-taking populations found that New York Statestudents performed better than comparable stu-dents in other states. Graham and Husted did not,however, test the statistical significance of theNew York State effect and used an unusual log-log specification.

Table 3 presents the results of a linear regressionpredicting 1991 mean SAT-M + SAT-V test scoresfor the 37 states for which data are available.With the exception of the dummy variable forNew York State, variables are proportions—generally the share of the test-taking populationwith the characteristic described. New Yorkers dosignificantly better on the SAT than students ofthe same race and social background living inother states. When this model is estimated with-out the NYS dummy variable, New York has thelargest positive residual in the sample. Wisconsinhas the next largest positive residual which is 87percent of New York’s residual. Illinois andNevada have positive residuals that are about 58percent of New York’s value. Arizona, California,Colorado, Florida, New Mexico, Ohio, RhodeIsland, Texas and Washington have negativeresiduals greater than 10 points. Many of thesestates have large populations of Hispanics andrecent immigrants, a trait that was not controlledfor in the analysis. When one considers thatHispanics and immigrants are a large share ofNew York children, this makes New York’sachievement all the more remarkable.

For individuals the summed SAT-V + SAT-M hasa standard deviation of approximately 200 points.Consequently, the differential between New YorkState’s SAT mean and the prediction for NewYork based on outcomes in the other 36 states isabout 20 percent of a standard deviation or aboutthree-quarters of a grade-level equivalent.

Adding the teacher-pupil ratio and spending-per-pupil to the model reduces the NYS coefficient by25 percent, although it remains significantlygreater than zero. The significant coefficient onteacher-pupil ratio suggests that heavy investmentin K-12 schooling in New York State (possiblystimulated in part by the Regents exam system)may be one of the reasons why New York Statestudents perform better than comparable studentsin other states.



Impact on Mathematics Achievement ofEighth Graders

The New-York-is-exceptional hypothesis can alsobe tested by analyzing data from the 1992 admin-istration of the NAEP mathematics assessment torepresentative samples of eighth grade students in41 states and the District of Columbia. As withthe analysis of SAT scores, state test score meanswere regressed on variables controlling for thesocio-economic characteristics of the state’spopulation and a dummy for New York State. Thefive variables that controlled for student back-ground were: the proportion of people under age18 living in poverty; a schooling index for theadult population; percent foreign-born; percentpublic school students who are black; and percentpublic school students who are Hispanic. Theresults are presented in Table 4. Parents’ educa-tion, the poverty rate, percent black and percentforeign-born all had significant effects on mathachievement in the expected direction. New YorkState’s mean NAEP math score was a statisticallysignificant 9.6 points (or about one grade-levelequivalent) above the level predicted by theregression model.

One of the ways Regents exams may improveperformance is by inducing the public to hire extrateachers to reduce class size and provide specialhelp. Models were estimated with pupil-teacherratios on the right-hand side. Point estimates ofthe effect of pupil-teacher ratios were negative butnot signigicantly different from zero for 8th grademath test scores.

Impact on High School Dropout Rates

Table 4 also presents the results of cross-stateregressions predicting school enrollment rates atage 17 and high school graduation rates. NewYork State’s high school dropout rate is notsignificantly different from that of other stateswith students from similarly disadvantaged back-grounds. Additional staff appears to facilitatehigher graduation rates. A 10 percent reduction inthe pupil-teacher ratio increases the high schoolgraduation rate by 1.5 percentage points.

Table 3Determinants of Mean Total SAT-1 Scores for States

NYS ParticRate

ParentsAA-BA+

PrivateSchool

Prop.Black

LargeSchool

3+MathCourses

3+Eng.Courses

lnTeach/stud

lnExpend/stud

R2RMSE

46**(2.7)

-68**(2.6)

370**(6.4)

60(1.6)

-135**(3.2)

-44*(1.8)

85(1.3)

-36(.3)

.92614.8

35*(2.0)

-88***(3.3)

367***(6.6)

69*(1.9)

-113(2.6)

-36(1.5)

45(.7)

-45(.4)

48*(1.7)

13(.8)

.93314.2

MeanSD

.027

.164.414.240

.581.097

.207

.082.078.064

.129

.113.617.067

.797

.038-2.822.113

1.648.215

SAT-I92555

*** p < .01 on a two-tailed test** p < .05 on a two-tailed test* p < .10 on a two-tailed test



Table 4The Impact of Regents Examinations on Achievement and High School Graduation

Math NAEP Mean TestScore for 8th Grade 4

Percent of 17 Year OldsEnrolled in High School 2

Secondary School Graduatesper 100 persons 17 Years Old 2

New York State 9.59**(2.05)

8.86*(1.86)

7.27(1.49)

.44(.36)

.55(.44)

.0046(.00)

.36(.074)

-1.71(.36)

-3.50(.72)

Parents EducationIndex1

.68**(2.71)

.65**(2.57)

.85***(2.89)

.091(1.58)

.10(1.63)

.15**(2.28)

.96***(4.09)

.85***(3.80)

1.028***(4.12)

Percent in Poverty(People 18 yearsor less)2

-.52**(2.49)

-.50**(2.38)

-.42*(1.91)

-.034(.69)

-.035(.71)

-.017(.35)

-.017(.085)

-.035(.19)

-.092(.48)

Percent ForeignBorn3

-.66***(3.21)

-.63***(3.04)

-.50**(2.17)

-.18***(3.39)

-.18***(3.39)

-.13**(2.23)

-.51**(2.41)

-.42**(2.06)

-.26(1.14)

Percent of PublicSchool StudentsBlack4

-.32***(6.06)

-.33***(6.10)

-.36***(6.21)

-.047***(3.59)

-.046***(3.43)

-.045***(3.90)

-.14***(2.73)

-.17***(3.27)

-.19***(3.62)

Percent of PublicSchool StudentsHispanic4

-.0092(.10)

-.0067(.070)

-.057(.55)

-.012(.48)

-.012(.50)

-.027(1.07)

-.067(.68)

-.055(.59)

-.10(1.05)

Pupil TeacherRatio4

-.29(.38)

-.27(.86)

.037(.47)

.042(.56)

-.74**(2.51)

-.73**(2.48)

Hours ofInstruction perYear2

.030(1.29)

.0098*(1.76)

.032(1.49)

Adj R Squared .8313 .8303 .8336 .5713 .5636 .5840 .5475 .5961 .6071

RMSE 4.232 4.244 4.203 1.111 1.121 1.095 4.510 4.262 4.203

N Observations 42 42 42 51 51 51 51 51 51

* Statistically significant at 10% level **Statistically significant at 5% level ***Statistically significant at 1% level

1 Average of the percent of parents obtaining a secondary school diploma and the percent of parents obtaining a university degree.Education in States and Nations. National Center for Education Statistics. U.S. Department of Education. 1991. Page 139.2 Education in States and Nations. National Center for Education Statistics. U.S. Department of Education. 1991. Pages 49, 73, 119,129 and 149.3 1990 Census of Population. Social and Economic Characteristics United States. Pages 174-179.4 Digest of Education Statistics. U.S. Department of Education. 1993. Pages 61, 76, and 120.



Does New York State Invest More in K-12Education?

The theory predicts that the existence of CBEEESwill induce New York State to spend more on K-12 education and to focus that spending on in-struction. Indeed, New York’s ratio of K-12teacher salaries to college faculty salaries issignificantly above average. New York teachersare also more likely to have masters degrees thanthe teachers of any state except Connecticut andIndiana. New York ranks seventh in both theteacher-pupil ratios and the ratio of per pupilspending to gross state product per capita (Bishop1996).

Clearly, New York invests a great deal in its K-12education system. If the cause of the high spend-ing were a strong general commitment to educa-tion or legislative profligacy, we would expectspending to be high on both K-12 and highereducation. This is not the case. New York is firstin the ratio of K-12 spending per pupil to highereducation spending per college student.

The Regents exams are currently low-to-mediumstakes tests, not high stakes tests. Exam gradescount for less than one-quarter of the final gradein the course and influence only the type ofdiploma received. Employers ignore examsresults when making hiring decisions. During the1980s, scholarships sponsored by the Regentswere based on aptitude test scores, not Regentsexam results. A passing score on Regents examsis not necessary for admission to communitycolleges or out-of-state colleges. Students wereaware that they could avoid Regents courses andstill go to college. Some perceived an advantageto avoiding them; as a student explained:

My counselor wanted me to take Regentshistory and I did for a while. But it was prettyhard and the teacher moved fast. I switched tothe other history and I’m getting bettergrades. So my average will be better for

college. Unless you are going to a college inthe state, it doesn’t really matter whether youget a Regent’s diploma. (Ward, 1994)

Indeed, the small payoff to taking Regents examsmay be one reason why so many students have notbeen taking Regents courses.

This is about to change. The Board of Regentshas announced that students graduating in the year2000 must take and pass (at the 55 percent level) anew Regents English examination. The class of2001 faces the additional requirement of passingan examination in algebra and geometry. Theclass of 2002 must pass separate Regents exami-nations in global studies and American history.The phase-in of all five new required Regentsexams will be completed, when laboratory scienceexam courses come on stream, with the graduatingclass of 2003. Once the system has adjusted tothe new exams, the Regents intend to raise pass-ing scores from 55 percent to 60 percent and thento 65 percent.

Conclusions

Our review of the evidence suggests that theclaims by advocates of standards based-reformthat curriculum-based external exit examinationssignificantly increase student achievement areprobably correct. Students from countries withsuch systems outperform students from othercountries at a comparable level of economicdevelopment. Not only did students from Cana-dian provinces with such systems know morescience and mathematics than students in otherprovinces, they watched less TV and talked withtheir parents more about school work. Schools inCanadian provinces with external exams weremore likely to:

• employ specialist teachers of mathematics andscience;



• hire math and science teachers who hadstudied the subject in college;

• have high-quality science laboratories;

• schedule extra hours of math and scienceinstruction;

• assign more homework in math, in scienceand in other subjects;

• have students do or watch experiments inscience class; and

• schedule frequent tests in math and scienceclass.

When student demography is held constant, NewYork State, the only state having a CBEEE systemin the early 1990s, does significantly better thanother states on the SAT test and the NAEP mathassessments without experiencing a reduction inhigh school graduation rates.

CBEEES, however, are not the only importantdeterminant of achievement levels. Generalproductivity levels and standards of living and anEast Asian culture appear to have even largereffects. CBEEES are common in developingnations where achievement levels are often quitelow (for example, Columbia and Iran). Belgium,by contrast, has a top quality education systemwithout a CBEEES. More research on the system-level determinants of average achievement levelsis in order.



American Federation of Teachers. (1995) SettingStrong Standards: AFT’s criteria for judgingthe quality and usefulness of student achieve-ment standards. Washington, D.C.: AmericanFederation of Teachers, 1-12.

Beaton, Albert et al. (1996). MathematicsAchievement in the Middle School Years:IEA’s Third International Mathematics andScience Study. CSTEEP, Boston College,Boston MA. (http://wwwcsteep.bc.edu/TIMSS)

Beaton, Albert et al. (1996). Science Achievementin the Middle School Years: IEA’s ThirdInternational Mathematics and Science Study.CSTEEP, Boston College, Boston MA. (http://wwwcsteep.bc.edu/TIMSS)

Bishop, John. (1996). “The Impact of Curricu-lum-Based External Examinations on SchoolPriorities and Student Learning.” Interna-tional Journal of Education Research.

Bishop, John. (1993). “The Impact of AcademicCompetencies on Wages, Unemployment andJob Performance.” Carnegie/Rochester Con-ference Series on Public Policy, edited byBurton Malkiel and Charles Plosser, Vol. 37.

Bishop, John. (1989). “The Productivity Conse-quences of What is Learned in High School.”Journal of Curriculum Studies.

Bob. (May 1996). Interview conducted inCalgary, Alberta.

Competitiveness Policy Council. (1993) Reportsof the Subcouncils, March, Washington, D.C.:Competitiveness Policy Council.

Costrell, Robert. (1994). “A Simple Model ofEducational Standards.” The AmericanEconomic Review. Vol. 84, # 4, 956-971.

Fleming, M. and Chambers, B. (1983). Teacher-made Tests: Windows on the Classroom. SanFrancisco: Jossey Bass.

Graham, Amy and Husted, Thomas. (1993)“Understanding State Variation in SATScores.” Economics of Education, Vol 12, No.3, 197-202.

Gummere, Richard. (1943). “The IndependentSchool and the Post War World.” IndependentSchool Bulletin. Vol 4, April, (quoted inArthur Powell, Standards, 1995 Chapter 6).

Hart (Peter D.) Research Associates. (1995).“Valuable Views: A public opinion researchreport on the views of AFT teachers on profes-sional issues.” Washington D.C.: AmericanFederation of Teachers, 1995, 1-24.

Hollenbeck, K., and Smith B. (1984). The Influ-ence of Applicants’ Education and Skills onEmployability Assessments by Employers.Columbus: National Center for Research inVocational Education, Ohio State University.

Interview with superintendent of an All-Regentshigh school (1997).

Kang, Suk. (1985). “A Formal Model of SchoolReward Systems,” in Incentives, Learning andEmployability, edited by John Bishop, Colum-bus Ohio: National Center for Research inVocational Education.

References



Madeus, George. (1991). “The Effects of Impor-tant Tests on Students: Implications for aNational Examination or System of Examina-tions.” American Educational ResearchAssociation Invitational Conference on Ac-countability as a State Reform Instrument,Washington, D.C., 1-19.

Madeus, George and Kellaghan, Thomas. (1991).“Examination Systems in the European Com-munity: Implications for a National Examina-tion System in the United States.” ContractorReport for the Office of Technology Assess-ment, U.S. Congress, Washington, D.C, 1-100.

National Center for Educational Statistics. (1993).The Digest of Education Statistics: 1993.Washington D.C.: U.S. Department of Educa-tion.

National Education Goals Panel. (1995). Data forthe National Education Goals Report: 1995.Vol. 1, Washington, D.C.: U.S. GovernmentPrinting Office, 1-199.

Rohwer, W.D. and Thomas, J.W. (1987). “Do-main specific knowledge, cognitive strategies,and impediments to educational reform.” InM. Pressley (Ed.), Cognitive Strategy Re-search. New York: Springer-Verlag.

Steinberg, Laurence, Brown, Bradford andDornbusch, Sanford. (1996). Beyond theClassroom. New York: Simon and Schuster,1-223.

Thomas, John W. (1991). “Expectations andEffort: Course Demands, Students StudyPractices and Academic Achievement.” Paperpresented at a Conference on Student Motiva-tion sponsored by the Office of EducationalResearch and Improvement.

Tinkelman, Sherman N. (1966). “Regents Exami-nations in New York State after 100 Years.”Albany, N.Y: The University of the State ofNew York, The State Education Department,1-15.

Ward. (1994). “A Day in the Life.” N.Y. Teacher.Albany, N.Y.



Appendix A— BibliographySources Used to Classify National Education Systems

General

American Federation of Teachers. What College-Bound Students Abroad are Expected to Knowabout Biology. Washington, D.C.: AFT, 1994,1-120.

American Federation of Teachers. What Second-ary Students Abroad are Expected to Know:Gateway Exams Taken by Average-AchievingStudents in France, Germany and Scotland.(Dawn Krusemark), Washington, D.C.: AFT,1995, 1-165

American Federation of Teachers. What College-Bound Students Abroad are Expected to Knowabout Chemistry and Physics. (MatthewGandal), Washington, D.C.: AFT, 1996, 1-145.

Beaton, Albert et al. (1996) Mathematics Achieve-ment in the Middle School Years: IEA’s ThirdInternational Mathematics and Science Study.CSTEEP, Boston College, Boston MA. (http://wwwcsteep.bc.edu/TIMSS)

Beaton, Albert et al. (1996) Science Achievementin the Middle School Years: IEA’s ThirdInternational Mathematics and Science Study.CSTEEP, Boston College, Boston MA. (http://wwwcsteep.bc.edu/TIMSS)

Britton, Edward and Raizen, Senta. Examining theExaminations. Boston: Kluwer AcademicPublishers, 1996.

Eckstein, Max and Noah, Harold. SecondarySchool Examinations. New Haven: YaleUniversity Press, 1993.

Eckstein, Max and Noah, Harold. (Eds.) Exami-nations: Comparative and InternationalStudies. New York: Pergamon, 1992.

Elley, Warwick, How in the World Do StudentsRead?, The Hague, The Netherlands: Interna-tional Association for the Evaluation ofEducational Achievement, 1992.

Heyneman, Stephen P. and Fagerlind, Ingemar.University Examinations and StandardizedTesting. Washington D.C.: World Bank, 1988.

International Assessment of Educational Progress,Learning Mathematics. Princeton, NJ: ETS,1992b.

International Assessment of Educational Progress,Learning About the World. Princeton, NJ:ETS, 1992c.

International Assessment of Educational Progress,IAEP Technical Report. Volume 1, Princeton,NJ: ETS, 1992c.

Kreeft, Henk P. J. (ed.) (1990) “Issues in PublicExaminations.” Paper prepared for the Inter-national Association for Educational Assess-ment, 16th International conference on Issuesin Public Examinations, Masstricht, TheNetherlands June 18-22, 1990.

Lundberg, Ingvar and Linnakyla, Pirjo. TeachingReading around the World. The Hague: Inter-national Association for the Evaluation ofEducational Achievement, 1993.

Madeus, George F. and Kellaghan, Thomas.Student Examination Systems in the Europeancommunity: Lessons for the United States.Contractor report submitted to the Office ofTechnology Assessment. June 1991



National Center for Educational Statistics. Surveyof examination practices. (Designed by JamesGuthrie, Marilyn Binkley and Gary Phillips),Conducted for OECD, 1991

Organization of Economic Co-operation andDevelopment. Education and Training afterBasic Schooling. Paris, France: OECD, 1985.

Organization of Economic Co-operation andDevelopment. Education at a Glance. Paris,France: OECD, 1995.

Phelps, Richard. Are U.S. Students the MostHeavily Tested on Earth? Washington, D.C.:American Institutes for Research, 1996, 1-26.

Postlethwaite, T. Neville and Wiley, David E.Science Achievement in Twenty-Three Coun-tries. London: Pergamon Press, 1992.

Robitaille, David and Garden, Robert. The IEAStudy of Mathematics II: Contexts and Out-comes of School Mathematics, New York:Pergamon Press. 1989

U.S. Congress, Office of Technology Assessment.Testing in American schools: Asking the rightquestions. OTA-SET-519 Washington, D.C.:U.S. Government Printing Office, Feb. 1992,1-301.

Specific Countries

Australia

Fendley, G. C. “Australia.” In Lee Deighton(ed.), The Encyclopedia of Education, vol. 1,Macmillan, 1971, 427-431.

Austria

Brockman, Erin. Education in Austria. Paper forILR 698, Cornell University, 1995, 1-19 .

Lahey, Helen. “Austria.” In Lee Deighton (ed.),The Encyclopedia of Education, vol. 1,Macmillan, 1971, 431-437.

Donnell, Julie, “Austria.” In George Kurian (ed.),World Education Encyclopedia, Facts on FilePublications, 1988, 96-105.

Federal Press Service, The Austrian EducationSystem. Vienna: Federal Chancellery, 1994, 1-46.

Belgium

Plancke, R. L., “Belgium,” Lee Deighton (ed.),The Encyclopedia of Education, vol. 5,Macmillan, 1971, 449-455.

Fletcher, Ann. Belgium. Washington, D.C.:American Association of Collegiate Registrarsand Admissions Officers, 1985, 1-183.

Ministerie Van de Vlaamse Gemeenschap. Educa-tional Developments in Belgium, 1992-94:The Flemish Community. Brussels: Centrumvoor Informatie en Documentatie, 1994, 1-42.

Ministerie van de Vlaamse Gemeenschap. Educa-tion in Belgium: The Diverging Paths. Reviewof National Policies of Education) Bussels:Ministerie van de Vlaamse Gemeenshap,1991, 1-463.

Vanberrgen, P. “Belgium.” in Postlethwaite, T. N.(ed.), The Encyclopedia of ComparativeEducation and National Systems of Education,Oxford: Pergamon Press, 1988,

Brazil

Interview with Romualdo Protela de Oliveira,Professor at University of Sao Paulo.



Canada

Alberta Education. Diploma ExaminationsReport, Annual Report, 1993-94. Edmonton,Alberta, Student Evaluation Branch, 1994, 1-56.

Alberta Education. Back to School InformationKit: 1995/96. Edmonton, Alberta, StudentEvaluation Branch, 1995, 1-48.

Calder, Peter. Impact of Diploma Examinations onthe Teaching— Learning Process. Edmonton:University of Alberta, Dec. 1990, 1-61.

McEwen, Nelly “Student Academic Outcomes inCanada.” In APEC Symposium on perfor-mance Measurement Systems, October 23-26.1995, Washington, D.C.: Asia-Pacific Eco-nomic Cooperation Education Forum, PelavinResearch Institute, 207-215.

Manitoba Education and Training. EducationalAssessment in Manitoba. Winnipeg: Curricu-lum Services-Assessment Unit, Sept 1993, 1-27.

Minister of Education and Training. RenewingEducation: New Directions, The Action Plan.Winnipeg: Manitoba Education and TrainingJan. 1995, 1-71.

Evaluation Branch, Department of Education.1993-94 Grade 11 Provincial ExaminationsReport and Analysis: English and Mathemat-ics. Fredericton, New Brunswick, Nov. 1994,1-44.

Saskatchewan Education, Saskatchewan Provin-cial Learning Assessment Program. Regina,Saskachewan: 1993, 1-49.

Saskatchewan Education, High School ReviewAdvisory Committee Final Report. Regina,Saskachewan: 1994, 1-104.

Saskatchewan Education, Training and Employ-ment. Secondary Level Statistics: Jan-Aug1993-94 Reporting Periods. Regina, March1995, 1-11.

Schweitzer, Thomas Robert Crocker andGeraldine Gilliss. The State of Education inCanada. Montreal: Institute for Research inPublic Policy, 1995, 1-141.

Statistics Canada. Education in Canada, 1995.Ottawa: Statistics Canada, 1996, 1-198.

U.S. General Accounting Office, EducationalTesting: The Canadian Experience withStandards, Examinations and Assessments.(written by Kathleen D. White) GAO/PEMD-93-11, Washington, D.C., April 1993, 1-74.

Columbia

Mora, J., “Columbia,” in Postlethwaite, T. N.(ed.), The Encyclopedia of ComparativeEducation and National Systems of Education,Oxford: Pergamon Press, 1988, 202-205.

Cyprus

______ “Cyprus,” in George Kurian (ed.), WorldEducation Encyclopedia, Facts on File Publi-cations, 1988, 276-285.

Denmark

Interviews with qjvind Brogger at aFOLKESKOLE (Main School) ) near Arhus,and with Johanus Andersen and DorteBollerup of Katedral Gymnasium and princi-pal, teachers and students at CrhusKrbmandsskole (Business College).

Ministry of Education and Research. Facts andFigures, Education Indicators for Denmark.Copenhagen: Undervisningsministeriers forlag1993, 1-83.



Organization of Economic Cooperation andDevelopment. Denmark. Paris: OECD, 1980,1-161.

Finland

Interviews with Rita Asplund at ETLA and withprincipals, teachers and students at threesecondary schools.

Kivinen, Osmo and Rinne, Risto. EducationalStrategies in Finland in the 1990s Turku,University of Turku, 1992. 1-134.

National Board of Education. The EducationSystem of Finland. Helsinki: National Boardof Education, 1994, 1-38.

National Board of Education. Senior SecondarySchool in Finland. Helsinki: National Boardof Education, 1994, 1-15.

France

Interviews of principals and teachers at two Lyceeand officials at the Ministry of Education.

Ministere de l’Education Nationale et de la Cul-ture, Reperes and References Statistiques surles enseignmements et la formation. 1992Edition, (Paris: 1992a)

Ministere de l’Education Nationale et de la Cul-ture, Education in France. 4th edition, Vanves,France: Direction Del =Evaluation et de laProspective, Oct. 1994, 1-41.

Warner, Raymond. France. Washington, D.C.:American Association of Collegiate Registrarsand Admissions Officers, 1975, 1-249.

Lewis, H. D. The French Education System, NewYork, St. Martin’s Press, 1985.

Germany

Interviews of Antonio Ruiz-Quintanilla, andMartin Behrens)

Behrens, Martin. Structure, Standards and Diver-sity of the German System of SecondaryEducation, paper for ILR698, Spring 1997, 1-6.

Lehman, R. H., “Germany: System of Educa-tion,” in Husen T. & Postelthwaite, T.N. (eds),The International Encyclopedia of Education,Pergamon, 1994.

Rust, Val D. “West Germany,” in George Kurian(ed.), World Education Encyclopedia, Factson File Publications, 1988, 450-457.

World Education News and Reviews. [abstractingnewsletter Faits Nouveaux], Vol 6, No. 2,Spring 1993, p. 6.

Greece

Massialas, Byron, “Greece.” Lee Deighton (ed.),The Encyclopedia of Education, vol. 1,Macmillan, 1971, 190-194.

Massialas, Byron G., Flouris, G., and Cassatakis,Michael, “Greece,” in George Kurian (ed.),World Education Encyclopedia, Facts on FilePublications, 1988, 488-495.

Iceland

Interview with chief of the Cultural SectionEmbassy of Iceland.

Fraser, Stewart, “Iceland,” in Lee Deighton (ed.),The Encyclopedia of Education, Macmillan,1971, 534-536.



Josepsson, Bragi, “Iceland,” George Kurian (ed.),World Education Encyclopedia, Facts on FilePublications, 1988, 560-567.

India

Interviews with Sukdeep Brar and SaroshKruvilla.

Bordia, A., “India: System of Education,” inHusen, T. & Postelethwaite T.N.(eds) TheInternational Encyclopedia of Education, 2nd.Pergammon, 1994.

Iran

Kurian, George, “Iran,” George Kurian (ed.),World Education Encyclopedia, Facts on FilePublications, 1988.

Afzal, Manuchehr, “Iran,” in Lee Deighton (ed.),The Encyclopedia of Education, vol. 5, TheMacmillan Co. and Free Press, 1971.

Italy

Burton, Maria Concetta. The Historic Develop-ment of Italian Education. Paper for ILR698.Dec 1995, 1-21.

Kurian, George. “Italy,” George Kurian (ed.),World Education Encyclopedia, Facts on FilePublications, 1988, 650-657.

Organization of Economic Cooperation andDevelopment. Educational Reform in Italy.Paris: OECD, 1985, 1-111

Japan

Dore, Ronald and Mari Sako. How the JapaneseLearn to Work.

Kogirima, Atusushi. Japanese Education in Early1960s. Cornell University, Dec. 1995, 1-14.

White, Merry, The Japanese Educational Chal-lenge.

Korea

Quan, Ducchi. The Korean Educational System:an Overview. Cornell University, Dec. 1995,1-22.

Shin, Se-Ho. “Republic of Korea: System ofEducation,” in Husen, T. & PostelethwaiteT.N.(eds) The International Encyclopedia ofEducation, 2nd. 1994, 3161-3170.

Malaysia

Faan, Hew Seok. Education in Malaysia. CornellUniversity, Dec 1989, 1-18

The Netherlands

Interviews of officials at the Ministry of Educa-tion and Principals, teachers and students at aVWO and a LBO near Gronigen.

Schuler, Peter. The Netherlands. Washington,D.C.: American Association of CollegiateRegistrars and Admissions Officers, 1984, 1-191

Nijhof, W.J. and Streumer, J.N. “The Nether-lands.” Postlethwaite, T. N. (ed.), The Ency-clopedia of Comparative Education andNational Systems of Education, Oxford:Pergamon Press 1988.

Ministry of Education and Science. The DutchEducation System. Docinform No. 332E,Zoetermeer: Ministry of Education and Sci-ence, 1988, 1-50.

Organization of Economic Cooperation andDevelopment. Netherlands. Paris: OECD,1991, 1-148.



Social and Cultural Planning Office. Social andCultural Report: 1988, The Netherlands.Rijswijk: Social and Cultural Planning Office,1988, 1-427.

Central Bureau Voor De Statistiek. “EducationStatistics,” The Hague, Netherlands, 1993.

New Zealand

Kallenbach, Dolores. New Zealand. Paper forILR698, Dec 1995, 1-17,

McMahon, Tim. “Performance MeasurementSystems in New Zealand.” In APEC Sympo-sium on performance Measurement Systems,October 23-26. 1995, Washington, D.C.: Asia-Pacific Economic Cooperation EducationForum, Pelavin Research Institute, 39-62.

Norway

Interviews with Tove Hammer and Johan____)

Kurian, George. “Norway” in George Kurian(ed.), World Education Encyclopedia, Facts onFile Publications, 1988, 952-961.

Bjorndal, I. R. “Norway: System of Education.” InHusen, T. & Postelethwaite T.N.(eds) TheInternational Encyclopedia of Education, 2nd.1994, 4188-4192.

Ministry of Education and Research, Education inNorway, Oslo: Royal Norwegian Ministry ofEducation and Research, 1990, 1-36.

Ministry of Education and Research, The Devel-opment of Education 1990-92: Norway,prepared for International Conference onEducation, 43 session Geneva 1992, Oslo:Royal Norwegian Ministry of Education andResearch, Aug. 1992, 1-53.

Ministry of Education and Research, The UpperSecondary School, Oslo: Royal NorwegianMinistry of Education and Research, 1992, 1-21.

Nordic Statistical Secretariat. Educational indica-tors in Nordic Countries. Stockholm,Norstedts Tryckeri AB, 1991, 1-62.

Philippines

Interviews with Cornell graduate students fromthe Philippines: Pia Gavino, Carol Hau, LornaAcebedo and Noel Yap.

Arcelo, Adriano, Higher Education and the LaborMarket in the Philippines. New Delhi: WileyEastern Limited, 1981.

Bascani, Ramon C. “Performance MeasurementSystems in the Republic of the Philippines.” InAPEC Symposium on performance Measure-ment Systems, October 23-26. 1995, Washing-ton, D.C.: Asia-Pacific Economic CooperationEducation Forum, Pelavin Research Institute,105-117.

Gonzalez, Hernando, Josefina Cortes, “ThePhilippines.” in George Kurian (ed.), WorldEducation Encyclopedia, Facts on File Publi-cations, 1988, 992-1002.

Guevara, Elizabeth. Educational System in thePhilippines. Paper for ILR698, 1995.1-23.

______, “Philippines: System of Education,” inHusen, T. & Postelethwaite T.N.(eds) TheInternational Encyclopedia of Education, 2nd.1994, 4435-4442.

Sutaria, M. C. “Philippines,” in Postlethwaite, T.N. (ed.), The Encyclopedia of ComparativeEducation and National Systems of Education,Oxford: Pergamon Press, 1988, 554-559.



Wilcox, Lee. (ed.) The Evaluation of AsianEducational Credentials. New York: NationalAsociateon for Foreign Student Affairs, 1996,34-52.

Poland

Wesolowski, Katherine. Education in Poland.Cornell University, December 1995, 1-16.

Portugal

Planchard, Emle, “Portugal,” in Lee Deighton(ed.), The Encyclopedia of Education,Macmillan, 1971, 181-185.

Ministry of Education. The Portugese System ofEducation: Facts and trends. Prepared forEducational and Cultural Spaces, LisbonMarch 1990.

Rau. J. M. “Portugal: System of Education,” inHusen, T. & Postelethwaite T.N.(eds) TheInternational Encyclopedia of Education, 2nd.1994, 4622-4631.

Romania

Neag, Marie, “Romania,” in George Kurian (ed.),The World Education Encyclopedia, Facts onFile Publications, 1988, 1046-1051.

Russia

DeWitt, Nicholas, “Union of Soviet SocialistRepublics,” in Lee Deighton (ed.), The Ency-clopedia of Education, Macmillan, 197, 295-299.

Slovenia

Pozarnik, B. Marentic, “Slovenia: System ofEducation,” in Husen, T. & PostelethwaiteT.N.(eds) The International Encyclopedia ofEducation, 2nd. 1994, 5490-5497.

Plut-Pregelj, Leopoldina. “Slovenia’s Education inthe Process of Change: Legal Aspects.”Slovene Studies, vol 13. No. 2, 1991, 129-141.

Plut-Pregelj, Leopoldina. “Changes in the SloveneEducational System, 1990-1992.” East/WestEducation. Vol. 14, No. 1, Spring 1993, 51-65.

Spain

Interview with Ferran Mane, lecturer at the Uni-versity of Barcelona

Borruso, Chris. The Spanish Educational System.Cornell University, Dec. 1989, 1-15.

Hoz, Victor Garcia, “Spain,” in Lee Deighton(ed.), Encyclopedia of Education, Macmillan,1971

Gil, G.A., “Spain: System of Education,” inHusen, T. & Postelethwaite T.N. (eds) TheInternational Encyclopedia of Education, 2nd.1994, 5650-5658.

Turner, Solveig and Colbert, Ava Goicoa,“Spain.” In George Kurian (ed.) The WorldEducation Encyclopedia, Facts on File Publi-cations, 1988, 1120-1129.

Sweden

Organization of Economic Cooperation andDevelopment. Educational Reform in Sweden.Paris: OECD, 1981, 1-105.

Rehn, Gosta and Petersen, K. Helveg. Educationand Employment in Sweden and Denmark.New York: Carnegie Council on Policy Stud-ies in Education, 1980, 1-191.

Ministry of Education and Finance. The SwedishWay Towards a Learning Society. Report toOECD, Stockholm: Ministry of Education andFinance, 1992.



Switzerland

Federal Statistical Office. The Swiss EducationalMosaic: A Study in Diversity. Berne: Federalstatistical Office, 1991, 1-71.

Gretler, A., “Switzerland,” in Postlethwaite, T.N. (ed.), The Encyclopedia of ComparativeEducation and National Systems of Education,Oxford: Pergamon Press, 1988, 641-646.

Gretler, A., “Switzerland: System of Education.”in Husen, T. & Postelethwaite T.N. (eds) TheInternational Encyclopedia of Education, 2nd.1994, 5876-5883.

Haffner, Walter. The Educational Systems ofSwitzerland. (Speech given on May 5, 1995 byhead of the Cultural Section of the SwissEmbassy).

Susskind, Jacob. “Switzerland,” The World Edu-cation Encyclopedia, Facts on File Publica-tions, 1988, 1184-1191.

Organization of Economic Cooperation andDevelopment. Switzerland. Paris: OECD,1991, 1-213.

Taiwan

Fang, Lih-Ling. The Examination System inTaiwan. Cornell University, Fall 1995, 1-20.

Kuo, Su-Feng. Education in Taiwan. CornellUniversity, Fall 1993, 1-17.

Su, Jin-Li “Performance Measurement Systemsfor Elementary and Secondary Education inTaiwan: Current status and Concerns.” InAPEC Symposium on Performance Measure-ment Systems, Washington, D.C.: Asia-PacificEconomic Cooperation Education Forum,

Pelavin Research Institute, October 23-26.1995, 105-117.

Thailand

Chantanrrich, S., & G. W. Fry, “Thailand,” inPostlethwaite, T. N. (ed.), The Encyclopedia ofComparative Education and National Systemsof Education, Oxford: Pergamon Press, 1988,658-662.

______, “Thailand: System of Education.” InHusen, T. & Postelethwaite T.N.(eds) TheInternational Encyclopedia of Education, 2nd.1994, 6374-6378.

________. “Thailand,” in The World EducationEncyclopedia, Facts on File Publications,1988

United States

American Federation of Teachers. (1995) SettingStrong Standards: AFT’s criteria for judgingthe quality and usefulness of student achieve-ment standards. Washington, D.C.: AmericanFederation of Teachers, 1-12.

U.S. General Accounting Office, Student Testing:Current Extent and Expenditures with costestimates for a National Examination. GAO/PEMD-93-8, Washington, D.C., Jan. 1993, 1-86.



End Notes

1. Appendix A provides a bibliography of the documents and individuals consulted when making theseclassifications. The TIMSS report’s information about examination systems does not distinguish be-tween university admissions exams and curriculum-based exit exams, so its classifications are not usefulfor this exercise. The Philippines, for example, is classified as having external exams by the TIMSSreport, but its exams are university admissions exams similar to the SAT. South Africa was excludedbecause its education system was disrupted for many years by boycotts that were part of the campaign toend apartheid. Kuwait was excluded because of the disruption of its education system by the Iraqiinvasion and the Gulf War.

2. The Phillipines, for example, had a math score mean of 399 in eighth grade and a mean of 386 inseventh grade. The mean age of eighth graders was 14 and the mean age of seventh graders was 12.9.The math score for 13.5 year olds was estimated by interpolation between seventh and eighth grademeans. Math13.5 = 386 + (399-386)*((13.5-12.9)/(14-12.9)).

3. This indicator of learning between age nine and 13 can only be constructed for the 25 countries thatparticipated in both the primary school and middle school TIMSS studies. The small size of the samplelowers the power of our statistical tests.

do curriculum-based external exit exam systems enhance student

Documents