test prediction and performance in a classroom context

8/12/2019 Test Prediction and Performance in a Classroom Context

1/11

Journal of Educational Psychology2000, Vol. 92, No.1,160-170 Copyright 2000 by the American Psychological Association, Inc.0022-0663/00/55.00 DOI: 10.1037//0022-0663.92.1.160

Test Prediction and Performance in a Classroom ContextDouglas J. Hacker, Linda Bol, Dianne D . Horgan, and Ernest A. RakowUni ver s i ty o f Mem phi sThis study focused on stu dents' ability to predict and postdict test performance in a classroomcontext. Ninety-nine undergraduate students participated during a semester-length course inwhich the relation between self-assessment and performance was stressed. Research questionswere (a) Can students accurately predict test performance? (b) Does accuracy vary withperformance? (c) Does prediction accuracy increase over multiple tests? and (d) Do priorperformance and predictions of performance influence subsequent predictions? High-performing students were accurate, with accuracy improving over multiple exams. Low-performing students showed moderate prediction accuracy but good postdiction accuracy.Lowest performing students showed gross overconfidence in predictions and postdictions.Judgments of performance were influenced by prior judgments and not prior performance.Performance and judgm ents of performance had little influence on subsequent test preparationbehavior.

Can students accurately predict test performance? Pred ic-tions of test performance are metacomprehension judg-men ts, which require, in part, that students self-assess whatthey know about the to-be-tested material, judge howthoroughly they understand it, and judge whether they willbe able to use their knowledge to optimize performance onthe upcoming test. Greater accuracy in prediction can helpstudents avoid either premature termination or prolongedduration of study, thereby helping with the management oftime and effort. Greater accuracy can also help studentsnarrow the focus of their study to specific area s judg ed tohave a low probability of success. Accurate predictions oftest performance, therefore, can play a critical role inmaximizing test preparedness.Although people are generally inaccurate in predictingtheir performance (e.g., Glenberg & Epstein, 1985; Glen-berg, Sanocki, Epstein, & Morris, 1987; Lichtenstein, Fis-chhoff, Phillips,1982;Mak i & Berry, 1984), under certainconditions and for certain tasks, people's prediction accu-racy can be somewhat better than chance (e.g., Gillstrbm &Ronnberg, 1995; Horgan, 1990; Horgan, Bol, & Hacker,1997;Magliano, L ittle, & Graesser, 1993;Maki, 1995; Maki& Serra, 1992; Pressley & Ghatala, 1989; Weaver, 1990;Weaver & Bryant, 1995). Prediction accuracy often isresistant to improvement (e.g., Gigerenzer, Hoffrage, &Kleinbolting, 1991; Koriat, 1997; Koriat, Lichtenstein, &Fischhoff, 1980), but sometimes there are modest improve-ments (Hertzog, Dixon, & Hultsch, 1990; Horgan, 1990;Horgan, Hacker, & Huffman, 1997; Koriat & Goldsmith,1994, 1996; Pressley, Snyder, Levin, Murray, & Ghatala,

Douglas J. Hacker, Linda Bol, Dianne D. Horgan, and Ernest A.Rakow, Counseling, Educational Psychology and Research, Univer-sity of Memphis.Correspondence concerning this article should be addressed toDouglas J. Hacker, who is now at the Department of EducationalStudies, University of Utah, 1705 East Campus Center Drive,Room 307, Salt Lake City, Utah 84112-9256. Electronic mail maybe sent to [email protected].

1987; Walczyk & Hall, 1989). Gains may be greater forhigher performing students than for lower performing stu-dents (Maki,1998;Maki & Berry, 1984; Shaughnessy, 1979;Sinkavich, 1995).Further, the studies that we could find about test predic-tion accuracy were restricted to laboratory settings, usedexperimentally learned materials, or asked questions concern-ing generic world knowledge. Therefore, studies of predic-tion accuracy, despite having important relevance to educa-tional issues, have generally lacked eco logical validity. Onlyin Shaughnessy (1979) and Sinkavich (1995) were actualclassroom tests used as the focus of metacognitive judg-ments. In addition, we have not found a study in which anactual classroom context was used. Even though severalresearchers have voiced a need for metacognitive researchconducted in mo re naturalistic environments (e.g., Koriat &Goldsmith, 1996; Nelson & Narens, 1994), most metacogni-tive studies are still conducted in the laboratory usingexperimentally based materials. Thus, little focus has beengiven to students' prediction accuracy and to improvementsin accuracy when students are asked to make judgmentsconcerning richer knowledge domains that have been devel-oped over longer periods of time under more motivatingcircumstances.

To shed light on these inconsistent or incomplete results,the present study was conducted in an undergraduateeducational psychology course. Over an entire semester,students received instruction that included, in part, anemphasis on the important relation between self-assessmentof performance prior to testing and actual performance. Weaddressed four specific questions: (a) Can students accu-rately predict test performance? (b) Does accuracy vary withtest performance? (c) Does prediction accuracy increaseover multiple tests? and (d) To what extent do priorperformance and predictions of performance influence sub-sequent predictions?Wealso addressed these questions withrespect to postdiction accuracy. Because a postdictive judg-ment serves as a self-evaluation of how well a student

160


2/11


3/11

162 HACKER BOL,HORGAN AND RAKOWmonitoring and may simply rely on their predictions ofperformance as a basis for making postdictions. In this case,upgrading of postdiction accuracy would be minimized.Thus, we examined whether the added complexities ofmaking memory judgments in a naturalistic context wouldcompromise students' postdiction accuracy.

Finally, we were interested in examining the relationsamong prior performance, judgments of performance, andsubsequent judgments of performance. Early in the semes-ter, with limited feedback on their actual performance,students'judgments of performance are likely based, in part,on some or all of the five contributing factors that wereidentified earlier. These initial judgments of performancewould then serve as anchors for subsequent judgments(Dunlosky & Hertzog, 1998). However, because priorperformance is one of the best predictors of future perfor-mance, with each test, students would learn to rely more onprior performance than on prior judgments of performanceto make accurate judgments of performance; that is, studentswould learn to reanchor their judgments of performancefrom prior judgments to prior performance. We hypoth-esized, therefore, that as the course progressed, the strengthof the relation between judgments of performance and priorjudgments would decrease, with a corresponding increase inthe relation between judgments of performance and priorperformance.

MethodParticipants

Participants were 99 undergraduate college students enrolledintwo sectionsofan introductory educational psychology courseat amid-south university.The two sections were taughtby thesameinstructor, had equivalent instruction, and had the same courserequirements. Approximately 90% ofthe students w ere enrolledina teacher education program and took the courseas arequirementof that program. The remaining 10% ofthestudents took the courseas an elective.Measures

Predictions were made before each examas thepercentageofitems students expected to answer correctly. Postdictions weremade after each exam as the percentageofitems they believed theyhad gotten correct. Asinother calibration research (e.g., Koriat&Goldsmith, 1996; Liberman & Tversky, 1993), the accuracyofpeople's subjective judgmen tsofperformance was determinedbycomparing their assessed probability that a collection of itemswould beorwere answered correctly with the actual percentageofitems correct. Last, students were askedtoestimatethenumber ofhours they spent studyingforthe exam .

Performance was measured by scores obtained on three multiple-choice exams. Most of the test items came from the test bankdistributed by the pu blisherofthe text used for the course (Good&Brophy, 1995). These items were mixed with respectto difficulty(low, medium,or high) and type (understanding, integration,orapplication). A fewoftheitems on each test were developed bythecourse instructor. There were 63itemson thefirst exam and 96itemson thesecond exam .Thethird examwas acom prehensivefinal that contained 133 items. Although the tests variedinlength,eachof the tests contained approximatelythesame proportionsof

low, medium, and high difficulty items (i.e., about 42% low, 52%medium, and 6% high). Also,aCronbach's alphaof.83amongthethree tests indicates that students maintained consistent perfor-mance across the three exams. An exampleofadifficult integrationitemisWhichone of the following ideas seemsto be thebasisforhow reciprocal teaching is done?a. Vygotsky's zone of proximal developmentb. Bruner's spiral curriculumc.Ausubel's advance organizerd. Gagne's eventsofinstruction.

An exampleofan easy understand ing questionisHo wdoconstructivistandassociationist theoriesofmemorydiffer?a. Associationists are more likelytostress the law of effect.b. Constructivists put more stresson the active creationofmemory.c. Constructivists base their theory on the contiguity principle.d. Associationists were more likelytouse school tasksas thebasisfortheir theory.

ProcedureA major emphasis of the course was on self-assessment.Throughout the course, students received instruction about theimportance of reflection in learning, particularly its importanceconcerning accurate self-assessments of one's knowledge andperformance. Specific lessons that the students received focused onthe benefitsofaccurate self-assessment, including how accuracyinself-assessment could lead to more effective use of feedback,improved time management, and appropriate goal setting.To help students develop their self-assessment skills in thepreparation for testing, 1week prior to taking eachof thethree

exams, students were given practice exams that were parallelversionsofthe actual exam s. Students w ere encouraged to use theirperformance on the practice examsas a way to identify strengthsand weaknesses in their understandingof the material. With eachpractice exam, students were given an answer key with pagenumbers referencing each question. They were urged totakethepractice exam without referringto thetextortheir notes, scoreit,and then, using their text and notes,try tounderstand their errors.Students were free to discuss the practice exams w ith other studentsor the instructor.One week after the practice exams, students were given theactual exams. Immediately before each exam, students recorded ona form attachedto thefront of the exam their predictionsofwhatpercentageofitems they expected to get correct and their estimated

study times. Immediately after the exams, they recorded theirpostdictions onaform attached at the end. To ensure confidentialityof students' predictions and postdictions, a graduate assistantassignedto thecourse collected theforms andcompiled thedatafrom them.After each of the firsttwo exams were scored and returned,students were urged to reflect andanalyze why their predictionsand postdictions were or were not accurate and then develop aplanto prepareforthe next exam. For example,iftheir predictions werenot accurate, students were encouraged to identify the possiblefactors that may have contributedtoeither their overconfidenceorunderconfidence and to focus on those factorsforthe next exam. Tohelp prepareforthe final exam, students were freetogo over priorexams and practice exams.


4/11

TEST PREDICTION AND PERFORMANCE 163Resul ts

The results are organized according to our researchquestions: (a) Can students accurately predict test perfor-mance? (b) Does accuracy vary with test performance? (c)Does prediction accuracy increase over multiple tests? and(d) To what extent do prior performance and predictions ofperformance influence subsequent predictions? These samequestions were addressed in regard to students' postdictions.Therefore, findings concerning predictions and postdictionsare described within each section.Can Students Accurately Predict and PostdictTestPerformance and D oes AccuracyVaryWithPerformance?

Students were divided into five performance groups onthe basis ofthepercentage of total items answered correctly:Group 1 = 80-100%, Group 2 = 70-79%, Group 3 =60-69%, Group 4 = 50-59%, and Group 5 < 50%. Thesegroups were formed to correspond roughly with the gradingsystem established in the course: Group 1 students earnedmostly As, Group 2 students mostly Bs, Group 3 studentsmostly Cs, and Group 4 students mostly Ds or Fs. Group 5was added to examine more closely students whose perfor-mance was very low. Figures 1, 2, and 3 show the meanpredictions and postdictions for each performance group forExam s 1, 2, and 3 , respectively.Across the three exams, students who scored in testperformance Groups 1 and 2 were mostly accurate in theirpredictions and postdictions of test performance (i.e., meanjudgments differed by less than 8 percentage points fromactual performance), with the highest scoring students

showing consistent underconfidence in both predictive andpostdictive judgments. Students in performance Groups 3and 4 gave accurate postdictions (i.e., mean judgmentsdiffered by 8 percentage points or less) but showed strongoverconfidence in prediction (i.e., mean judgments were ashigh as 17 percentage points over actual performance).Students who scored below 50% showed little predictive orpostdictive accuracy, with their predictions and postdictionsexhibiting gross overconfidence (i.e., mean judgme nts dif-fered by a s much as 31 percentage po ints).

The strong relation between performance and predictiveand postdictive accuracy also w as evident in the correlationsbetween performance and absolute difference scores forprediction and postdiction (i.e., difference between students*prediction or postdiction scores and their actual performancescores). Higher performance scores were associated withsmaller differences between performance and prediction orpostdiction (i.e., greater accuracy). Pearson correlationswere - . 7 4 , .62, and .78 for prediction and .34,.51,and -.67 for postdiction for Exams 1,2, and 3, respectively( a l l p s < . 0 0 2 ) .

Does P redictive and Postdictive Accuracy IncreaseOver M ultiple Tests?Throughout the course, students received multiple formsof instruction and feedback on self-assessment. By th e thirdexam, they had received feedback not only on their predic-tions, postdictions, and performance on two exams, but onthree practice exams as well. They also had been encouragedrepeatedly during class periods to consider the im portance ofself-assessment on performance. We believed that if meta-

9

1009590858075

65605550

40

Underconfident

y i /


5/11

164 HACKER,BOL, HORGAN, AND RAKOW

Predicted Postdicted

1009590858075

_ 70ftg 65t 60

55504540 40 45 50 55 60 65 70 75 80 85 90 95 100Predicted and Postdicted Scores

Figure 2. Mean performance versus mean predicted and postdicted performance by subgroups forExam 2.

. Underconfident

GrOUD T

Gr =

.005.

Prediction= 36.44= 37.22= 10.91

+ .+ .+ .

test prediction and performance in a classroom context

Documents