running head: teacher made assessment activity…student.fgcu.edu/mlenglan/eportfolio/images/teacher...

Teacher Made Assessment Activity 1

Running head: TEACHER MADE ASSESSMENT ACTIVITY: EDF 6432

Teacher Made Assessment Activity: Accelerated Reader Teacher Made Tests

Margaret England

Florida Gulf Coast University


Abstract

This paper investigates the Validity and Reliability of one teacher made

Accelerated Reader quiz. During Item Analysis it was determined that the book

Two Crows Counting was not a valid selection for a 1.0 Accelerated Reader quiz.

The book contained sentence fragments, lacked verbs, and included incorrect

punctuation. The book did not support the reading skills being taught

in the classroom.


Teacher Made Assessment Activity: Accelerated Reader Teacher Made Tests

Statement of Purpose:

This paper investigates decisions about managing classroom instruction by evaluating a

teacher made Accelerated Reader quiz in an elementary school’s Accelerated Reader

(AR) technology-based literacy program database. AR Criterion Reference Tests (CRT)

are low-stakes formative assessments “intended to determine whether or not a student has

read a book” (Renaissance Learning1999 p.2) within their tested Zone of Proximal

Development (ZPD) reading level. “These quizzes are encouraging rather than

intimidating, chiefly because a student who has read a book should be able to pass the

quiz” (Renaissance Learning 1999, April). The AR program provides students with

immediate feedback after they take a 5-20 question quiz. The quiz score, points,

student’s average, number of quizzes, and goal are included on the TOPS (Three

Opportunities to Praise Students) report. A computer-generated comment is included with

space at the bottom for comments from the monitor and teacher.

The school’s database includes over 8,000 quizzes purchased from Renaissance Learning

(AR). Up to 500 teacher made quizzes may be added to the database. The teacher made

quizzes have not been evaluated for validity or reliability during the eight years AR has

been implemented in the school. The purpose of this paper is to improve the validity and

reliability of one teacher made quiz in level 0.2-1.9 through item analysis.

Table of Specifications

The Table of Specifications (Appendix A) for Two Crows Counting could be used with

any Accelerated Reader Quiz or group of quizzes with adjustments being made for 10 or

20 item quizzes. The percentages of the evaluation are 50% Literal Comprehension, 15%

Skill in Book Selection, 15% skill in Paired Reading, 15% Reading Independently, and


5% Recording in Reading Log. The main purpose of an AR quiz is to check the student’s

literal comprehension of text as a way to motivate students and determine their

understanding (knowledge) of the content of a book (Renaissance 1999 p.2). The table

could be modified for use as a 9-week unit with percentage from quizzes scores used for

the content score and the totals percentages from the book selection, reading log, dualog

(paired) reading, and independent reading as the skill scores. When the student reads

book independently or with a partner the text itself can provide instruction “that

effectively promotes both comprehension development and the exercise of critical

thinking” (Institute p.6). The teacher provides instruction through frequent monitoring of

students during independent and paired reading. During Status of the Class (monitoring)

the teacher has the opportunity to “assess ongoing work, monitor individual progress,

provide informative feedback to students, adapt instruction as needed, and ultimate

improve overall student performance” (Renaissance, 2002 p.1). The importance of the

teacher guiding reading and intervening when needed are key elements to a successful

AR program. (Institute 1999 p.6-7). Another skill in this table is the use of the Reading

Log, “student-involved record keeping” (Willis), which is used to motivate students and

to help them record and reflect on their improvement. Students are also assessed for

selecting a book within their Zone of Proximal development (ZPD).

Choices of Assessment and Item Format

In order to choose a teacher made AR quiz to evaluate a Quiz Takers Reading Practice

Report was run on all teacher made quizzes in the 0.2 - 1.9 ATOS (AR readability

formula) range taken by students during the current school year (2003). The percentage

scores were reviewed for these quizzes. The quiz, Two Crows Counting, taken by 21

different students, had the lowest percentage correct with 63.8%. The goal of AR is for

students to earn an average score of 85 –92% on AR quizzes within their ZPD range

(Institute 1997). Students in the school’s database who had taken the quiz during the


previous 2002 school year scored an average of 65% and 74% in the 2001 school year.

The AR database includes records for students currently enrolled in the school. The

averages of students who have withdrawn are not included in the yearly averages.

In order to develop the teacher made quiz the teacher writes 5 - 20 multiple-choice

questions, an answer and 3 distracters taken from a book. Five questions are usually

written for shorter books and 10 or 20 questions for longer books. The directions for AR

quizzes are the same for all quizzes and quiz takers. After reading a book, the student

logs into the AR database using their name and password, they locate the quiz for the

book they read, and then they take the selected-response AR Quiz. The questions are

presented in the order that matches the chronology of the book and reinforces the story

grammar of the book (Renaissance 1999, April). The program randomly changes the

order in which the 4 responses to questions are presented. Only one answer may be

selected to each question. Once a student selects an answer and goes to the next question

the program does not allow them to go back to change an answer. The actual quiz taken

by a student can’t be printed. Appendix I. shows the performance of students after the 2nd

revision of the quiz.

In order to earn points on a quiz students “need to score at least 60 percent on a 5- and

10-quesiotn quiz and 70 percent on 20-question quizzes” (Renaissance1999, April). The

number of questions is determined by the difficulty and content of the book. Lower level

picture books usually have 5 questions. “Questions typically focus on significant events,

character’s and literacy features of a book” (Renaissance 1999, April). If a student scores

60% or more the program will show them the correct answers to questions they answer

incorrectly. Information concerning the questions is not retained in the database.


Individual interviews or students writing the number of missed the questions are two

ways to record the questions answered incorrectly.

Instrument and Performance

In order to determine the ATOS readability of Two Crows Counting, a text file

(Appendix C) was sent twice to the Renaissance Learning ATOS (Readability For Books)

Analyzer. The ATOS Analyzer results (Appendix D) included a word count of 113

words, book level of 3.7, and 0.5-point value if a student answers all questions correctly.

When the teacher made quiz was developed the word count entered into the database the

database required that a minimum of 1,000 words or more be entered in the word count

field. Three years ago AR went from Fry Readability to ATOS Reading Levels. The

book level using the Fry Readability Index (Advantage Learning, 1996) was 1.0.

Analysis of Instrument: Validity, Reliability, and Item Analysis

As stated earlier the purpose of taking an AR quiz is to determine if a student has read a

book and if they have read a book then they should be able to pass the quiz. Is it valid to

interpret the scores from Two Crows Counting as measuring reading comprehension

(Nitko p.37)? Upon examination of Two Crows Counting it was noted that the book is a

counting book written in rhyming sentence fragments (Appendix C). The book contains

no verbs. Periods and commas were used in some of the sentence fragments. The text in

Two Crows Counting was then compared with text in 1.0 and 3.7 level books in the

school library collection. The 1.0 and 3.7 books contained complete sentences and

punctuation. The vocabulary and word count in the 1.0 book more closely matched the

Two Crows Counting vocabulary. Complete sentences, verbs, and punctuation are skills

that are taught to beginning readers. In order for validity to be attained in assessments,

including AR assessments, they should emphasize what you taught and contain content


worth learning (Nitko, p.39). A quiz on Two Crows Counting is not a valid assessment

of what is being taught in the classroom.

The scores for students taking the Two Crows Counting quiz were evaluated using

Livingston’s Formulae for Reliability of a Criterion Referenced test (RCR). In 2001 ten

students took the quiz with a Livingston’s RCR of .84 (Appendix E). In 2002 twelve

students had a RCR of .83 (Appendix F) and in 2003 twenty-one students had a RCR

of .85 (Appendix G). These scores support the stability of the performance of students’

with a 1.0 ZPD reading level that took the quiz during the past three years. The Formula

was used to determine the RCR of .80 for 16 students (Appendix H) who to the first

revision of the quiz and 11 students who took the 2nd revision of the quiz with the RCR of

.81. The two revisions were administered one day only to a limited number of students.

Appendix J. includes the student’s scores after the 2nd revision. Appendix B includes the

original questions and the two revisions. .

Analysis of Instrument

Interviews were conducted after students took the original and first revision quiz. The

changes in the instrument, observations, and comments follow.

Question 1:

One _____ rising

Answer: sun Distracters: crow bird balloon

Students who missed this question were looking at the picture clue of the crows flying

towards the sun (p.6) and chose crow or bird for an answer. The question will be

rewritten with a blank at the end of the stem and distracters that are not on the same page

as the text for the stem.

Question 1 Revision 1:

One sun _______

Answer: rising Distracters: smiling shining hiding


Students missing this question were using prior knowledge and chose shining as the

answer.

Question 1 Revision 2:

What was rising?

Answer: sun Distracters: boats cats people

After collaboration with a first grade teacher and reading the suggestions for improving

the quality of stems of multiple-choice items (Nitko, p. 150) the question and stem were

rewritten.

Question 2

Three people ______ and four trees blowing

Answer: rowing Distracters: running sitting eating

Students incorrectly chose running. When rewritten the blank was put at the end of the

stem.

Question 2 Revison 1

Three people rowing and four trees ___________

Answer: blowing Distracters: growing crowing glowing

Using prior knowledge students incorrectly chose growing. After collaboration the stem

was shortened.

Question: 2 Revison 2

Four trees __________

Answer: blowing Distracters: growing crowing glowing

Question 3

Seven ____wading and eight guess parading.

Answer: herons Distracters: crows birds ducks

Students missing this question chose ducks. The stem was shortened with the blank

added at the end of the stem.

Question 3 Revision 1


Seven herons ___________

Answer: wading Distracters: fading parading swimming

The stem was shortened with the blank added at the end of the stem. Students missing

this question chose swimming. The herons in the picture were walking in the water.


Seven herons ________

Answer: wading Distracters: fading eating flying

The distracters were rewritten.

Question 4

Four trees standing and _______ boats landing

Answer: three Distracters: two five six

Students are not allowed to open their AR book during testing and did not remember how

many boats were landing.


Four trees standing and three boats ________-

Answer: landing Distracters: sailing rowing sinking

The question was rewritten with the blank at the end of the stem

Students chose rowing as an incorrect answer. After collaboration with another teacher

the stem was shortened and another question selected.


The socks are ________

Answer: flapping Distracters: sailing rowing resting

Question 5

Big crow and small crow are __________

Answer: fast asleep Distracters: counting eating flying

This question was rewritten and different distracters were selected.



At the end of the story big crow and little crow are ________

Answer: fast asleep Distracters; counting sheep eating corn flying alone

One student chose flying alone.


At the end of the story big crow and little crow are ________

Answer: fast asleep Distracters; counting sheep eating corn flying alone

No changes were made to the stem or distracter.

After two revisions of the quiz, review by a colleague, and review of the ATOS Analyzer

there was evidence that the Two Crows Counting was not the appropriate book for a 1.0

teacher made AR quiz. The content did not assess the reading comprehension skills

being taught to beginning readers. The incomplete sentences, lack of verbs, and incorrect

punctuation were not an appropriated text for assessment. According to Popham reading

vocabulary and/or sentence structure that is too difficult is a factor affecting the validity

of an assessment.

Performance of Group and Individual Students

With each revision of the quiz there was an increase in the total percentage answers

correct. The average for this school year was 63.0 %, after the 1st revision it was 66.2%,

and 72.7 % after the 2nd revision. The target percentage was 80%. Only six students took

the quiz for both revision 1 and 2 after they reread the book. Each student’s score

showed an increased.

Student: Revision 1 # Correct Revision 2 # Correct

AA2 2 4

AA3 3 4

AA4 3 5

AA6 2 5


AA7 3 4

AA11 3 5

Five students took the quiz for the 2nd revision, but not revision 1.

AA1 5

AA5 1

AA8 1

AA8 3

AA10 3

The reading records for AA5 and AA8 were reviewed in the AR database. These

students consistently score 60% or more on AR tests in the 1.0 - 1.2 ZPD range.

Conclusion

AR quizzes are intended to determine whether or not a student has read a book within

their ZPD. Students who have read a book in their range should be able to pass a quiz.

(Renaissance Learning, 1999, April). Teachers need to be skilled in developing high

quality assessments (Nitko p.81). After careful analysis it was determined that a quiz

from the book Two Crows Counting is not a valid assessment of what is being taught in

the 1.0 reading level curriculum. The quiz and quiz records will be removed from the AR

database. In order to improve the validity of quizzes in the database the teacher made

quizzes will be evaluated using the Validity, Reliability, and Item Analysis approaches in

order to improve provides high quality assessments. It is our professional responsibility

to provide students with valid assessments (Nitko p.455-462).


References

Advantage Learning Systems. (1996). Accelerated Reader Installation User’s

Manuel, 82-84

Hutchcraft, G. Table of Specification for a Weather Unit in Middle School

Science. Message posted to http://courses.fgcu.edu~141

Hutchcraft, G. Livingston's Formulae for Reliability of a Criterion referenced test

Relval.xls. Message posted to http://courses.fgcu.edu

Hutchcraft, G. Steps for Building a Teacher-made Classroom Assessment.

Message posted to http://courses.fgcu.edu~144


Hutchcraft, G. Livingston's Reliability. Message posted to

http://courses.fgcu.edu~144

Institute for Academic Excellence, Inc. (1997, November). Critical Thinking and

Literature-Based Reading. Report from the Institute for Academic

Excellence, 1-8.

Kehoe, F. (1995, October). Basic Item Analysis for Multiple-choice Tests. Eric/

AE Digest, ED398 237.

Nitko, A. J. (2001). Educational Assessment of Students. Upper Saddle River,

New Jersey: Merrill Prentice Hall. (Original work published 1996)

Orgel, D. (1996). Two Crows Counting. Milwaukee, WI: Garth Stevens

Publishing.

Paul, T. J., M.B.A. (1996, October 4). Learning Information Systems: Theoretical

Foundations. Paper presented at National Research Center Conference.

Popham, W. J. What Do Classroom Teachers Really Need to Know about

Validity?. Retrieved from Dr. Gilbert Hutchcraft:


Popham, W. J. What Do Classroom Teachers Really Need to Know about

Reliability? Retrieved from Dr. Gilbert Hutchcraft:


Renaissance Learning. ATOS Analyzer,

http://www.renlearn.com/ar/atosanalyzer.htm

Renaissance Learning. (1999, April). The Design of Accelerated Reader

Assessments. Report from Renaissance Learning, 1-8.

Renaissance Learning. (2002). National Reading Studies Validate Accelerated

Reader, Reading Renaissance. Scientifically Based Research Abstract

Accelerated Reader and Reading Renaissance, L1542, 1-4.

http://www.renlearn.com/ar/atosanalyzer.htm


Renaissance Learning. (2002, July). How Scientific Research Support the School

Renaissance School Improvement Process. Report from Renaissance

Learning.

Spencer, S. J. (1995, November). Modifying tests for diverse learners.

Educational tests & measurements, 31(2), p.84. Retrieved April, 2003,

from Academic Search Elite: http://207.203.214.140:2211

Stiggins, R. (2001, Fall). Educational Measurement: Issues and Practices.

Retrieved March, 2003, from Dr. Gilbert Hutchcraft:


Willis, S. (1997, December). Using Assessment to Motivate Students.

Educational Update, 39(8). Retrieved March, 2003, from Dr. Gilbert

Hutchcraft: http://www.ascd.org/readingroom/edupdate/1997/1dec.html


Appendix A

Specifications:Table of Specification for book: Two Crows Counting

ObjectivesKnows Skill In Skill In Skill In Skill In Total

# items

Content Literal Comprehension

Selecting Use status of the

class or ratting scale to access.

Use status of the class or rating scale

to

Use Reading Log To

Evaluate Recording

of

VocabularyPractice and

Reinforcement

5 BookIn ZPD range

Dualog reading, students reading with a partner

Access students reading

independently

Title, LevelAnd

ScoreOn Quiz

by students

5

% Of Evaluation

50% 15% 15% 15% 5% 100%

The above table of specifications is the book Two Crows Counting by Doris Orgel. Knowledge, literal comprehension of the vocabulary in the book would be assessed through the Accelerated Reader Computer Program. Performances would be accessed through frequent monitoring and status of the class taken by

the classroom teacher.

The numbers in the cells represent the number of items measuring the content area and the type of target objective. There are 5 items on the whole test so the percentages show what percent of the test measures each type of objective (you

could show the same thing for each content area. The percentage of emphasis on the test should roughly match the percentage of emphasis given in instruction.


Appendix B


Appendix C

Big crow, small crow, on the wing, counting, counting everything...

1 ONE sun rising

1 Two shadows gliding

3 Three people rowing

4 Four trees blowing

5 Five cats lapping

6 Six socks flapping

7 Seven herons wading

8 Eight geese parading

9 Nine farmers haying

10 Ten children playing

Small crow, big crow, home the go, counting things they see below...

10 Ten children snoozing

9 Nine farmers snoring

8 Eight geese resting

7 Seven herons nesting

6 Six socks hardly stirring

5 Five cats purring

4 Four trees standing

3 Three boats landing

2 Two shadows slowing

1 One sun glowing

Big crow, small crow, dreaming deep, counting nothing fast asleep.


Appendix D

Renaissance Learning, Inc.

Wednesday, April 09, 2003 4:39:45 PM

MessageFrom: ATOS <[email protected]>

Subject: ATOS for book results (Confirmation # 8891)To: Margaret L. England

Attachments:

Thank you for your inquiry. The file you submitted was analyzed, and these are the results:

Confirmation Number: 8891Title of entry: Two Crows Counting

Author First Name: DorisAuthor Last Name: Orgel

ISBN: 083681617xATOS book level: 3.7AR Points: 0.5Total Word Count: 113

Renaissance Learning, Inc.

Appendix E


Appendix F


Appendix G

Appendix H


Appendix I


Appendix J

running head: teacher made assessment activity…student.fgcu.edu/mlenglan/eportfolio/images/teacher...

Documents