designing a classroom test

Anthony Paolo, PhD, Desigining A Classroom Test, 2008, Retrieved online from http://www2.kumc.edu/comptraining/documents/DesigningaClassroomtest.ppt

Designing a Classroom Test


Content

• Understanding by Design (UbD)• Purpose of classroom test• Test blueprint & specifications• Item writing• Assembling the test• Item analysis


How do we currently identify what we are going to teach in our classrooms?


PDE’s Standards Aligned System and Understanding by Design (UbD)

Framework


PDE’s Standards Aligned System


UbD Framework

3 parts:

1. What do you want students to know?

2. How are you going to assess them to determine they learned what you wanted them to learn?

3. How are you going to teach the content to ensure all students learn?


UbD Templates

• Sample Unit Template• http://wilkes-ed520.wikispaces.com/file/vie

w/UbDOverview.pdf

• Blank Unit Template• http://wilkes-ed520.wikispaces.com/file/vie

w/UBD_Lesson_Template.doc

http://wilkes-ed520.wikispaces.com/file/view/UbDOverview.pdf

http://wilkes-ed520.wikispaces.com/file/view/UbDOverview.pdf

http://wilkes-ed520.wikispaces.com/file/view/UBD_Lesson_Template.doc

http://wilkes-ed520.wikispaces.com/file/view/UBD_Lesson_Template.doc


So what IS our test measuring?


What is the purpose of a classroom test?


Purpose of Classroom Test• Establish basis for assigning grades

• Determine how well each student has achieved course objectives

• Diagnose student problems

• Identify areas where instruction needs improvement

• Motivate students to study

• Communicate what material is important


How do you currently design your tests?


Test Blueprint

• Sometimes called the “test specifications”, the goal is to ensure the test covers the content and/or objectives in the proper proportions.


Test Blueprint• To ensure the test assesses what you

want to measure

• Ensure the test assesses the level or depth of learning you want to measure

• PSSA Test Blueprints: http://www.portal.state.pa.us/portal/server.pt/community/pennsylvania_system_of_school_assessment_(pssa)/8757/resource_materials/507610


Setting up a test blueprint

• Identify content to be assessed

• Identify depth of learning (ex. Bloom)

• Identify time spent on topic as compared to time taught in class

• Emphasis/number of points for each target


• http://www.teach-nology.com/worksheets/time_savers/bloom/


Test Blueprint Example

Learning Level(Number of test items)

Content/

Objective

Knows facts

(Recall)

Understanding Applies Principles

(Application)

Total

Kreb Cycle 3 5 2 10

Aquaporins 2 2 4 10

Cell Types 5 0 0 5

Total 10 7 8 25


Test Specifications

Topics Time spent on Topic

% of total class time

Number (%) of test

items

Kreb Cycle 10 hrs 40% 10 (40%)

Aquaporins 10 hrs 40% 10 (40%)

Cell Types 5 hrs 20% 5 (20%)

Total 25 hrs 100% 25 (100%)


Let’s Evaluate our own tests


Preparing & Assembling the Test• Provide general directions

– Time allowed (allow enough time to complete test)– How items are scored– How to record answers– How to record name /ID

• Arrange items systematically• Provide adequate space for short answer and

essay responses• Placement of easier & harder items


Item Writing – General Guidelines1

• Present a single clearly defined problem that is based on a significant concept rather then trivial or esoteric ideas

• Use simple, precise & unambiguous wording

• Exclude extraneous or irrelevant information

• Eliminate any systematic pattern of answers that may allow guessing correctly



• Avoid cultural, racial, ethnic & sexual bias. • Avoid presupposed knowledge which

favors one group over another (“fly ball” favors those that know baseball)

• Refrain from providing unnecessary clues to the correct answer.

• Avoid negatively phrased items (i.e., except, not)

• Arrange answers in alphabetical / numerical order



• Avoid “None of the above” or “All of the above” type answers

• Avoid “Both A & B” or “Neither A or B” type answers


Item Writing – Correct Answer is

• Longer

• More qualified or more general

• Uses familiar phraseology

• Is grammatically correct for item stem

• Is 1 of the 2 similar statements

• Is 1 of the 2 opposite statements


Item Writing – Wrong Answer is

• Usually the first or last option

• Contain extreme words (always, never, nonsense, etc.)

• Contain unexpected language or technical terms

• Contain flippant remarks or completely unreasonable statements


Item Writing – Grammatical Cues


Item Writing – Logical Cues


Item Writing – Absolute Terms


Item Writing – Word Repeats


Item Writing – Vague Terms


Item Writing• Effective test items match the desired

depth of learning as directly as possible

Applying & Analyzing• Applying: Implementing a procedure or process. • Analyzing: Breaking material into constituent parts,

determining how the parts relate to one another and to an overall structure or purpose through differentiating, organizing, and attributing.

– ITEM TYPES: MC, Short Answer, Problems, Essay


Comparison of MC & Essay1

Essay MC

Depth of learning

Can measure application and more complex outcomes. Poor for recall.

Can be designed to measure application and more complex outcomes as well as recall.

Item prep Fewer test items, less prep time

Relatively large number of items, more prep time

Content sampling

Limited, few items Broader content sampling


Comparison of MC & Essay2

Essay MC

Encouragement Encourages organization, integration & effective expression of ideas

Encourages development of broad background of knowledge & abilities

Scoring Time consuming, requires special measures for consistent results

Easy to score with consistent results.


Item Writing - ApplicationMC application of knowledge items tend to

have long vignettes that require decisions.

Case, et al. at the NBME investigated the impact of increasing levels of interpretation, analysis and synthesis required to answer a question on item performance.

(Academic Medicine, 1996;71:528-530)


Item Writing - Application


Interpreting test scoresTeachers

High scores = good instruction

Low scores = poor students

Students

High scores = smart, well-prepared

Low scores = poor teaching, bad test


Interpreting test scoresHigh scores

too easy, only measured simple educational objectives, biased scoring, cheating, unintentional clues to right answers

Low scorestoo hard, tricky questions, content not covered in class, grader bias, insufficient time to complete test


Item Analysis• Main purpose of item analysis is to

improve the test

• Analyze items to identify:• Potential mistakes in scoring• Ambiguous/tricky items• Alternatives that do not work well• Problems with time limits


Reliability• The reliability of a test refers to the extent

to which a test is likely to produce consistent results. • Test-Retest• Split-Half• Internal consistency

• Reliability coefficients range from 0 (no reliability) to 1 (perfect reliability)


Internal Consistency Reliability• High reliability means that the questions of

the test tended to hang together. Students that answered a given question correctly were more likely to answer other questions correctly.

• Low reliability means that the questions tended to be unrelated to each other in terms of who answered them correctly.


Reliability Coefficient Interpretation

General guidelines for homogeneous tests

• .80 and above – Very good reliability

• .70 to .80 – Good reliability, a few test items may need to be improved

• .50 to .70 – Somewhat low, several items will likely need improvement (unless short test 15 or fewer items)

• .50 and below – Questionable reliability, test likely needs revision


Item difficulty1

• Proportion of students that got the item correct (ranges from 0% to 100%)

• Helps evaluate if an item is suited to the level of examinee being tested.

• Very easy or very hard items cannot adequately discriminate between student performance levels.

• Spread of student scores is maximized with items of moderate difficulty.


Item discrimination1

• How well does the item separate those that know the material from those that do not.


Evaluation of Distractors • Distractors are designed to fool those that

do not know the material. Those that do not know the answer, guess among the choices.

• Distractors should be equally popular.

(# expected = # answered item wrong / # of distractors)


Thank you for your time

Questions ???

designing a classroom test

Documents

test measuring

test specifications

classroom testanthony

templ anthony paolo

important anthony paolo

measurepssa test blueprints

systemanthony paolo

classroom testestablish