summative evaluation-rubric and test.ppt

8/10/2019 SUMMATIVE EVALUATION-RUBRIC AND TEST.ppt

1/52

Summative Assessment:

Rubrics and Tests

Effective Teaching and Learning

Baker College

1


2/52

Outcomes

Apply a systematic process for creating atest blueprint

Identify attributes of effective test questions Explain the advantages and disadvantages

of different types of test questions

Assess the quality of tests and test items Create samples of effective questions

2


3/52

What type of assessment?

Procedural knowledge

Declarative knowledge

3


4/52

Test Writing Process

Planning the test

Writing test items

Selecting test items Formatting the test

Assessing the test

Revising the test Using the test

After the test

4


5/52

Planning the Test

Content Blueprint

Learning outcomes

Weight

Length

Types of items

Number of items

5


6/52

Test Blueprint

Learning outcomes Weight

Apply a systematic process for creating a test

blueprint

Identify attributes of effective test questions

Explain the advantages and disadvantages of

different types of test questions

Assess the quality of tests and test items

Create samples of effective questions

6


7/52


8/52

Average Response Time

Item Type

True-false

30 seconds

Multiple-choice and Multiple-

answer

6090 seconds

Matching and Ordering

30 seconds per response

Short Answer

120 seconds

Completion

60 seconds

Essay

1030 minutes

8


9/52

Writing Test Items

Simple and direct wording

Avoid jargon

Avoid trivia items Match items to learning outcomes

Each item has an agreed upon correct

answer Write more questions than you will

need

9


10/52

Multiple-choice Items

Stem

Direct question

Incomplete statement

Responses

One correct answer

Multiple distracters

10


11/52


12/52

Responses

3-5 per item

Avoid all of the above and none of the

above Grammatically correct with stem

Similar length and structure

Avoid absolute words Listed in a logical order

Mutually exclusive and not overlapping

12


13/52

Distracters

Plausible

Common misconceptions

Logical misinterpretations

Clichs

Partial answers Technical terms or jargon

13


14/52

Example

What is the minimum number ofresponses for a multiple-choice item?

A) 2B) 3

C) 4

D) 5

14


15/52

Application Example

What problem exists in the following multiple-choice stem:

________ is the most common type of test item.A) Absolute words should be avoided in the stem.

B) The stem contains more than one idea orconcept.

C) Not enough information is presented to answerthe question.

D) The fill-in the blank should come at the end.

15


16/52

Analysis and Evaluation

Example StemAn instructor was asked the following question: "Briefly list and explain howyou develop a test. As an answer, this instructor wrote the following:

I begin by going through the chapter and writing questions based on the

material in the text and my lectures. Then I decide how many questions I wantand select the best questions from the list that I have developed. I format thetest and add instructions. After a few days, I review the questions and makeany revisions that need to be done and remove any jargon or wording lifteddirectly from the text. Then I use the test in class. Based on how the classdoes, I may make changes for the next time I teach the class.

16


17/52

Analysis and Evaluation

Example ResponsesBased on the process described in EffectiveClassroom Tests, how would you judge this answer?

A) EXCELLENT (all steps in the right order with correct,

clear, and complete descriptions)B) GOOD (all stages correct in the right order, but thedescriptions are not as complete as they should be).

C) MEDIOCRE (one or two stages are missing, OR thestages are in the wrong order, OR the explanations arenot complete, OR the explanations are irrelevant)

D) UNACCEPTABLE (one or more stages are missing ANDthe explanations are not complete AND/OR they areirrelevant)

17


18/52

Poor Question 1

Good multiple choice items:

A) are easy to write

B) can only test memorized contentC) are better than essay items

D) there is no such thing

E) can test a wide range of content

18


19/52

Poor Question 2

Which of the following characteristics isnot true of completion test items but is

an important distinguishing attribute ofmatching tests, multiple-choicequestions, and true-false items?

A) They are objective test items.

B) They require knowledge recognition butnot production.

C) Much more difficult to construct.

19


20/52

Poor Question 3

Which of the following statements is FALSE?A) Misfeasance is the improperly doing of an illegal

act.B) Nonfeasance is improperly doing a legal act.

C) Nonfeasance is the failure to do an act that onemust do legally.

D) Misfeasance is the failure to PROPERLY do anact that one has a duty to perform.

E) None of the above.

20


21/52

Poor Question 4

__________ is/are the best method todetermine if students have learned

something.A) Comprehensive Exam

B) Homework Assignments

C) Pop QuizzesD) Research Paper

21


22/52

Selecting Test Items

Outcome Weight x # Questions by Type =#Questions of Each Type for Outcome

22


23/52

Formatting the Test

Group items by type

Sort items by increasing difficulty

Add instructions

Review layout and pagination

Write answer key

23


24/52

Assessing the Test

Self 2-3 days after writing the test Clarity Clues in items to other items

Non-expert Clarity Contextual clues

Peer Content Weighting to outcomes Answer key

Students Clarity Content

24


25/52

Test Taking Procedures

Use of notes or other materials

Time limits

25


26/52

After the Test

Item Analysis

Areas for review

Test revisions

26


27/52

Activity

Write two test questions on any topic

One question should be an example of

a good test item One question should be an example of

a poorly written test item

27


28/52

Share

Share your two questions with apartner

Can they determine which is good andwhich is bad?

Can they explain what makes one

poorly written?As a team, how can you fix the poorly

written questions?28


29/52

Intermission

29


30/52

Outcomes

Determine what characteristics areimportant in evaluating student work

Evaluate rubrics, analytic scales, andother evaluation methods

Describe the contents of a good rubric

Identify rubrics already in use at BakerCollege

Begin work on a rubric for a class

30


31/52

What is a Rubric?

A rubric is a scoring tool or guide that liststhe specific criteria and the ranges for

multiple levels of achievement for a pieceof work or performance. A rubric consistsof a set of well-defined factors and criteria

describing the dimensions of anassignment to be assessed or evaluated.

31


32/52

Parts of a Rubric

Scale (columns)

Dimensions (rows)

Criteria descriptions (cells)

32


33/52

Benefits of Rubrics

Communicates the instructorsexpectations

Streamlines the process for feedbackto the student

Facilitates equitable grading

Standardizes assessment acrossdifferent instructors

33


34/52

Uses for Rubrics

Papers

Presentations

Projects Essays

Homework

Case Studies

Participation/Class

Discussion Portfolios

34


35/52

Types of Rubrics

Analytic

Page 11

Holistic Page 12 and 13

Check List

Page 14

Scoring Guide

Page 1535


36/52

Creating a Rubric

Identify components/outcomes of theassignment

Determine a scale Add criteria

Assign points

Set component weights (optional) Assess the rubric

Test and revise

36


37/52

Activity

Split into groups of 3-4

Determine team roles

Select an assignment that needs arubric

Can be a specific assignment, such as a

research paper for ENG 102 Can be of a more general nature such as

a class presentation37


38/52

Step 1: Identify

Components List 5 major objectives/outcomes of

the assignment

Write these items as the row headersof the sheet provided

38


39/52

Step 2: Determine a Scale

Aim for 3-5 levels

Can use an odd or even number of

items Use the headings on the next slide for

ideas

Write these as column headings on thesheet provided

39


41/52

Step 3: Add Criteria

Create descriptions for each level ofperformance for each criteria in the

cells of the rubric Bullet points

Paragraphs

Write these criteria in the cells of thesheet provided

41


42/52

Step 4: Assign Points

Assign points for each level ofperformance

Can use either of the following: Discrete values (5, 4, 3, 2, 1)

Ranges (10-9) for each level

Indicate the point value on the sheetprovided Normally placed with the scale

42


43/52

Step 5: Set Component

Weights Allows for different levels of importance

Spelling/grammarmore or less important thancontent?

Determine if weights are necessary for yourrubric

Assign weights accordingly

See example on page 17-18 of the handout

43


44/52

Step 6: Assess the Rubric

Assess your rubric using a metarubric See examples on page 11-15 of your handout

Conduct a peer reviewAsk one or two other instructors to review your

rubric

Provide time for student review

Allow students to ask questions and makecomments

44


45/52

Group Project

Trade rubrics with another group

Assess the rubric using a metarubric

from page 11-15

45


46/52

Discussion

What metarubric(s) did you use? Why?

What did you see on the other teams

rubric that you liked? Could you understand the assignment

easily by reviewing the rubric?

46


47/52

Step 7: Implement andrefine

Refine your rubric based on feedbackfrom other instructors and students

Make notes each time you use therubric for continuous improvementpurposes

Share with others

47


48/52

Rubric Reliability & Validity

Reliability the likelihood that a given measurement

procedure will yield the same description of a

given phenomena if the measurement isrepeated.

Validity the extent to which a specific measurement

provides data that relate to commonly acceptedmeanings of a particular concept.

Babbie, 1986

48


49/52

Reliability Requires

Instructor should reach sameconclusion each time

Different instructors should reachsimilar conclusion (interrater reliability)

49


50/52

Interrater Reliability

Independently score a set of studentsamples

Review responses for consistent andinconsistent responses

Discuss and reconcile inconsistencies

Repeat with second group of samples Maki, 2004, p. 127

50


51/52

Validity Requires

Reliability

Comprehensiveness

Cover all outcomes

Economy

Space is usually limited, so be selective

about what goes into the rubric Balanced scoring and weighting

51


52/52

End of lectureAny question

52

summative evaluation-rubric and test.ppt

Documents