construct evaluations

66
Construct Evaluations

Upload: jadyn

Post on 12-Jan-2016

58 views

Category:

Documents


1 download

DESCRIPTION

Construct Evaluations. True-False. Meaningful true-false items are difficult to compose Many statements are neither completely true nor completely false Amount of guessing may be high Test takers have a 50 percent chance of being correct Tend to measure recognition only. True-False. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Construct Evaluations

Construct Evaluations

Page 2: Construct Evaluations

True-False

• Meaningful true-false items are difficult to compose– Many statements are neither completely true

nor completely false– Amount of guessing may be high

• Test takers have a 50 percent chance of being correct

– Tend to measure recognition only

Page 3: Construct Evaluations

True-False

• Precautions– Provide accurate directions– Balance the true and the false items and

distribute them so no pattern is evident– Keep true and false statements about equal in

length– Used positive rather than negative statements

• Avoid the use of none, never, no, or not and always

Page 4: Construct Evaluations

True-False

– Construct true-false items that are either true or false, not partially true or false

– Provide a space to the left of each item where the student can insert the correct answer for an item which is false

• This distinguishes an accidental correct guess from a correct response

Page 5: Construct Evaluations

True-False• Example• Directions: read each of the following statements.

If a statement is dramatically correct, circle the T. If the statement is incorrect, circles the F and change the underlined word to make this statement dramatically correct. Place the correct word in the blank beside the F.

T F _____ 1. To whom should I give the umbrella?

T F She__ 2. Her and I signed up for the trip

Page 6: Construct Evaluations

Multiple Choice

• Consists of an incomplete statement or a question followed by four alternatives

• Three parts– The incomplete statement-the stem– The correct answer– 3 alternatives-distractors

Page 7: Construct Evaluations

Multiple Choice• Example

Directions: in the space provided beside each question, write the letter of the response which best completes the statement.

_a_ 1. The term used to describe the one who receives the stated payments to be made on the death of the insured is

a. Beneficiary

b. Insured

c. Policy owner

d. Survivor

Stem

Correct Response

Distractors

Page 8: Construct Evaluations

Multiple Choice

• 12 guidelines1. Word each multiple-choice item as either a

statement or a question• What is the period covered by the balance

sheet?a. Year

b. Month

c. Day

d. Week

Page 9: Construct Evaluations

Multiple Choice

2. Make sure the stem contains a single clearly worded idea• Good: the rule of 72 is used to determine

a. Average annual rate of returnb. Number of years needed to double investmentc. Total return on investmentd. Daily rate of return

• Poor: what is the chief purpose of using the rule of 72?a. Average annual rate of returnb. Number of years needed to double investmentc. Total return on investmentd. Daily rate of return

The term purpose may confuse the reader The alternatives are not phrased in a way as to answer the

question

Page 10: Construct Evaluations

Multiple Choice

3. Keep the correct response the same length as each distractor

4. If a negative statement is used, underline it, italicize it, or place it at the end of the stem or do both• Good: an employee whose only employment is in the

factory is eligible to invest in all of the following excepta. An individual retirement account

b. Keogh plan

c. Stock option plan

d. Municipal bonds

• Poor: An employed factory worker is not eligible to invest in which of the following?

Page 11: Construct Evaluations

Multiple Choice

5. Make certain that the correct response is accurate or certainly the best of alternatives

6. Avoid any clues such as repeating keywords from the stem in the alternatives

7. Determine that all the distractors are believable

8. Vary the length of the correct response from item to item to avoid giving clues

9. Make certain that all alternatives are grammatically correct and written in parallel form

Page 12: Construct Evaluations

Multiple Choice

10.Avoid alternatives like “none of the above” or “all of the above”

11.Vary good position of the correct response from question to question

12. In formatting the questions, put each alternative on a separate line to permit easy reading

Page 13: Construct Evaluations

Matching

• Parallel columns are usually provided for matching items

• The items below column A are called premises and below column B responses

Page 14: Construct Evaluations

MatchingDirections: in this space provided beside column A, write a letter of the statement in column B that best completes each thought expressed. Each item in column B may be used more than once.

Column A Column B

b. 1. An equal exchange of assets and liabilities

a. Increases owners equity

a. 2. And unequal exchange of assets resulting in an increase in total assets with no change in liabilities

b. Does not affect owners equity

b. 3. An equal exchange of liabilities with no change in assets

c. Decreases owners equity

b. 4. An equal increase in both assets and liabilities

c. 5. An equal exchange of assets resulting in a decrease in total assets with no change in liabilities

PremisesResponses

Page 15: Construct Evaluations

Matching

• Matching items are best suited to large units of subject matter which are sufficiently homogeneous to require discrimination on the part of the students

Page 16: Construct Evaluations

Matching

• Guidelines for designing matching questions

1. Provide clear, simply stated directions

2. Restrict the number of items in the stem column (column A) (no more than 10)

3. Include fewer or more alternatives (column B) than items in the stem column (column A) to prevent students from guessing

Page 17: Construct Evaluations

Matching

4. Arrange items in the stem column in logical order or in ascending or descending order

5. Arrange alternatives in logical order

6. Keep all items in a matching question on one page

Page 18: Construct Evaluations

Completion

• Simply an incomplete statement

• The examinee applies the correct word, amount, or date

Page 19: Construct Evaluations

Completion

• Directions: for each of the items below insert the correct response in the blank space at the end of the sentence

1. The maturity value for an $850, 120-day note at 14 percent is _?_.

2. The date of the census for which punch cards were first use is _?_.

Page 20: Construct Evaluations

Completion

• Use completion items only for those questions for which there is only one correct response

• Some guidelines– Give explicit instructions– But the blank at the end of the sentence

Page 21: Construct Evaluations

Completion

– Keep all blank lines the same size but provide enough room for the longest answer

– Provide no hints• First letter• Number of blank spaces• Length of the blank line

– Avoid copying statements directly from the textbook

• Only evaluates recognition

Page 22: Construct Evaluations

Short Answer

• The examinee is presented with a question and required to supply a word, name, date or amount

Page 23: Construct Evaluations

Short Answer

• Directions: Insert the correct response in the space to the left of the number of each test item

1960 1. In what year was the computer first used to process census data?

Page 24: Construct Evaluations

Short Answer

• Use test items for which there is only one correct response– Correct

• What is the name of the COBOL program division which gives step-by-step commands for the computer to follow in processing the data?

– Answer: procedure division

– Incorrect• Explain the divisions of a COBOL program and the function

of each– Suggests an essay, which is inappropriate for “understanding-

level” outcomes– Not a short answer item

Page 25: Construct Evaluations

Design of Mastery Tests

• Chief purpose-discriminate between the student who has fully mastered a competency and one who has not

• The progress test– The person identified as non master can seek

additional learning experiences and later be retested

– Each learner should eventually achieve the minimum level of performance required

Page 26: Construct Evaluations

Design of Mastery Tests

• Three critical decisions1. Eliminate subjectivity from scoring

• Scored each item either as correct (1 point) or incorrect (0 points)

• No judgment is made; the answer is either right or wrong

Page 27: Construct Evaluations

Design of Mastery Tests

2. Set a cutoff score• A student who achieves a 70 percent score and

passes the test has not mastered 30 percent of the items

• The cutoff score must be set high in order to increase the probability that it will differentiate masters from non masters

• Experts suggest a cutoff of 85 to 90 percent

Page 28: Construct Evaluations

Design of Mastery Tests

3. Decide on number of items per competency• A useful rules found is between 12 and 20 test

items are needed for each enabling objectives that you wish to test

• A mastery test is appropriate only for competencies which are essential

• Nice-to-know competencies should not be tested by mastery test

Page 29: Construct Evaluations

Design of Norm-Referenced Test

• The various types of objective items used for mastery test are also appropriate for an norm-referenced test

• The norm-referenced test is designed differently

Page 30: Construct Evaluations

Design of Norm-Referenced Test

• Four major guidelines– The purpose is to gather information to rank

individuals within the group-only a sample of the content in the unit is selected for the test

– Seek to avoid questions which all students are likely to answer correctly

– A wide variation in scores is sought– Since several competencies are being

measured, the norm-referenced test contains more questions than the mastery test

Page 31: Construct Evaluations

Design of Norm-Referenced Test

• No decision on mastery can be made from a norm-referenced test

Page 32: Construct Evaluations

Performance Test

• Includes essay questions, product performances, and process performances

• It is essential that as much objectivity as possible be built into these tests

Page 33: Construct Evaluations

Performance Test

• Judgments made on performance test are more likely to be made objectively when the teacher uses– Rubrics– Rating scales– Checklist– Assessment instruments designed at the time

to test is constructed

Page 34: Construct Evaluations

Essay Questions

• Appropriate for evaluating the competency when a student is required to– Select relevant information– Organize– Integrate– Analyze– Synthesize– Make decisions

Page 35: Construct Evaluations

Essay Questions

• The process of assigning points to each essay question is subjective and subject to human error

• Essay questions fall into two categories– Restricted response– Extended response

Page 36: Construct Evaluations

Essay Questions

• Restricted response– Limits the content involved as well as the

response• Example 1: describe two causes of inflation. For

each cause describe two actions the government may take in controlling it

– The student must simply recall two causes– Evaluates understanding of inflation and demands

analysis of remedies

Page 37: Construct Evaluations

Essay Questions

• Example 2: analyze the two income statements attached and identify any problems the Best Company is experiencing. For each problem, explain a possible cause or causes and recommend one or more solutions

– Involves analysis– Content is limited by the case, as is the response

Page 38: Construct Evaluations

Essay Questions

• Extended response

• Gives the student full responsibility for– Selecting relevant information– Organizing– Analyzing– And presenting the analysis

Page 39: Construct Evaluations

Essay Questions

• Example: contrast Keynesian theory with supply-side theory in terms of objectives, applications, and possible effects on (a) a declining economy and (b) an inflationary economy

– Forces a student to analyze and communicate effectively in writing

Page 40: Construct Evaluations

Essay Questions• Guidelines

1. Use essay only for those learning outcomes which cannot be measured by objective items• Develop a classification system• Summarize prominent ideas• Analyze the task and formulate logical steps for

accomplishing it• Create a classification• Explain a phenomenon• Decide which data are relevant• Conclude from information given• Defend decisions or actions• Predict trends or actions based on data or information

Page 41: Construct Evaluations

Essay Questions

2. Design the question to elicit the behavior specified in the learning outcome• Example: if you want a student to analyze

financial statements, “conclude from information given,” is appropriate

• Essay question: from the financial statements provided for the Lexion Company, determine the financial condition of the company and present a complete analysis of the company’s financial strengths and weaknesses

Page 42: Construct Evaluations

Essay Questions

3. State each question precisely so that the individual understands what is expected

– Poor: described the impact of word processing systems on office equipment• Fails to provide a clue as to what is expected

(impact on numbers of people, task performed, or organization, or all?)

– Better: describe the impact of word processing in the office on productivity, work measurement, workflow, and career paths

Page 43: Construct Evaluations

Essay Questions

4. Specify the maximum number of points to be awarded for a correct and complete answer beside each question

– Aids the student in managing time

5. Prepare a key to use in evaluating responses at the time the questions are designed and written

Page 44: Construct Evaluations

Product and Processed Performances

• Subjective judgments sometimes interfere with the evaluation process

• Teachers have many opportunities to observe students affective behaviors

Page 45: Construct Evaluations

Product and Processed Performances

• Teachers may structure observations using some form of assessment instrument or checklist, for example– Individual working independently—reading,

performing the task, organizing for performance, or observing a peer

– Individuals seeking assistance from peers– Individual giving assistance to peers– Individual working with one or more peers while

interacting; observing: perhaps as part of a team; peers giving report or performing; organizing a project; or performing a task

Page 46: Construct Evaluations

Product and Processed Performances

• A teacher planning to observe the student’s oral presentation might decide to observe both cognitive and affective behaviors– Cognitive

• Accuracy of content• Organization• Quality of summary• Use of correct grammar

Page 47: Construct Evaluations

Product and Processed Performances

– Affective• Eye contact and other nonverbal language• Attitude of the speaker when responding to

questions

• A rubric, rating scale, checklist, or assessment instrument can be used to reduce subjectivity– Such instruments are designed before the

learning process begins and shared with the learner

Page 48: Construct Evaluations

Product and Processed Performances

• Rating scales– Sets of statements describing qualities of the

product or process being judged– Descriptors placed on a scale guide the

evaluator in determining where on the scale the product being judged falls

Rating Scale for Oral Presentations.doc

Page 49: Construct Evaluations
Page 50: Construct Evaluations

Product and Processed Performances

• Students should be provided with copies before the learning process begins

• When learners are aware of the criteria on which a evaluation will be based they can concentrate their energies on meeting them

• A single performance does not necessarily represent later performances– Many performances are required

Page 51: Construct Evaluations

Rubrics

• Allow assessment to be more objective and consistent

• Greatly aid in grading authentic assessment

• Describes criteria clearly and the ranges of acceptable and unacceptable performance behavior

• Offer clear performance standards to students for agreed-upon standards

Page 52: Construct Evaluations

Rubrics

• Common features– Focus on measuring a stated objective

(performance, behavior, or quality) – Use a range to rate performance – Contain specific performance characteristics

arranged in levels indicating the degree to which a standard has been met

Page 53: Construct Evaluations

Rubrics

• Development and design– Instructors may prefer starting with a checklist– Determine precisely what behaviors will be

looked for – Determine the standard for the performance

• When possible, focus on presence of behaviors rather than the absence of behaviors-avoid negatives

Page 54: Construct Evaluations

Rubrics

• Rubrics force a teacher to clearly define expected student outcomes

• Rubrics require refinement and change as lessons and content are updated

• Rubrics should be teacher and student driven– The more ownership the student carries the

better the performance

Page 55: Construct Evaluations

Rubrics

• Concerns– Should rubrics be given to students initially?– Will students selected lower criteria levels for

the “commendable” level?– Rubrics take time to develop

• Additional time may be needed for students' ideas to be solicited

• Even more time is necessary if an instructor tests the rubric or request peer review

Page 56: Construct Evaluations

Rubrics• Endorsements

– Students enjoy the amount of feedback– Student ownership in the learning process

increases with the use of authentic assessment• Students see the purpose and real world connectedness

– Instructors enjoy the ease of grading• Written contents are minimized

– Rubrics minimize the potential for student-teacher confrontations

– Subjectivity is reduced

Presentations Rubric Teaching Act Rubric

Page 57: Construct Evaluations

Rubrics

• Development and design (Cont.)

• Used demonstrated verbs such as– Expresses– Discusses– Selects

• Numeric values may be assigned to each column, row, or individual cell

Page 58: Construct Evaluations

Rubrics• Rubric Template

Beginning1

Developing2

Accomplished3

Exemplary4

Score

 Stated Objective or Performance 

Description of identifiable

performance characteristics reflecting a

beginning level of

performance.

Description of identifiable

performance characteristics reflecting

development and

movement toward

mastery of performance.

Description of identifiable

performance characteristics

reflecting mastery of

performance.

Description of

identifiable performance characteristics reflecting the highest

level of performance

.

 Stated Objective or Performance  

Description of identifiable

performance characteristics reflecting a

beginning level of

performance.

Description of identifiable

performance characteristics reflecting

development and

movement toward

mastery of performance.

Description of identifiable

performance characteristics

reflecting mastery of

performance.

Description of

identifiable performance characteristics reflecting the highest

level of performance

.

If the description is to vague the

benefits of student

understanding and objectivity

are lost

Page 59: Construct Evaluations

Rubrics

• http://teach-nology.com

• http://school.discovery.com/schrockguide/assess.html

• http://rubistar.4teachers.org/index.php

Page 60: Construct Evaluations

Grades

• Summative grades typically have two (2) purposes– Report the student’s readiness to move to the

next level– Motivate students

Page 61: Construct Evaluations

Grades

• Possible data sources– Classroom observations– Quizzes– Assignments– Tests– Performance-based assessments

Page 62: Construct Evaluations

Grades

• Using technology is highly recommended for data organization– Spreadsheets– database

Page 63: Construct Evaluations

Grades

• Data analysis/development– Two common methods

• Weighted averages• Percentage of total points

– Weighted averages• The weight (percentage value) of each item is

determined beforehand.• The sum total of all of the items evaluated must

equal 100 percent.

Page 64: Construct Evaluations

40%

40%

15%

5%

Test

Homework

In-class assignments

Extra Credit

Grades

Page 65: Construct Evaluations

Grades

• Percentage of the total points– The weight of each item is reflected in the number of

points each item is valued

• Example– Each test may be worth 100 points– Each homework assignment may be worth 20 points– Each in class assignment may be worth 10 points

Each graded item may have a different value!

Page 66: Construct Evaluations

Grades

Test 1

Homework 1

Class Assign 1

Test 2

Homework 2

StudentTotal

TotalPossible %

Item Value 100 20 25 50 100 295 295 100%

Joe Blow 58 5 20 47 96 226 295 77%

Ima Headcase 85 16 25 44 89 259 295 88%

Lay Z. Student 28 9 3 17 21 78 295 26%

Chapped Lipps 99 18 25 48 99 289 295 98%