developing and applying tools: rubrics and exam questions linda dickens, ph.d dawn zimmaro, ph.d....

Developing and Applying Tools: Rubrics and Exam Questions

Linda Dickens, Ph.D

Dawn Zimmaro, Ph.D.

John Kucsera, M.A.

Agenda

1. Overview of SACS

2. Mapping Assessments to Program Outcomes

3. Writing Critical Thinking Exam Questions

4. Creating Rubrics

•Academic units

•Administrative units

Agenda

1. Overview of SACS

2. Mapping Assessments to Program Outcomes

3. Writing Critical Thinking Exam Questions

4. Creating Rubrics

Mapping Assessments to Program Outcomes

At what level should you collect data to measure your outcomes?

• Classroom• Program

Issues to consider

• What existing course assignments, exams, projects, etc. can you use to measure your learning outcomes?

• What program level assessments can you identify or develop?

• What is unique about your program that is measurable?

Classroom level

• Exam• Assignment• Presentation• Paper• Project

Program level

• Exit exam• Thesis/dissertation• Capstone course paper or project• Portfolio• Licensure/professional exam• Exit survey

English Department B.A.

Example learning outcome:

Students will be able to construct arguments and communicate them effectively in writing.

Classroom level assessment

Read description of E 370W – Major Authors (Emerson)

Assignment

• Writing assignment that asks students to develop an argument about an issue– Emerson’s efforts to examine the authority and effects of

religious and educational institutions help frame discussions about literature and education for subsequent generations.

– Given the above statement develop a central thesis with supporting arguments about how Emerson’s works have helped to frame discussions about literature and education.

Program level assessment

• A collection of individual student’s writing assignments on constructing arguments taken from several English courses – All writing assignments are evaluated as a body of work

– Over the course of the whole degree program

Second learning outcome

Example learning outcome:

Students will be able to demonstrate knowledge of the history and the structure of the English language.

Classroom level assessment

• Classroom exam– Topics:

• History of English: Old to Early Modern English

• History of English: Modern and Future English

Topics should all relate to the learning objective

Program level assessment

• Exit exam– Questions taken from all the courses that relate to the

learning outcome

Sample assessment map

• See handout

Discussion

• Have you tried this approach? What worked? What didn’t?

• If you haven’t tried this, why not?

Agenda

1. Overview of SACS

2. Mapping Assessments to Program Outcomes

3. Writing Critical Thinking Exam Questions

4. Creating Rubrics

Writing Critical Thinking Exam Questions

• Purpose of session• Learning Outcomes• Critical thinking exam questions• Objective and Subjective exam questions• Application activity• Final thoughts

Purpose

• Objective: – To compose critical thinking questions for program or

classroom level exams

• WYMIWYG

• Aligning outcomes with methods

Learning Outcomes

• Program to classroom level (alignment)• General statements to specific outcomes

– Person centered

– Clear, specific, and measurable

• Classroom learning outcomes should contain– Procedure/method (assignments that will be provided)

– Behavior (what the students will do; usually the action verb)

– Criteria (how well a learner performs to be judged adequate and the qualitative description of the criteria)

PhD Program Example

• Program Educational Objective: – Sixty percent of graduates from the College of Education will attain positions as

college faculty.• (Program) Learning Outcome 1:

– “Will prepare students as future researchers”– Graduating students will demonstrate fundamental knowledge of quantitative,

qualitative, and mixed research designs.”• Procedure 1.1 (Classroom learning outcome):

– “Course EDP 382 will focus on the fundamental knowledge necessary to conduct quantitative research.”

– “Given an exam administered in EDP 382, at least 80% of students will demonstrate a fundamental knowledge of quantitative research designs by achieving a score at or above an 80 out of 100. The exam will test the extent to which students can compute common statistics, identify assumptions of research designs, evaluate….

Writing Critical Thinking Exam Questions

• For critical thinking:– Focus on higher levels of Bloom’s taxonomy

Bloom’s taxonomy for the cognitive domain

• Remember– recall or memorize information

• Understand – translate from one form to another (e.g., restate in own words)

• Apply– apply knowledge to a novel situation

• Analyze– examine a concept and break it down into parts

• Evaluate– make judgments using standards of appraisal

• Create– put information together in an unique or novel way

Writing Critical Thinking Exam Questions

• For critical thinking:– Focus on higher levels of Bloom’s taxonomy

• Behavior component of learning outcome– Ex: Given an exam administered in in EDP 382, at least 80%

of students will demonstrate a fundamental knowledge…

– Action verbs (handout)

Writing Critical Thinking Exam Questions

• For critical thinking:– Focus on higher levels of Bloom’s taxonomy

• Behavior component of learning outcome– Ex: Given an exam administered in __ course, at least 80% of

students will demonstrate a fundamental knowledge…

– Action verbs (handout)

• Outcome to exam questions (and instruction)– Can use the action verb

» Ex: Compute the standard deviation of….

– Test blueprint (handout)

Objective and Subjective Exam Questions

• Objective questions– Select a correct response from several alternatives

• Subjective questions– Present an original answer

• Strengths/weaknesses of each• Assumptions (True/ False/ or Undetermined)

1. Essay exams require more thorough student preparation and study time than objective exams.

2. Essay and objective exams can be used to measure the same content or ability.

• Examples of objective, critical thinking exam questions (packet)

Application Activity

• In groups or pairs and with using the Emerson content from Dawn’s previous presentation, 1. Construct one objective and one subjective exam

question.

2. Explain why you believe your questions are assessing critical thinking.

Application Activity

Objective examples:• (Analysis) Emerson might be characterized as any of the

following EXCEPT:(A) a transcendentalist(B) an American Renaissance writer(C) a public speaker(D) a political prophet(E) a literary critic

• (Evaluate) Determine whether the following statement is true or false:– Melville and Hawthorne were both disillusioned

transcendentalists.

Final thoughts

• At the program/course level, what do you want your students to learn/develop (outcomes)?

• Are your exams (methods) matching these objectives?• Use multiple assessment methods

– Ex: Lower stakes testing

• Can use rubrics for subjective exam questions

Agenda

1. Overview of SACS

2. Mapping Assessments to Program Outcomes

3. Writing Critical Thinking Exam Questions

4. Creating Rubrics

1.What is a rubric?

2.When should you use a rubric?

3.What are the elements of a rubric?

4.How do you create a rubric?

Creating Rubrics

What is a rubric?

-A scoring guideline that measures student achievement systematically:

– specifies instructor expectations– lists performance criteria– describes levels of quality

When should you use a rubric?

• On assignments that require subjective grading– Places structure on a subjective process

• Essays• Presentations• Multi-media

What are the elements of a rubric?

What the student needs to do

Descriptors and criteria that identify performanceexpectation for each point on the scale

I.

Example:

Include a compelling, descriptive, and persuasive conclusion that summarizes arguments set forth in paper

Standards that identify the range of quality or performance levels

How well the student does it

II.

Example:

--no or poor conclusion or summary of argument

--some summary of points made, but nothing beyond

summary; no broad conclusions/lessons

--a conclusion going beyond summary of what was written in

the body of the essay

A scale of points awarded to specific elements of student performance

III.What score the student will receive

Example:

5 = outstanding

3 = acceptable

1 = poor

Critical Thinking Rubric

A. Clearly and precisely analyzes key information, questions, and problems

B. Uses inference to reason carefully from clearly stated premises to important implications and consequences

C. Uses deductive and inductive reasoning and problem-solving skills consistently and with ease

A. Is unable to analyze information, questions, and problems or does so superficially

B. Is unable to or infrequently uses inference to reason from clearly stated premises or recognize implications and consequences

C. Is unable to or infrequently uses deductive and inductive reasoning and problem-solving skills

1 [2 3 4] 5

•Analytic Rubrics

•Holistic Rubrics

How do you create a rubric?

• Develop assignment

• Identify learning objectives

• Decide which criteria to evaluate

• Decide what kind of rubric to use

• Define performance indicators

• Identify standards of performance

• Decide on grading or scoring scale

• Review rubric to ensure it measures what you intend

Determine key criteriaIf the assignment objectives are …

… then consider these criteria

Creative products •“Wow” factor, novelty•Technical quality •Adherence to conventions of the field

Writing •Grammar/mechanics•Development of ideas•Clarity

Oral presentation •Voice projection •Body language•Organization

Identify Indicators

Concrete sign or evidence of a criterion being met—not meant to be exhaustive

Example:Learning objective: Students will be able to write in

an engaging and persuasive mannerCriterion: Student clearly and precisely analyzes

key information, questions, and problems Indicator:

Identifies main tension in story and relates it to sub-plots

Any Questions?