1

Holistic Scoring Using ACPS Measurement Topic Rubrics

Curriculum Implementation Modules(2011-2012)

Session Four: December 2011

Alexandria City Public Schools

2

Essential Questions

1. How can ACPS educators collaboratively score student transfer task products using ACPS measurement topic rubrics?

2. How can we get students to become involved in self-scoring and peer scoring using these rubrics?

3

Session ObjectivesSession Objectives

By the end of this workshop, you By the end of this workshop, you should be able to:should be able to:

1. Build inter-rater reliability when collaboratively scoring student transfer task products using ACPS measurement topic rubrics.

2. Promote student self-reflection and self-regulation using rubrics as part of formative and summative assessment.

4

Our Agenda at a GlanceOur Agenda at a Glance

Warm-Up Activity: Participants Share Their Students’ Performance on Transfer Tasks and Related Balanced Assessment Strategies

Modeling & Discussion: An Introduction to the Concepts of Measurement Topics, ACPS Measurement Topic Rubrics, and Building Inter-Rater Reliability During the Scoring Process

Participant Analysis of an ACPS Curriculum Guide Transfer Task: How Can You Use Measurement Topic Rubrics to Evaluate Student Achievement on This Task?

A Holistic Scoring Committee Simulation: Participants Form Small Scoring Teams and Build Consensus About the Range of “4, 3, 2, and 1” Score Points in a Sample Set of Student Writings

Planning Opportunity: Developing an Action Plan for an In-House Scoring Committee

5

Participant Debriefing

ThinkThink:: How did your students do in responding to transfer tasks during the past months? (What were your success stories? What challenges did you face?)

PairPair:: Compare your experiences with a partner.

ShareShare:: Pairs share a “headline” (25 words or less) of their experiences.

6

The Big Ideas of Balanced Assessment

Part I: The Importance of Balanced Assessment: diagnostic, formative, and summative forms of assessment; rapid/on-the-spot feedback; standards driven; metacognitive/student self-regulation; differentiation

Part II: Diagnostic/Pre-Assessment and Formative Assessment: providing on-the-spot, criterion-based feedback to students on a daily basis; aligning assessment with teaching-learning; focusing on clear learning targets; criterion-based feedback; “closing the gap”

Part III (Module Four): Lesson Planning Using Balanced Assessment to Prepare Students for Unit Transfer Tasks: using assessments that promote understanding; emphasizing oral and written language; using a range of question types and strategies; combining constructed-response test items with reflective assessments, academic prompts, and culminating projects; scoring transfer tasks

7

Part Three

Preparing Students for Transfer Tasks and

Scoring Transfer Task Work Products

8

The Balanced, Photo-Album Approach to Assessment

(pp. 5-6)

1. Complete the questionnaire on pages 5-6pages 5-6, “Creating a Photo Album, Not a Snapshot, of Assessment Results.”

2. To what extent do you individually address each of the 10 identified components of a balanced, photo- album approach to assessment?

9

Based upon Your Self-Based upon Your Self-Assessment, Consider Assessment, Consider (Part (Part

I)…I)…

Do you select the appropriate assessment tool or process to assess each desired result?

Do you use a range of assessment tools, rather than just tests and quizzes?

Do you strive for a photo album, not a snapshot, of student performance data?

Does your photo album provide a full portrait of what your students know, do, and understand relative to your desired results?

10

Based upon Your Self-Based upon Your Self-Assessment, Consider Assessment, Consider (Part (Part

II)…II)…

Do you make use of… Tests and quizzes that include

constructed-response items? Reflective assessments (reflective

journals, think logs, peer response groups, interviews)?

Academic prompts with a FAT-P (audience, format, topic, purpose) clearly stated?

Culminating performance assessment tasks and projects?

11

Constructed-Response Test Items (Page 7)

Require some form of performance by the student within the testing situation.

Involve students in demonstrations of understanding, not just knowledge-recall learning.

Are often written, but can be differentiated to allow for alternative approaches (e.g., for younger students).

Can involve some form of choice by the learner.

12

Sample Constructed-Response Test Items (Page

7)

1. Draw a picture to explain how a bill becomes law.2. Examine the following political cartoons. Write a

one-paragraph analysis of how they reflect conflicting perspectives on the Vietnam War.

3. Assume that you are one of the surviving characters at the conclusion of Romeo and Juliet. Write a letter to a relative explaining the major causes leading to the demise of the two protagonists.

4. Observe the following videotape, which highlights elements of a local eco-system. Describe your observations and conclusions about the health of that system.

5. Examine other examples on page 7.

13

Activity

Write a sample test or quiz item that requires a “constructed” (rather than “selected”) response from students about some aspect of the knowledge, skills, and/or understandings students will need for success on the transfer task you selected.

14

Formal and Informal Reflective Assessments (pp. 8-

10) Encourage students to internalize and

apply to themselves and peers significant evaluation standards and criteria.

Engage students in self-evaluation and meta-cognitive processing.

Ensure that all learners are becoming self-monitoring and are “owning” the evaluation criteria.

Encourage active feedback and adjustment.

15

Sample Reflective Assessment Activities (pp.

8-10)

1. Reflective Journal Entries: How well do you understand this passage? What are the main ideas from this lesson? What did this material mean to you?

2. Think Logs: How would you describe the process of classification? How has your approach to problem-solving changed during this unit?

3. Self-Evaluations: Based upon our evaluation criteria, what grade would you give yourself? Why?

4. Peer Response Group Activities: What can you praise about the work? What questions can you pose? What suggestions can you make for polishing the product?

5. Interviews: Tell me about your perceptions of this project. What do you consider to be your strengths and areas in need of improvement?

16

Activity

Think about the transfer task for which you are planning. Create a reflective journal entry and a think log entry for your students related to this task.

17

The Academic Prompt(pp. 11-12)

A structured performance task that elicits the student’s creation of a controlled performance or product.

These performances and products should align with criteria expressed in a scoring guide or rubric.

Successful prompts articulate a format, audience, topic/content focus, and purpose.

18

A Sample Academic A Sample Academic Prompt with a Prompt with a FFAAT-T-PP

Think about a time when you were surprised (topic).(topic). Write a letter (format)(format) to a friend (audience)(audience) in which you describe that experience. Use a logical narrative sequence with concrete sensory details to help your friend understand what this event was like and how you experienced it (purpose).(purpose).

19

Activity Think about an important piece of declarative or procedural knowledge required by the task for which you are planning. Create a sample academic prompt that embodies each of the FAT-P elements (format, audience, topic, purpose) aligned with that targeted knowledge.

20

Elements of an Effective Performance Elements of an Effective Performance Task and Culminating ProjectTask and Culminating Project

GG=real-world goals=real-world goals RR=real-world role(s)=real-world role(s) AA=real-world audience=real-world audience SS=real-world situation=real-world situation PP=real-world products and =real-world products and

performancesperformances SS=standards for acceptable =standards for acceptable

performanceperformance

21

A Sample A Sample GG..RR..AA..SS..P..SS. . (Elementary)(Elementary)

You and your classmates have been asked to become an historian! Each of you will research the research the historyhistory of a major person or event in the history of Alexandria. Then, you will write and illustrate a report that tells what you have learned, including why the person or event is important, how they influenced our city, and why we should all know about your subject. When you’ve finished, we’ll publish our reports in a class anthologyclass anthology (which we’ll donate to the Alexandria Historical SocietyAlexandria Historical Society). Your reports and illustrations should be accurate, complete, clearly written, and interesting for your complete, clearly written, and interesting for your readers.readers.

22

A Sample A Sample GG.R..R.AA..SS..PP..SS. . (Secondary)(Secondary)

You are a member of a team of scientistsmember of a team of scientists investigating deforestation of the Amazon rain investigating deforestation of the Amazon rain forest.forest. You are responsible for gathering scientific gathering scientific datadata (including such visual evidence as photographs) and producingproducing a scientific reporta scientific report in which you summarize current conditions, possible future trends, and their implications for both the Amazon itself and its broader influence on our planet. Your report, which you will present present to a United Nations sub-United Nations sub-committeecommittee, should include an action plan that is detailed and clear with well supported recom-detailed and clear with well supported recom-mendationsmendations for addressing the issues you raise.

23

Distinguishing Between an Academic Prompt and a Culminating Performance

Task and Project

In designing performance tasks, we need to ask ourselves: What is the level of independent transfer students are expected to demonstrate?

If students are still in the area of “guided practice,” an academic prompt may be more appropriate; if students are expected to demonstrate independent transfer and a high level of conceptual understanding, a culminating project or authentic performance task (cornerstone performance) may be most appropriate.

Review the performance task samples on pages 11-15 of your handout. What are the major differences between these academic prompts and projects?

24

A Reflective Question for Your Consideration…

Is the transfer task for which you are planning closer in format to an academic prompt or a G.R.A.S.P.S. culminating project? Why?

25

Assessing Performance Tasks

Modified Holistic Scoring Rubrics

Analytic-Trait RubricsAnalytic Scoring Guides

26

Evaluating Academic Prompts and G.R.A.S.P.S. Culminating

Projects

Each of the transfer tasks in the ACPS curriculum guides has two or more accompanying scoring rubrics.

Examine one or more of the rubrics in the guides—and use the following slides (and the materials on pages 16-18) to identify the type of rubric included in your curriculum guide(s).

Also, consider when you might choose to use a modified holistic scoring rubric, an analytical rubric, and/or an analytic scoring guide.

27

Sample Modified Holistic

Scoring Rubric (P. 16)3=All data are accurately represented on the graph. All

parts of the graph are correctly labeled. The graph contains a title that clearly tells what the data show. The graph is very neat and easy to read.

2=Data are accurately represented on the graph or the graph contains only minor errors. All parts of the graph are correctly labeled or the graph contains minor inaccuracies. The graph contains a title that generally tells what the data show. The graph is generally neat and readable.

1=The data are inaccurately represented, contain major errors or are missing. Only some parts of the graph are correctly labeled, or labels are missing. The title does not reflect what the data show, or the title is missing. The graph is sloppy and difficult to read.

28

The Analytic-Trait Rubric (P. 17)

Traits Understanding Performance or Performance Quality

Scale Weights: 65 percent

35 percent

4 Shows a sophisticated understanding of relevant ideas and processes…

The performance or product is highly effective…

3 Shows a solid understanding of the relevant ideas and processes…

The performance or product is effective…

2 Shows a somewhat naïve or limited understanding of relevant ideas or processes…

The performance or product is somewhat effective…

1 Shows little apparent understanding of the relevant ideas and processes…

The performance or product is ineffective.

29

Analytic Scoring Guide (P. 18)

50%=Content: Clearly-presented thesis statement with fully-developed supporting ideas and balanced evidence to make a compelling and convincing argument.

25%=Organization: Consistent support of thesis statement with all ideas and supporting evidence aligned with the controlling ideas of the composition. Consistent attention to the use of transitional expressions and other techniques to ensure coherence and clarity.

25%=Editing: Elimination of major grammar and usage errors with clear attention to correct syntax and sentence variety.

30

A Few Key Points About ACPS Measurement Topic

Rubrics Every transfer task has one or more

measurement topic rubrics.

These rubrics are holistic in design and reflect one of the K-12 measurement topics associated with the content area being taught, learned, and assessed.

Teachers are encouraged to work with students to identify task-specific “bullet” descriptors for the “4” and “3” score points.

Ideally, students will use these rubrics and task-specific performance descriptors to guide their work as they complete the transfer task.

31

Forming a Holistic Scoring Team

Ideally, teachers responsible for the same grade level and unit transfer tasks will establish consensus about a set of “anchor” papers before scoring their own student work products.

The group should assemble (e.g., during a common team or grade level planning time) and review a set of student work products for the unit transfer task (e.g., 10 or more samples).

The group should first discuss the rubrics for the task and what each team member agrees they will look for while scoring.

Then, team members individually score the sample papers.

If a discrepancy exists (i.e., a particular paper elicits a two-point or higher spread of scores), the team must come to resolution/consensus about the score for that paper.

32

A Scoring Team Simulation

Form table teams of three-four members.

Review the assigned (or self-assigned) scoring rubrics.

Individually score each paper.

Determine if any papers require resolution/consensus building.

If time permits, reflect on “outlier” papers, i.e., those that may be unusual or potentially troublesome for scoring.

33

Designing Assessments for Your Lesson Plan

Consider the types of assessments and scoring tools we’ve addressed in this module.

How can you use one or more of them as part of your lesson design? How will these assessment tools help your students achieve success on your unit’s transfer task?

Before you begin to work on your individual lesson, do you have any questions or ideas you would like to share?

34

Preview of Coming Attractions: Module Five

Our next module will deal with the issue of language acquisition, i.e., strategies for helping all learners (esp. ELL students) engage in effective reading, writing, speaking, and listening within your content area.

For next time, come ready to share your experiences with scoring unit transfer tasks.

Also, be thinking about how you have involved your students in self-monitoring and self-assessment.