a presentation by dr. dawn h. souter, project manager georgia department of education
DESCRIPTION
Lighting the Way The Georgia Formative Item Bank Sheds Light on Student Performance Expectations and Achievement. A Presentation by Dr. Dawn H. Souter, Project Manager Georgia Department of Education Assessment and Accountability. - PowerPoint PPT PresentationTRANSCRIPT
Lighting the WayThe Georgia Formative Item Bank Sheds
Light on Student Performance Expectations and Achievement
A Presentation by Dr. Dawn H. Souter, Project Manager
Georgia Department of EducationAssessment and Accountability
1
Major reviews of research on the effects of formative assessment indicate that it might be one of
the more powerful weapons in a teacher‘s arsenal.”
(Robert Marzano, 2006)
2
Purpose of Georgia’s Assessment Initiatives
• To provide assessment resources that reflect the rigor of Georgia’s state-mandated content standards
• To balance the use of formative and summative assessments in the classroom
• To promote student learning• To sustain implementation of Georgia’s rigorous
content standards
3
• Development of a three-prong toolkit to support teachers and leaders in promoting student learning– An expansive bank of formative instructional assessment
items/tasks based on CCGPS in ELA and Mathematics as a teacher resource - Phase I Release Fall 2012; Phase II release Fall 2013
– A set of benchmark assessments in ELA, mathematics – Initial Pilot Fall 2013
– An assessment literacy professional learning opportunity that focuses on implementation of research-based formative instructional practices (FIP) – Initial Pilot January/February 2013 with statewide launch 2013-2014
4
Inside the Formative Assessment Toolbox
Formative Assessment InitiativesBringing a Balanced Assessment Focus to the Classroom
5
Phase I items released into OAS fall 2012; Phase II items to be released in fall 2013
Pilot in winter 2013; Statewide launch in summer 2013
Phase I item pilot in fall 2013; Phase II pilot in winter 2014
Using Formative Assessment • Conducted during instruction (lesson, unit, etc.)• Identifies student strengths and weaknesses• Helps teacher determine next steps
– Review– Differentiation– Continuation
• Offers teachers immediate feedback to provide students with detailed feedback
• Assessment for the purpose of improving achievement• LOW stakes – no student performance reports generated
6
Formative Instructional Practices—Formative Assessment in Action
7
• Design• Teach
• Re-Design• Teach
• Re-Design• Teach
• Re-Design• Teach State-Mandated
Content Standards
Re-Design might involve changing activities, instructional techniques, assessment methods or content, and/or differentiation based upon student needs.
8
The Georgia Formative Item Bank
9
Purpose of the Formative Item Bank
The purpose of the Formative Item Bank is to provide items and tasks used to assess students’ knowledge while they are learning the state-mandated standards. The items will provide an opportunity for students to show what they know and show teachers what students do and do not understand.
The Georgia Formative Item Bank• Bank of over 1600+ classroom assessment items aligned with
the state’s content standards in ELA and Mathematics• Created for exclusive use in Georgia classrooms• Piloted with Georgia students• Reviewed by Georgia educators• Housed in the Georgia Online Assessment System (OAS)• Preponderance of items at DOK 3 and 4• Item, rubric and scored student sample papers provided• Available to ALL Georgia Teachers!
– 700+ items piloted in spring 2012; currently in OAS– 900+ new items piloted in spring 2013; available in OAS fall of 2013
10
Formative Item Bank UseThe Georgia Formative Item Bank can be used in order to:•Prepare students for the more rigorous expectations of the state-mandated standards.
– To show these expectations, students must engage with a variety of item formats beyond multiple-choice
•Provide students practice with open- and constructed-response items•Provide educators insight and access to evaluating open-ended response items
11
The Nature of Georgia’s Formative Item Bank
• Items created for Georgia educators as an instructional resource to be used formatively during instruction– Provide information about student performance
throughout the academic year to inform instruction and interventions
– Reteach, remediate, move forward, enrich– Low stakes; grading discouraged
• Support Georgia educators foster learning with an informative tool that keeps students, parents, administrators and educators themselves informed of students’ current position on the pathway to proficiency.
12
The Nature of Georgia’s Formative Item Bank (continued)
• Aligned with state-mandated content standards in English Language Arts (ELA) and Mathematics, grades 3 – HS
• Various formats, but primarily constructed - response, in order to measure the full expectations of what students need to know and be able to do to be on the trajectory of exiting high school college- and career-ready
13
Item Content• Georgia’s Content Standards
– Mathematics: Grades 3 – 8; high school Coordinate Algebra, Analytic Geometry and Advanced Algebra
– English/Language Arts (including Reading): Grades 3 – 8; high school 9th and 10th grade literature and American Literature
• Items aligned to multiple standards– One primary standard– One or more secondary standards
• Alignment verified by Georgia educators
14
Valuable Features of Formative Items – Mathematics
• Items include intentional focus on assessing processes used by students as well as the required content
• Items applied in real-world context• Writing requirements, such as explanations and
reasoning• Student responses on constructed-response
items/tasks− make student knowledge and skills transparent to teachers− illuminate student misconceptions
15
Item Formats• Multiple Choice• Primarily Constructed-response
– Extended Response– Scaffolded
• Constructed-response items require students to provide explanations/rationales, provide evidence, and/or to show work
• Preponderance of items at DOK 3 and 4• Provide teachers with evidence of true student
understanding of content and process
16
Multiple Choice Items
• Have four answer options• Distractors (incorrect answers) should be
believable and represent common conceptual and/or application errors
• Distractor rationales should assist teachers to identify specific student misconceptions to inform instruction
1717
Extended Response Items• May address multiple standards, multiple
domains, and/or multiple areas of the curriculum
• May allow for multiple correct responses and/or varying methods of arriving at a correct answer
• Scored through use of a rubric and associated student exemplars
18
Example of Extended Response Item Math—Advanced Algebra
19
Sreeja and Brandon solved the equation shown in different ways.
2 3 2 21
x x xx
Part A Before solving the equation, what solution could Sreeja and Brandon identify as extraneous? Explain your reasoning. Part B
Sreeja solved the equation by creating the proportion 2 3 2 2 .
1 1x x x
x
Demonstrate how Sreeja used the proportion to solve the equation. In each step, explain the properties she used to determine the solution. Part C Brandon solved the equation by simplifying the left side of the equation first. Demonstrate how Brandon simplified the expression on the left and then solved the equation. In each step, explain the properties he used to determine the solution.
Be sure to complete ALL parts of the task. Write your answer and show your work on the paper provided.
Do NOT type your answer in the text box below.
Advanced Algebra, Standards A.REI.2; A.REI.4; A.APR.6, A.REI.1; DOK 3
Scaffolded Items• Include a sequence of items or tasks• Designed to demonstrate deeper
understanding• May be multi-standard and multi-domain• May guide a student to mapping out a
response to a more extended task• Scored through use of a rubric and associated
student exemplars
20
Example of Scaffolded ItemMath—Analytic Geometry
21
In the figures shown, KLM is similar to ,PQR with measurements as shown.
Part A Write an equation to determine tan .L Solve for the tangent. Explain your answer. Part B Determine and .m L m Q Round your answers to the nearest tenth of a degree. Show your work and explain your answer. Part C Use the Pythagorean theorem and corresponding sides of similar triangles to determine the lengths of , , and .KL PR PQ Show your work. Part D Prove the measures of , , andKL PR PQ you found in Part C are correct by using the trigonometric ratios relating to and .L Q Show your work or explain your answer.
Be sure to complete ALL parts of the task. Write your answer and show your work on the paper provided.
Do NOT type your answer in the text box below.
Analystic Geometry Standards G.SRT.6; G.SRT.5; DOK 4
Rubrics
• Holistic• 5-point scale (0 – 4)
– 4: Thoroughly Demonstrated– 3: Clearly Demonstrated– 2: Basically Demonstrated– 1: Minimally Demonstrated– 0: Incorrect or Irrelevant
22
Rubric
23
Rubric Score Designation Description
4 Thoroughly Demonstrated
The student successfully completes all elements of the item by demonstrating knowledge and application of the relation between the trigonometric function of an angle and the side lengths of the triangle (G.SRT.6), the use of trigonometric ratios and the Pythagorean Theorem to solve right triangle problems (G.SRT.8), and the similarity criteria for triangles (G.SRT.5). All four parts are completed correctly with sufficient explanations or work shown.
3 Clearly Demonstrated
The student demonstrates clear understanding of the standards listed above by completing all four parts, but makes one or two minor calculation errors or omissions Or The student completes all four parts, but the explanation or work shown in one part is insufficient Or The student correctly completes three of the four parts with sufficient explanations or work shown.
2 Basically Demonstrated
The student demonstrates basic understanding of the standards listed above by completing two of the four parts with at least one sufficient explanation or work shown.
1 Minimally Demonstrated
The student demonstrates minimal understanding of the standards listed above by completing one of the four parts with or without sufficient explanation or work shown.
0 Incorrect or irrelevant
The response is incorrect or irrelevant to the skill or concept being measured.
Exemplar Papers• Prototype answer – the “ideal” response• Set of responses from actual Georgia students,
collected during item pilots• Samples scored by trained raters using rubric • Papers allow teachers to review and compare their
own students’ work to the sample responses for each score point– Helps standardize expectations of the standards
• Score point and annotations provided for each sample item response
Note: The pilot was conducted using standard administration procedures in order to ensure that results were comparable across the state. When items/tasks are used during instruction, these administration rules do not have to apply and student results may vary; thus, teachers may want to modify the rubrics and even raise expectations. Rubrics and exemplars should remain focused on high expectations.
24
Exemplar Paper--Excerpt
25
Exemplar Part A Step 2: ( 1) 0, so 1f x is a factor, not 1.x
Step 3: 121156126)1( 232 xxxxxx , not 121176 23 xxx .
Step 4: 12176)32)(43( 2 xxxx , not 126 2 xx . Part B
1x is a factor since ( 1) 0.f Student may use synthetic division or long division. Work should be shown.
2
3 2
6 12
1 6 7 11 12
x x
x x x x
26 12 (2 3)(3 4),x x x x so 3 26 7 11 12 (2 3)(3 4)( 1).x x x x x x
Set each factor equal to 0 to determine the zeros/ intercepts. 32
2 3 0 2 3x x x
43
3 4 0 3 4x x x
101 xx Part C
6 > 0, so as )(, xfx and as )(, xfx
Formative Item BankPilot I Item Examples and
Results Mathematics
26
Grade 6 Mathematics Item
27Item from FIB Phase I
Grade 6 MathematicsExemplar and Sample Student Response
28
Grade 6 Mathematics Sample Student Responses
29
Grade 6 Mathematics ItemPerformance Data
• Item Type: Extended Response• DOK: 3• Number of Students in Pilot: 87• Response Data
– 4’s: 0 for 0%– 3’s: 5 for 5.75%– 2’s: 34 for 39.08%– 1’s: 39 for 44.83%– 0’s: 9 for 10.34%
30
Grade 3 Mathematics Item
31
Grade 3 Mathematics Item Performance Data
• Item Type: Scaffolded • DOK: 3• Number of Students in Pilot: 44• Response Data
– 4’s: 5 for 11.36%– 3’s: 3 for 6.82%– 2’s: 18 for 40.19%– 1’s: 15 for 34.09%– 0’s: 3 for 6.82%
32
Overall Mathematics Pilot I (Spring 2012) Summary Data
33
Grade Number of students and percent falling into each score point Total student N/ %
0 1 2 3 4
3 771 667 373 81 36 1928
40.00% 34.60% 19.30% 4.20% 1.90% 100%4 795 800 360 87 58 2100
37.90% 38.10% 17.10% 4.10% 2.80% 100%5 548 513 252 124 44 1481
37.00% 34.60% 17.00% 8.40% 3.00% 100%6 927 768 269 65 14 2043
45.40% 37.60% 13.20% 3.20% 0.70% 100%7 896 632 243 62 11 1844
48.60% 34.30% 13.20% 3.40% 0.60% 100%8 984 791 314 100 51 2240
43.90% 35.30% 14.00% 4.50% 2.30% 100%9-10 798 697 186 45 27 1753
45.50% 39.80% 10.60% 2.60% 1.50% 100%11-12 690 602 178 63 9 1542
44.70% 39.00% 11.50% 4.10% 0.60% 100%
Overall Mathematics Pilot II (Spring 2013) Summary Data
Grade
Number and Percent of Students Achieving Each Score Point
Total Student N / %0 1 2 3 4
3 1378 1152 539 121 47 3237
42.57% 35.59% 16.65% 3.74% 1.45% 100%
4 1323 1264 325 83 25 3020
43.81% 41.85% 10.76% 2.75% 0.83% 100%
5 1351 1049 391 64 15 2870
47.07% 36.55% 13.62% 2.23% 0.52% 100%
6 1579 1171 370 135 53 3308
47.73% 35.40% 11.19% 4.08% 1.60% 100%
7 1602 856 219 72 36 2785
57.52% 30.74% 7.86% 2.59% 1.29% 100%
8 1529 1049 619 217 88 3502
43.66% 29.95% 17.68% 6.20% 2.51% 100%
9 - 12 2570 1435 299 59 23 4386
58.60% 32.72% 6.82% 1.35% 0.52% 100%
Key Findings from Phase I Pilot(Spring 2012)
• Overall performance shortfalls– Many students lacked organization and neatness– Don’t seem to understand what question was requiring – Don’t follow directions well – Didn’t answer all parts of questions– Don’t follow through with “units” in Mathematics
answers– Don’t have keyboarding skills – future of testing is online
and there are ELA standards (W6) for keyboarding/word processing in grades 4, 5, and 6.
35
Formative Item Bank Phase II Examples
Mathematics
36
Grade 6 MathematicsItem
37Item from FIB Phase II
Coordinate Algebra Mathematics Task
38
Advanced Algebra Mathematics Task
39
J ocelyn is given 2 polynomial functions. They are 3 2( ) 6 7 11 12f x x x x and 4 3( ) 2 2 6 8.g x x x x J ocelyn needs
to find zeros of the function ( ).f x The work J ocelyn did is shown in the four steps below.
Step 1: 3 2(1) 6(1) 7(1) 11(1) 12 0 10 0f
Step 2: 3 2( 1) 6( 1) 7( 1) 11( 1) 12 0 0 0, so 1f x is a factor
Step 3: 2( ) ( 1)(6 12)f x x x x Step 4: ( ) ( 1)(3 4)(2 3)f x x x x Part A Identify any steps in which J ocelyn made an error. Explain your answer. Part B
Find the zeros of the function 3 2( ) 6 7 11 12.f x x x x Show your work. Part C Sketch a graph of ( ).f x Include all intercepts. Show your work or explain your answer. Part D Find the difference when ( )f x is subtracted from ( ).g x
Be sure to complete ALL parts of the task. Write your answer and show your work on the paper provided.
Do NOT type your answer in the text box below.
Advanced Algebra Task
40
J ocelyn is given 2 polynomial functions. They are 3 2( ) 6 7 11 12f x x x x and 4 3( ) 2 2 6 8.g x x x x J ocelyn needs
to find zeros of the function ( ).f x The work J ocelyn did is shown in the four steps below.
Step 1: 3 2(1) 6(1) 7(1) 11(1) 12 0 10 0f
Step 2: 3 2( 1) 6( 1) 7( 1) 11( 1) 12 0 0 0, so 1f x is a factor
Step 3: 2( ) ( 1)(6 12)f x x x x Step 4: ( ) ( 1)(3 4)(2 3)f x x x x Part A Identify any steps in which J ocelyn made an error. Explain your answer. Part B
Find the zeros of the function 3 2( ) 6 7 11 12.f x x x x Show your work. Part C Sketch a graph of ( ).f x Include all intercepts. Show your work or explain your answer. Part D Find the difference when ( )f x is subtracted from ( ).g x
Be sure to complete ALL parts of the task. Write your answer and show your work on the paper provided.
Do NOT type your answer in the text box below.
FIB Items in OAS
• 700+ Phase I items already in Georgia Online Assessment System (OAS)
• 900+ Phase II items will be loaded into OAS in fall of 2013
• Same OAS log-in as used in the past– If you need an OAS log-in access code,
• Contact your School Administrator for OAS log-in access code
• School Administrators should contact their system test coordinator for assistance if needed
41
Searching in the OAS
43
You create test name and ID that are meaningful to you.
Naming Idea: “Formative” and Domain Name, such as literary comprehension
Searching in the OAS
44
Searching the OAS for Formative ItemsExample Search
45
The name you created in previous steps
All of the formative items are in Level 2 of the OAS which means all teachers have access.
Drop down to select subject and grade level
Drop down to select domain and standard (or select all)
1. Create Test (Test Name and Test Identifier, submit)2. Item Level (Select Level 2, which provides access to all Georgia teachers)3. Subject (Select Language Arts or Mathematics)4. Grade Level (Select grade 3 – 8)5. Domain ( Select Formative)6. Domain (Select Appropriate Domain or ALL, i.e. Information and Media Literacy)7. Standard (Select Standard or ALL)8. Show Items
46
Searching the OAS for Formative ItemsSearch Steps
Finding the Rubrics and Sample Student Papers
Once you create a test in the previous steps:•Assign the test to a class•Assign the test to a student in the class•Take the test as if you were a student•Click on “Score the Test”•At that point, you will be able to see the rubrics and student work samples(Screen shots are unavailable at this time due to annual repairs being made to the OAS.)
47
Formative Item Bank Information On-line
48
For more information about the Formative Item Bank Project
www.georgiaoas.org
Formative Item Bank Information On-line
49
http://www.gadoe.org/Curriculum-Instruction-and-Assessment/Assessment/Pages/OAS-Resources.aspx
Includes:•About the Formative Item Bank (document)•About the Formative Item Bank (presentation)•Student Checklist for ELA•Students Checklist for Mathematics•Link to the OAS•Link to Georgia Standards.org
50
Upcoming Benchmark Assessments
The purpose of the benchmark assessments is to provide a tool for teachers to use throughout the year in support of classroom instruction and frequent or continual evaluation of student proficiency, keeping students, parents, administrators and educators informed of students’ progress on the pathway to proficiency.
51
Purpose of the Benchmark Assessments
Interim Benchmark Assessment Item and Benchmark Development
• Benchmark Assessments will be created for:– ELA: Grades 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 and 11– Mathematics: Grades 1, 2, 3, 4, 5, 6, 7, 8, Coordinate
Algebra, Analytic Geometry and Advanced Algebra– Social Studies: U.S. History– Science: Biology
• Additional items that will be housed in an item bank will be created for:– Science: Grades 3-8– Social Studies: Grades 3-8
52
• ELA – Grades 1, 2, 3, 6, 7, 8,
and 10
• Mathematics – Grades 1, 2, 3, and
Coordinate Algebra
• U.S. History
• ELA – Grades 4, 5, 9, and 11
• Mathematics – Grades 4, 5, 6, 7, 8,
Analytic Geometry, and Advanced Algebra
• Biology
53
Benchmark Assessment Production Schedule
Phase I Fall 2013 pilot
Available Winter 2014
Phase IIWinter 2014 pilot
Available Fall 2014
Comparison to Formative Item Bank
Formative Item Bank• ELA and Mathematics• Grades 3 – High School• Item Bank only• Housed in OAS Level 2 (for
teacher-use)• Variety of item types
– Mostly constructed response
Benchmark Assessments• ELA, Mathematics, Science and
Social Studies• Grades 1 – High School• Item Bank and one benchmark
test form per grade/subject• Housed in OAS Level 3 (for
district-use)• Variety of item types
– Mostly multiple-choice and short-answer
54
On the Horizon for Georgia’s Formative Assessment Initiatives
• Phase II Formative Item Bank loaded into OAS—Fall 2013
• Phase I and Phase II Interim Benchmark Assessments loaded into OAS, projected Spring and Fall 2014
• Formative Instructional Practices (FIP) professional learning, will be available summer 2013 and implementation in school systems TBD by each school system
55
“Quality assessment is a system of assessing what students know and
are able to do in a manner that garners accurate information
from students for the purpose of improving learning.” (Rick
Stiggins, 2008)
56
Questions
57
Georgia Department of Education Assessment and Accountability
Dr. Melissa FincherAssociate SuperintendentAssessment and [email protected]
Dr. Melodee Davis Director Assessment Research and [email protected]
Michael HunekeAssessment Specialist, OAS [email protected]
Dr. Dawn SouterProject Manager, [email protected]
Dr. Jan ReyesAssessment Specialist, RT3Interim Benchmark [email protected]
Kelli Harris-Wright, Ed.S.Assessment Specialist, RT3Assessment Literacy Professional
58