Real Reading Achievement AND (not or) Better Test Scores

1Assessments to support deeper learning and ambitious instructionP. David PearsonUC BerkeleySlides will be posted at www.scienceandlitercy.org1OROn-demand performance assessment: The good, the bad, the ugly, and the incredibly beautiful!23Assessing Literacy: Problems and Possibilities??? How did we get to where we are?Where are we headed?Where should and could we be headed?P. David PearsonUC Berkeleywww.scienceandliteracy.org3Lest we fail to heed the lessons of history4Some Context for Todays TalkMy interest stems from several sourcesMultiple attempts to build state assessment systemsSeveral years of attempts to build alternative assessmentsportfolios, performance assessments, formative systems.Attempts to work with schools to build assessment systems rather than collections of testsA conviction that accountability should not drive us into a corner in which we engage in practices that are not in the best interests of students, teachers, or the public.45Part IHow did we get to where we are?Where did it all start?Early 1960s: A kinder, gentler timeTitle 1 reauthorization in 1967: the first trade of accountability for flexibility56The 1970sBehavioral objectivesCriterion referenced assessmentsCurriculum-embedded assessmentsMinimal competency tests: New JerseyStatewide assessments: Michigan & Minnesota

67Skill 1Skill 2TeachAssessConcludeTeachAssessConcludeThe 1970s Skills management mentality: Teach a skill, assess it for mastery, reteach it if necessary, and then go onto the next skill.Historical relationships between instruction and assessmentFoundation: Benjamin Blooms ideas of mastery learning 78Skill 1Skill 2Skill 3TeachAssessConcludeTeachAssessConcludeTeachAssessConcludeSkill 4Skill 5Skill 6TeachAssessConcludeTeachAssessConcludeTeachAssessConcludeThe 1970s, cont.And we taught each of these skills until we had covered the entire curriculum for a grade level.89Rule of ThumbThe finer the grain size at which we monitor a process like reading and writing, the greater the likelihood that we will end up teaching and testing bits and pieces rather than global processes like comprehension and composition.910The 1980sIn like a lamb, out like a lionQuiet consolidation of skills based learning (especially in commercial materials)A Nation at Risk leads to call for school reformThe evolution of higher order thinking in all subject matter areasLate 1980s: Do we have assessments to match our models of learning----------Performance and authentic assessment movementsEradicate the distinction between instruction and assessment10Key movements of the late 1980s/1990sKentucky and Vermont portfolio assessmentsGraduation by portfolio presentationCentral Park EastWaldenAssessment by exhibition: Coalition Performance Assessment: New Standards, CLAS, MarylandSome models of teacher performance assessment: NBPTS, Connecticut1112Late 1980s/early 1990s:PortfoliosPerformance AssessmentsMake Assessment Look Like InstructionOn standards 1-nActivities ConclusionsFrom which we drawWe engage in instructional activities, from which we collect evidence which permits us to draw conclusions about student growth or accomplishment on several dimensions (standards) of interest.1213The complexity of performance assessment practices: one to manyAny given activity may offer evidence for many standards, e.g, responding to a story.

Activity XStandard 5Standard 3Standard 4Standard 2Standard 11314Standard XActivity 1Activity 2Activity 3Activity 4Activity 5For any given standard, there are many activities from which we could gather relevant evidence about growth and accomplishment, e.g., reads fluentlyThe complexity of performance assessment practices: many to one1415The complexity of portfolio assessment practices, many to manyActivity 1Activity 2Activity 3Activity 4Activity 5Standard 1Standard 2Standard 3Standard 4Standard 5Any given artifact/activity can provide evidence for many standards Any given standard can be indexed by many different artifacts/activities1516Some people can tell what time it is by looking at the sun, but I never have been able to make out the numbers. There are four seasons: salt, pepper, mustard, and catsup.The perils of performance assessment: or maybe those multiple-choice assessments arent so bad after all.1617"Water is composed of two gins, Oxygin and Hydrogin. Oxygin is pure gin. Hydrogin is gin and water."The perils of performance assessment1718"Germinate: To become a naturalized German." "Vacumm: A large, empty space where the pope lives."

The perils of performance assessment1819Genetics explains why you look like your father, and if you don't, why you should.

The perils of performance assessment1920You can listen to thunder and tell how close you came to getting hit. If you don't hear it, you got hit, so never mind."When you breath, you inspire. When you do not breath, you expire."

The perils of performance assessment20211990sStandards based reformState initiativesIASA modelTrading flexibility for accountabilityMove from being accountable for the means and leaving the ends up for grabs (doctor or lawyer model) TOBeing accountable for the ends and leaving the means up for grabs (carpenter or product model)2122Standards-Based ReformThe Initial Theory of ActionStandardsAssessmentAccountabilityClearExpectations

MotivationHigherStudentLearningAla Tucker and Resnick in the early 1990s22We began our work with the same set of assumptions about standards-based reform that undergirded the IASA of 1994. The theory of action, to use our chair Dick Elmores favorite term, was that if you put in place a standards based accountability system (comprised of standards and assessments and the accountability requirement), that will be sufficient to drive the reform engine. The standards determine the content, the assessments make the expectations clear to all, and the accountability system provides the motivation to improve. The final ingredient, which is a critical assumption in this classic standards based reform mode,l is flexibility; that is, in return for being accountable, schools and teachers will be granted wide rein in the processes, strategies, and methods they use to improve student learning. But the studies we reviewed and the experiences of our committee members suggested that this model does not necessarily achieve the goal of higher student learning. Too often, for example, a probationary or reconstituted school threatened with takeover or severe penalties will focus on improving scores rather than changing instruction.We also found evidence that assumptions in this model did not correspond to reality; namely the assumption that teachers would develop improved practices if they had both the freedom and the motivation to do so. Changes in practice, we found, seldom occurred without intentional and arduous effort on behalf of school leaders.23More Recent Expanded Theory of ActionStandardsAssessmentAccountabilityClearExps

MotivationHigherStudentLearningInstruction

Professional DevelopmentAla Elmore and Resnick in the late 1990s.23So we expanded our theory of action to match what the research we reviewed and the experiences we shared told us.

In our expanded theory of action, two key elements are inserted between the clear expectations provided by assessments and the motivation provided by accountability on the one side and student learning on the other. And those two elements are instruction and professional development. The implication here is that standards, assessment, and accountability are not enough, that standards have to be explicitly and deliberately transformed into instructional practices and that professional development is the pathway to improved instruction. Only then, our work told us, would student learning improve in the way the theory predicts it should24No Child Left BehindA definite retreat from performance-based assessment as a wide-scale toolThe persistence of standards-based reform...NCLB and Reading FirstEvery grade level outcomes assessmentAssessments for placement, progress monitoring, diagnosis and outcomesMania for alignment2425Alignment: Part of the theory of actionSpecify the content standardsAlign everything else to them:AssessmentsInstructionCurriculum materialsProfessional developmentStandards for teacher learning25First the matter of aligning assessments to standards, which may well be the most common form of alignment we see in schools today. First, let me give you some data from the state level. As you are all no doubt aware, the Improving Americas Schools Act of 1994 mandated that all states develop and submit high and rigorous standards for ALL students by this academic year, with assessments designed to measure mastery of those standards due in 2000-2001. The logic was that of trading accountability for flexibility: We will hold ourselves and our students accountable to a set of standards, if you, meaning the federal or state government, will stay out of our face with respect to how we manage to get our students.

26Alignment: The centrality of content standardsContent StandardsPerformance StandardsFor StudentsStudent AssessmentsWild CardProfessional DevelopmentStandards for Teacher EducationAccreditationStandards for Licensure Teacher Evaluation26As this document suggests, just about everything in the latest model of Standards Based Reform is driven by the fundamental content standards in a curriculum area.The primary criteria for new standards for teacher education and teacher licensure are standards for students. As a part of the more general strategy of aligning everything under the sun with curriculum standards, teacher standards are being so aligned. And it was because of the emerging importance of standards for teachers in the public forums that I asked Cathy when we met at a conference in February, if if I could change the topic to the knowledge base for teaching reading... on the simple grounds that that knowledge base, whether it came from science, art, or religion, was something we must address right now. I hoped, then, to begin a conversation, albeit a short one, about what that knowledge base might look like.

It is my intention to spend most of my time this morning talking about how we might develop that knowledge base and standards and assessments to go along with it. By the way, did you notice that I left in a wild card option, just in case something arise, something we have not yet thought of, to align with content standards. BUT, before I do, there are a few contextual issues that I must set straight!27Problems in aligning assessments to standardsThe foxes are guarding the hen house problem (the test companies do the alignment)The backwards mapping problem (When the standards and the instruction look just like the test): Deal with DIBELS laterThe alignment is a coat of many colors problem (recall the earlier mapping)

27

28Skill 1Skill 2TeachAssessConcludeTeachAssessConcludeThe 1970s Skills management mentality: Teach a skill, assess it for mastery, reteach it if necessary, and then go onto the next skill.Remember the lessons of the 1970s!!!The bureaucratization problem2829Skill 1Skill 2Skill 3TeachAssessConcludeTeachAssessConcludeTeachAssessConcludeSkill 4Skill 5Skill 6TeachAssessConcludeTeachAssessConcludeTeachAssessConcludeThe 1970s, cont.And we taught each of these skills until we had covered the entire curriculum for a grade level.The bureaucratization problem2930Standard 1Standard 2TeachAssessConcludeTeachAssessConcludeWe could be back where we started.What could happen if we are not careful with standardsThe bureaucratization problem3031Standard 1Standard 2Standard 3TeachAssessConcludeTeachAssessConcludeTeachAssessConcludeStandard 4Standard 5Standard 6TeachAssessConcludeTeachAssessConcludeTeachAssessConcludeIn the name of ensuring a place for each standard in our assessment systems, we could end up in the same place we were in in the 1970sWhat could happen if we are not careful with standardsThe bureaucratization problem3132DIBELS DetourSchooling the cognitive process of test taking: Kids who are alike in terms of accuracy will differ substantially in terms of speed and fluencyFace validity problem: What virtue is there in doing things faster? naming letters, sounds, words, ideasWhat would you do differently if you knew that Susie was faster than Ted at X, Y, or Z???3233The Achilles Heel: Consequential ValidityGive DIBELSUse results to craft instructionGive DIBELS againGive Comprehension TestGive Comprehension TestThe emperor has no clothes3334Key Concept: HaladynaTest Score Pollution: a rise or fall in a score on a test without an accompanying rise or fall in the cognitive or affective outcome allegedly measured by the test3435Rule of thumb re-emergesThe finer the grain size at which we monitor a process like reading and writing, the greater the likelihood that we will end up teaching and testing bits and pieces rather than global processes like comprehension and composition.35Rule of Thumb for the FutureWherever we go, we must not go back to the world in which we link instruction to assessment at the very finest grain-sizeKeep our eyes on the prizewhat does accomplished performance look like?36Where are we headed in the Post NCLB era?Obama administration keeping up the accountability pressureTeacher accountability for student achievement$350,000,000 investment in new accountability assessments (PARCC, SBAC)CCSS movementAnother round of curriculum reformYou are a part of that effort37A New Opportunity: A Confluence of ForcesCCSS movements/Foundation fundingDisciplinary delivery of of literacy Reincarnation of Ambitious PedagogyDeeper LearningPARCC and SBACHybrid assessmentsIncrease weights given to open ended responses

38Why performance assessment?39Performance Tasks: Why bother?External validityCollege readyCareer readyPARCC readyCurricular validityHigher Order ThinkingPowerful learningDeeper learningPerformance Tasks: Why bother?Consequential validityWhat curricular activities will it lead teachers and students toward?Scaffolding toward independenceUtilityFormative AssessmentFeedback to the studentdifferentiationFeedback to you as the teacherreteaching

42Problems in aligning assessments to standardsThe assessment system problemOne assessment will not do the job. You need:Big outcomesMilestone assessments along the way (charting the trajectory of progress)Analyses of the skill infrastructureMatching tools to audience and purpose

42The last piece of the standards-assessment alignment piece is the creation of an assessment system. I am using that term in the sense in which I am sure Dick Elmore talked about last night, which I take to mean an array of measures all of which are, more or less loosely aligned to a standard or set of standards, but each serving a slightly different purpose, and for a slightly different audience. For example, we need big outcome measures, ultimate indices of whether students can meet the standard. For writing, it might well be a project in which students demonstrate that they can participate in all the steps in the writing process to produce a piece that is evaluated with a holistic rubric. For reading, it might be a performance task in which students construct a thematic interpretation across several texts, or it might be a score on a decent standardized reading test. We also will need what Deanna Birdyshaw referred to yesterday as milestone assessments, a way of taking stock at regular intervals to determine whether kids are on the right trajectory. In District 2 in NYC, they are the benchmark books that students read regularly to determine how students are progressing on the level of difficulty that they can read fluently plus some regularly administered writing prompts. Finally, we need some curriculum-embedded assessments that allow us to probe more deeply into the skill or knowledge infrastructures that students are building around a particular set of standards. I guess the point is that it is not enough to simply map assessments onto standards; we need to make sure that the assessments are also well-suited to purposes they serve and audiences who need information for making decisions.PARCC digression43

44

4546

My analysisStudents who have learned how to read and write in response to your tasksANDWhatever curriculum it is that supports progress that will lead to good performance on those assessmentsWill do well on these PARCC tasks47Why?They will have developed some transferrable practices that will serve them well in these new circumstances.Thats what we are aboutdeveloping transferable knowledge and skills48Closing AdviceThings you never really wanted to know or even ask aboutThree things:Think about assessment as a system of indicatorsReturn to our accountability rootsBe careful about how we link instruction and assessment4950

Matching tools with decisions and clientsFormative AssessmentsAssessment Systems50 In schools I am working with, one of the assessment exercises that I always start with is to complete a matrix in which we inventory the tools we have available for these different audiences and purposes.51AccountabilityAccountability follows responsibilityBased upon multiple indicators (both external and internal), e.g., External assessments (both state and standardized)Internal assessments (progress in our curriculum)Activity indicators (attendance, pass rates, library use)Client satisfactionAccountability should lead to assistance to build capacityAccountability should be reciprocal: Teachers take responsibility for learningGovernment for resources to support learningSociety for resources that lead to healthy lives51The last chapter addresses building a fair and rationale accountability system.

I mention only two of our principles in passing:Accountability follows responsibility: accountable all the way up: teachers, principals for supporting learning, districts and states for professional development

Building capacity.

Thank you for allowing me to lead you on a quick gallop through the guidebook. We hope you will take the opportunity to stroll through it more leisurely, and we hope above all, that you find it useful in building the kinds of school improvement systems that will promote the teaching, learning, and performance we all aspire to.52Aligning everything to the standards:A model worth rejectingStandardsAssessmentInstructionThis model is likely to shape the instruction too narrowly. Lead to test score pollution.5253A better way of thinking about how standards can link instruction and assessmentStandards our way of operationalizing our values: What we care about in teaching and learning literacy.Teaching andLearningActivitiesAssessmentActivitiesThey guide the development of both instruction and assessmentThis relationship can operate at the state or local levelThe logic of lots of good reform projects!5354My bottom lineWe desperately need instructionally sensitive assessments that have first rate psychometric characteristics so that we can build trustworthy internal systems for monitoring student progressNo decision of consequence about any individual, school, district or other aggregation should be based upon a single indicator of anything.Tests are a means to an end: Their value is measured by the degree to which they allow us to make good decisions and provide good instruction. They are not the ends themselves.High stakes and low challenge is the worst possible scenario5455The real bottom lineNever send a test out to do a curriculums job!The End!!!!Instead, send the test out to support the curriculum and those held accountable to it55ClientDecisions to be Made

Questions to be answered.Assessment Tools

studentsHow am I doing?

What shall I do next?[Portfolio entries (self-evaluated) and/or feedback from benchmark tasks.]

teachersHave the kids met my learning goals?

How did my teaching go?

How can I help Amy?

Should Amy enter X?->[Portfolio entries and Benchmark tasks]

->curriculum-embedded tests

->Informal diagnostic tools

->An array of converging evidence

parentsHow is my child doing?

Compared to the average bear?->Portfolios/work samples

->Normed test of some sort

administratorsHow effective is our program? How are our teachers doing?[Aggregated data of some sort (portfolios, norm-referenced tests)]

policy makersHow well are schools meeting public expectations?[Trends, over time, on some aggregated data (nrt would do just fine)]

taxpayersHow well is our money being spent?[Trends, over time, on some aggregated data (nrt would do just fine)]