author paulson, leon f.; paulson, pearl r. title assessing … · 2013-08-02 · portfolio has the...

DOCUMENT RESUME

ED 324 329 TM 015 516

AUTHOR Paulson, Leon F.; Paulson, Pearl R.TITLE How Do Portfolios heasure Up? A Cognitive Model for

Assessing Portfolios. Revised.

PUB DATE 15 Aug 90NOTE 32p.; Paper presented at the Annual Meeting of the

Northwest Evaluation Association (Union, WA, August.2-4, 1990).

PUB TYPE Reports - Evaluative/Feasibility (142) --Speeches/Conference Papers (150)

EDRS PRICE MF01/PCO2 Plus Postage.DESCRIPTORS *Academic Achievement; *Cognitive Processes;

*Educational Assessment; Elementary SecondaryEducation; Models; Outcomes of Education; *PortfolioskBackground Materials); Qualitative Research;Statistical Analysis; *Student Developed Materials;Student Evaluation

IDENTIFIERS Aggregation (Data); Stakeholder Evaluation

ABSTRACTIssues that must be addressed in designing procedures

for aggregating portfolio data are considered. These issues haveprofound implications for what is aggregated and how data arecollected, combined, and interpreted. Portfolio assessment occurs atthe intersection of instruction and assessment; it requires studentsto collect and reflect on examples of their work, providing both animportant instructional component to the curriculum and offering anopportunity for complex authentic assessments. A multidimensional,cognitive process model of assessment is recommended in this paper.The cogniti7e model for assessing portfolios is structured much likethe program evaluation model developed by R. Stake (1967). The modelis designed to be broadly descriptive, yet to provide a framework forpresenting high quantitative and qualitative data in a coherentfashion. Its results are judgments taken from the perspectives ofmultiple parties. The model does not provide a single score; rather,it provides a comprehensive view of complex learning outcomes in acontext where instruction and assessment are inseparable. Thethree-dimensions of the model are the following: (1) activity,including assessment activities intrinsic to both compiling aportfolio and aggregating across portfolios; (2) historical,including changes beginning from conditions at the outset throughtransactions to outcomes; and (3) stateholder, which reflectsviewpoints of stakeholders with an interest in the portfolios. Threesample portfolio projects are outlined: the Pupil Product Portfolio;a writing portfolio; and a metacognitive letter. Five figures and a65-item list of references are included. (TJH)

Reproductions supplied by EDRS are the best that can be madefrom the original document.

-7

U.S. EMPARTMENT O IIDUCATIOAOffice ot Educational Research and Improyomont

EDUCATIONAL RESOURCES INFORMATIONCENTER (ERIC)

tb.,. document has boon reproducod astocoivod from tho person or organizationoriginating it

0 Minor changes hay* boon made to improwreproduction oughty

o Points of yew or opinions stat al in this docu-ment do not nocessanly roprosont officialOERI position or policy

"PE.MISSION TO REPRODUCE THIS

MATERIAL HAS BEEN GRANTED BY

F. L. Aussoio

TO THE EDUCATIONAL RESOURCES

INFORMATION CENTER (ERIC)...

How Do Porte' c) s M riet 1C.31:)"?

A Cognitive Model for Assessing Portfolios

by

F. Leon Paulson

Pearl R. Paulson

Revised 8/15/90

This paper was presented at a conference Aggregating Portfolio Data sponsored by the NorthwestEvaluation Association in Union, WA, August 2-4, 1990.

F. Leon Paulson is an Assessment Specialist, Multnomah ESD, Portland OR.Pearl R. Paulson is a Student Services Coordinator, Beaverton (OR) School District.

2

"Paulson & Paulson

Contents

How do Portfolios Measure Up?

Introduction 1

Issues in assessing portfolios 1

The traditional approach to measuring achievement 2

A cognitive approach to defining achievement 3

A cognitive approach to assessing portfolios 5

A cognitive model for assessing portfolios 6

The activity dimensionThe historical dimension 8

The stakeholder dimension 10

The cognitive model: Implications f or aggregation 12

Assessment using judgment 13

Methodology: judgment, description, and (yes!) numbers, too 14

Practical applications of the model 16

Example portfolio projects 17

Example 1. The Pupil Product Portfolio 17

Example 2.A writing portfolio 19

Example 3. The metacognitive letter 21

Recap of the three examples 23

Conclusion 24

References 25

List of Figures

Figure 1. The cognitive model for assessing portf olios (CMAP) showing theactivity, historical, and stakeholder dimensions. 6

Figure 2. The activity dimension of CMAP. 8

Figure 3. The activity and historical plane of CMAP. 9

Figure 4. The activity and stakeholder plane of CMAP. 11

Figure 5. historical and stakeholder plane of CMAP. 11

3

Many millions of years ago a race of hyperintelligent pandimensional beings got so f ed up withthe constant bickering about the meaning of lif e that they decided to sit down and solve theirproblems once and for all. And to this end they built themselves a stupendous super computerwhich was so amazingly intelligent that even bef ore its entire data banks hed been connected upit started from "I think therefore I am" and got as far as ..educing the existence of rice puddingand income tax before anyone managed to turn it off.

[The builders confronted the computer, Deep Thought, with their ultimate question.] "... the task we havedesigned for you to perform is this. We want you to tell us the Answer...to Life...TheUniverse...Everything."

"Y es," said Deep Thought, "I can do it." [To be continued...)

-- Douglas Adams The hitchhiker'sguide to the galaxy

Parents, school boards, state departments, and federal agencies are calling for accountability ineducation. Usually, they call for test scores. Are they up? (Up is good!) Down? (Down is bad!)

Soon, the evaluation community will be called on to use portf olios for accountability. Theportfolio has the potential to be a powerful educational tool for encouraging students to take charge oftheir own learning (Swing, Stoiber & Peterson, 1988). Portfolio assessment requires students tocollect and reflect on examples of their work, providing both an important instructional component tothe curriculum and offering an opportunity for complex, authentic assessments.

Portfolio are an intersection of instruction and assessment -- they are neither instruction norassessment, they are both instruction and assessment. Portfolio assessment forces us to adoptalternative ways of thinking about what we do and how we measure. To answer questions ofaccountability, we must aggregate but without corrupting the portfolio's instructional benefits.

We begin our paper with several issues that must be addressed when designing an aggregationstrategy. Next, we contrast the unidimensional asse3ssment of achievement tests withmultidimensional assessment from a cognitive process perspective. We recommend that portfolioassessment requires a multidimensional, cognitive process model. We propose such a model, CMAP.We conclude with a review of methodologies appropriate to aggregating portf olio data and illustratethe model in use.

Issues in assessing portfolios

Our paper is not a recipe for aggregating portfolio data. Rather, it explores issues that must beaddressed when designing procedures for aggregating data while preserving the integrity of theportfolio itself. These issues have profound implications for what is aggregated and how data arecollected, combined, and interpreted.

4

Paulson & Paulson - How do Portfolios Measure Up?

o Aggregation. A dictionary defines an aggregation as "a total or whole: a group of distincttnings gathered together." If we view aggregation as ways to report about distinct things, wego about our task one way. If we view it as ways of representing commonalities, we go aboutour task in another. In aggregation, is diversity welcome, or a nuisance?

o Knowledge. Education is often thought of as a means throtigh which students acquireknowledge. But what do we really mean by the term knowledge? Is it something we have, or

is it something we do? Is it stmething educators give, or is it something students construct?This epistemological debate between structuralism and constructivism has profoundimplications f or how we go about aggregating portf olio data.

o Authenticity. We hear the call for authentic assessment from almost every corner. There isan assumption that authentic assessment techniques like portfolio assessment are going to tellsomething different from the assessment techniques already in place. What kinds ofinformation does authentic assessment offer not f ound in traditional techniques?

o Scope. Should portfolio assessment try to replace traditional assessment, or should it focuson complex performance not tapped by traditional methods? Is it all things to all people, or isit specialized?

o Standardized Contents. As evaluators, we routinely look f or ways to standardize theconditions under which we observe in order to reduce error of measurement and produce highreliability and validity coefficients. Portf olios celebrate diversity -- each portf olio is highlypersonal in nature. Standardizing portf olio contents changes the nature of the portfolio itself.Should we be willing to pay the price of modifying the portf olio as an instructional tool toobtain better statistics?

o Measurement Standards. While educators often use the terms reliability and validity in anarrow, quantitative sense, the Joint AERA, APA, NCME Committee f or the Review of theStandards for Educational and Psychological Tests (1985, see pages 9 and 19) defines thembroadly in relation to purpose and context. In portfolio assessment, should we be guided bystandards often applied to multiple choice tests, or should we rethink the ways we accumulateevidence that portf olio assessments are consistent (reliable) and accuratc (valid)?

o Observer Impact. One of the profound realizations of 20tn century science is that the act ofobserving changes the thing observed. In education, assessment has major impact on theeducational experience itself. Because what is assessed becomes valued, should we look atbest perf ormance, typical performance, or deficit perf ormance?

o Empowerment. A major instructional goal of portfolios is metacognitive, to help studentsdevelop their capacity f or self-reflection and making judgments. The portfolio encourages thestudent to develop a set of values, assess their work according to those values, celebrate whenthey meet the expectationc implied in those values, and develop new directions for themselves.How can we empower both aggregators and students so that the judgvents of one do notinvalidate the judgments of the other?

Page 2.

5

Paulson & Paulson How do Portfolios Measure Up?

The traditional approach to measuring achievement

Traditionally, educators think of measuring achievement by using norm or criterion referencedachievement tests. As a rule, achievement tests are built on the assumption of unidimensionality, thatis, what is measured occurs on a single, underlying dimension. Achievement is the knowledge basegrowing bigger fact by fact. This assumption has a profound impact on the tests we develop. We usemathematical models like classical test theory and item response theory to refine our tests and contv)1

how they w3rk. Achievement becomes defined by what our achievement tests measure. We producetests that conform to our models, then we use those same models to demonstrate that those tests arereliable and valid. But the nagging question remains -- reliable and valid measures of what? Has ourability to derive equations surged ahead of our understanding of learning andcognition? John Tukey(1972), the mathematician, has said,

It is well to understand what you can do before you lear.i to measure how well you seem to be able to do it (p.

52).

Our unidimensional theories of testing may be dictating defacto theories of learning that we useto drive instruction (see Shepard's, 1990 discussion). These theories assume that students first maaerbasic skills, then go on to higher ones. Higher order thinking skills, a designation frequently usedto identify worthy instructional goals, implies that thinking is just another skill to be taught. But thedimensional model simply does not fit the complexity of human behavior in context. Resnick and

Resnick (1989) observe:

One of the most important findings of recent research on thinking is that the kinds of mental processesassociated with thinking are not restricted to an advanced or "higher order* stage of mental development.Instead, thinking and reasoning are intimately involved in successfully learning even elementary levels ofreading, mathematics, and other school subjects. Cognitive research on children's learning of basic skillsreveals that reading, writing, and arithmetic -- the three Rs -- involve important components of iaference,judgment, and a,- Ave mental construction. The traditional view that the basics can be taught as routineskills, with thinking and reasoning to follow later, can no longer guide our educational practice. (p. 4)

The fact that the nature of thinking has been misinterpreted as the sum of its parts maj be the

reason that many of us have been frustrated in our attempts to develop paper and penc:( tests ofthinking. Shavelson, Carey & Webb (1990) put it this way:

...the response format of multiple-choice tests, despite its efficiency. actually prevents us from measuringsome of the things that we consider most Important. For example. educators want to know how well a

student formulates problems and develops answers, not simply whether the student selects the correct

alternative....

Moreover, our culture has become so accustomed to achievement testing that citizens do not even ask how

achievement test scores were derived -- but only who is on top and who is on the bottom. Achievement testscores fit the Amencan belief in a single dimension of ability that distinguishes winners from losers, (p 693)

If thinking does not lend itself to unidimensional measurement, then neither should the evaluation ofthinking be constrained by standards derived for unidimensional measurement.

Portfolio assessment occurs at the intersection of instruction and assessment. The relationship ismultiplicative: chanu one and you change the other, set either to zero and both are lost. Ifachievement test methodology imposes a unidimensional viewpoint on the interaction, then

instruction will also be compromised.

Page 3.

1;

Paulson & Paulson

A cognitive approach to defining achievement


One thing handy about a unidimensional learning theory is that there is a limit to the possibilities.Cognitive theory, on the other hand, can be bewildering. Yet there are important themes that recur,ones that we should take into account in the way ' e instruct and measure.

One theme is that there k more than one dimension to mental processes that must be considered.Howard Gardner (1983) says there is a combination of seven kinds of intelligence relevant todesigning educational programs. The combination includes visual-spatial, bodily-kinesthetic,musical-rhythmical, interp;tisonal, intraperronal, logical-mathematical, and verbal-linguistic.Gardner makes the case for the multidimensional nature of human ability by noting that

the linguistic" sensitivity to the sounds and constructi( ., of language is exemplified by the poet, whereasthe interpersonal ability to discern and respond to the moods and motivations of other people is representedin the therapist. Other occupations more clearly illustrate the need for a blend of intelligences. Forinstance, surgeons require both the acuity a spatial intelligence to guide the scalpel and the dexterity of thebodily-kinesthetic intelligence to handle it. (Gardner & Hatch, 1989, P. 5)

A second theme is that so called higher functioning develops parallel with, not out of, lower levelfunctioning. The passage from Resnick & Resnick (op.cit.) reflects this view. One attempt to makesense out of how humans think has come from the study of artificial intelligence (AI). AI researchersbelieve that if we understand how people think, we should be able to program a computer to mimic theprocess (sec McCordick, 1972). They report that the so called higher order thinking skills arcsurprisingly easy to program, yet the approach is really not very helpful in simulating intelligentbehavior (sec Rose, 1984; Schank, 1984). Minsky (1984) offers this comment:

It is interesting to note that some of the earliest fAl) computer programs excelled at what people consider tobe experr skills A 1956 program solved hard problems in ma:hematical logic, and a 1964 program solvedcollege-level problems in calculus. Yet not until the 1970s cotAd we construct robot programs that could seeand move well enough to arrange children's building blocks into simple towers and playhouses. Why couldwc program computers to do expert things before we could program to do childish things/ The answer mayseem paradoxical. much of 'expert" adult th;nking is actually simpler than what's involved when ordinarychildren play.

To be considered 'expert', one needs a large amount of knowledge of only a relatively few varieties Incontrast, an ordinary person's "lornmon sense" involves a much larger variety of different types ofknowledge -- and this requires more complicated management systems. (1986 p 72 )

These words should have profound impact on those performing educational evaluations. There isconsiderable difference between looking for straightforward, bottom-line changes on basic skillsmeasures and dealing with multiple intelligences and cognitive management strategies.

A third theme is that many kinds of processes combine to produce outcomes. In his classic 1963paper or criterion-referenced testing, Robert Glaser wrote,

Lnderlying the concept of ach_vement measurement is the notion of a continuum )f knowledge acquisitionranging from no proficiency at all to perfect pert ormance. (1971, p. 7, emphasis added)

Twenty-seven years later, he describes major aspects of competence as

...the compiled, automized, functional, and procedurahzed knowledge characteristic of a well-developedcognitive skill. the effective use of internalized self-regulatory control strategies For fosteringcomprehension. and thc structuring of knowledge for explanation and problem solving (Glaser, 1990, p 29f )

Page 4.

Paulson "IL Paelson How do Portfolios Measure Up?

Instead rf a continuum of knowledge acquisition, Glaser's more recent quote talks aboutmetacognition and knowledge-as-process.

Finally, some investigators have begun to examine the highly complex ways in which cognitiveprocesses are influenced by context. Consider the question of Aptitude-Treatment Interactions(ATI). ATI is assumed when testing specialists interpret the meaning of achievement test scores andtheir implications for classroom instruction. In 1957, Cronbach wrote that a complete science Jfpsychology is possible if researchers looked at interactions. Later, Cronbach and Snow (1977)analyzed hundreds of research and evaluation reports that examined interactions. While concludingthat ATIs are indeed very important, they are also very complex, often highly context specific, anddifficult to generalize. Cronbach observed that looking at interactions is like entering a "hall ofmirrors" that extends to infinity (1975, p. 119). Later (1988), he updated his simile to reflect thelanguage of the emerging science of chaos:

...t's like walking through a maze where walls rearrange themselves with every step you take. (1988, p. 47,quoting Gleick, 1987 p. 24.)

Then, putting it into a cognitive framework, he continued

...serious cognitive scientists can no longer mos.el the mind as a static structure. They recognize a hierarchyof scales, from neutron upward, providing an opportunity for th: interplay of microscale and macroscale socharacteristic of fluid turbulence and other complex dynamical processes. (1988, p. 47, quoting Gleick, 1987.p 299)

A cognitive approach to assessing portfolios

Through assessment we attempt to discover what is really being achieved -- the educational bottomline. By using test theory models, we attempt to improve on our ability to measure what is achieved.But, whenever we go after objectivity we make subjective choices to get there. We accumulateachievement test scores in the hope that they correspond to "objective" reality. However, to knowwhether cur observations correspond to what is real externally, Eisner (1990) tells us that we need twothings. First, we need to know what reality actually is, and second, we need to know what we observe.But all we actually know is what we observe; we never know reality directly. And what we observedepends on what we happen to be looking at and why. It is a product of the evaluator's ownperspective, and no two people have quite the same perspective. 0. J. Hardison (1989) explains thedilema like this:

The problem of the world seen from different perspectives is both fa mthar and profound To someonewearing dark glasses thc world looks dark, but it is not necessarily dark. Take off thc glasses and thc world isflooded in intolerable light. One kind of mathematics reveals one kind of pattern: another reveals anothcrThe brilliance of the world becomes yellow with yellow lenses, blue with blue lenses. (p.46)

What we see when we evaluate a portfolio is the product of the glasses we wear when we evaluateportfolios. If we wear glasses designed to reveal mathematical thinking as the sum of basic skills, thenthat may be all we see. And by using these glasses we may never notice those aspects of mathematicalthinking that led the wrench mathematician Henri Poincare to assert that mathematical thinking mayhave as much to do with aesthetics as formal logic (Pappert, 1980).

Page 5.

8

Paulson & Paulson

Donmeyer (1990) warns that our theories


...can easily become reality for those who employ them. When this occurs, other conceptions of reality arenot even considered; indeed, the possibility that alternative conceptions of reality exist is normally not evenrecognized. (p. 182)

A cognitive model for assessing portfolios

The Cognitive Model for Assessing Portfolios (CMAP) is structured much like the programevaluation model developed by Robert Stake (1967). CMAP's activity dimension uses Stake's wordsrationale, intents, contents, standards, and judgments and CMAP's historical dimension usesStake's antecedent, transaction, and outcome. Although we use many of the same words, we haveredefined the concepts consistcnt with portfolio rather than program assessment.

The model is designed to be broadly descriptive, yet provide a framework for presenting bothquantitative and qualitative data in a coherent fashion. It also provides results that are the judgmentsfrom the perspectives of multiple parties. The model does not necessarily produce convergence in asingle, bottom-line, snapshot kind of result. Rather, its purpose is to provide a comprehensive view ofcomplex learning outcomes in context. Stake (1975) says

More ambiguity rather than less may be needed in our reports. Ov;rsimplification obfuscates (p. 23)

00Ctt

0°- Antecedent

Transaction

ll Outcomes

ILI ZCI 0__I ......

ACTIVITY DIMENSION

Rationale Intents Contents Standards Judgments

Figure 1. The cognitive model for assessing portfolios showing the activity, historical, and stakeholderdimensions.

Page 6.

9


The model nas three dimensions, each reflecting a major aspect of the portf olio. The firstdimension is a :tivity. This dimension includes assessment activities intrinsic to both compiling aportfolio and aggregating across portf olios: rationale (what is the purpose for forming the portf olic),intents (what, in general terms, are the goals), contents (the materials found in the portf olio itself),standards (wht t is good and not so good perf ormance), and judgments (what conclusions we draw).The second dimension is historical. The, historical dimension is sensitive to changes over time:antecedents (conditions at the outset), transactions (what is occurring over the the time span coveredby the portf olio), and outcon.es (conditions at present). Finally, the third dimension, stakeholder,reflects viewpoints of the various groups with an interest in the portf olio program. The identity of thestakeholders may vary. The student is the central one but stakeholders may also include teachers,parcnts, district specialists, representatives of the community, the aggregator, and others.

Figure 1 represents all three dimensions of the model and provides a reference as we describe thevarious rows, columns, and planes within the model.

The activity dimension

The activity dimension relates to the operations involving putting together portfolios.

Rationale describes the philosophical basis and operational guidelines for collecting materialsand putting them into a portfolio. The rationale varies with the stakeholder in question. The student'srationale may be similar and different from the teacher's (or any other stakeholder's) rationale ininteresting and important ways.

Intents describe the areas to be represented by the portfolio. These may be the goals andobjectives that indicate the curricular scope of the portfolio (e.g., "math and science"); they may bestated operationally (e.g., "demonstrate improvement in..."); they may show what is valued (e.g.,"originality", "conceptualization", "accuracy"). Intents may reflect an interest ia unintended as well asintended effects; the value of some work is discovered only in retrospect (see Scriven, 1972, on goal-f ree evaluation). Intents should be stated in middle distance (Hardison, 1989) terms so that theyare easy to recognize and work with (Valencia. 1990). The statement of portf olio intent plays a keyrole in portfolio assessment by providing the basic organization and outline for indexing contents.

Contents are the exhibits, the actual things found in the portfolio'. Contents are primarilystudents' work. These may include classroom assignments (along with student comments) as well aswork students developed especially for their portfolios. The materials that form the portf olio contentstell us much about the level of conceptual understanding (cognitive fidelity) and the relevance of

the performance to context (process -elevance) (see Shaveison, Carey & Webb, 1990). A studentcan demonstrate -.1 underatanding of a concept in many ways and the cognitive model permits thestudent maximum freedom in choosing what they may put into their portfolios.

Standards tell how well students should perf orm; they are the performance criteria and may berelative or absolute. Relative criteria may compare a student's earlier and later work, or an individualswdent with other students in some appropriate reference group (as with standardized achievementtests). Absolute criteria reflect judgment about what is acceptable performance on an externallyestablished standard (e.g., judgment of Olympic style competition).

i In developing guidelines for assessing portfolios, the Northwest Evaluation Association (1990c) refers to componentsfound in a portfolio. CMAP refers to this collection of components as 'contents'.

Page 7.

1 0


Judgment refers to making statements about how well a program is working. Portf olios are allabout judgment. In developing a personal portfolio, students develop the capacity to judge the qualityof their work and reflect on growth. In aggregating, the cognitive model also calls for the aggregator'sjudgment. This is jn sharp contrast to assessments using achievement test scores and other "objective",quantitative procedures. Wolf (1975) points out a basic problem with evaluations based on test scores.

Great collections of numbers, such as those found in children's cumulative files and school or programevaluation studies, tend to blur and obscure rather than sharpen and illuminate the education process. Inseeking objectivity, the decision maker may exclude a factor that ought to be of fundamental concern: humanjudgment. (p. 185)

Eisner (1990) also sees no benefit in letting test scores replace judgment but points out some of thereasons that evaluators find doing so attractive.

...the creation of procedures that eliminate judgment are certainly possible. Hermetically sealed saran-wrapped achievement tests whose questions are to be answered by filling in blanks with graphite so that theycan be scored by machine untouched by human hands, provide ample testimony to the attractiveness of suchprocedures. Such tests are not only politically safer than exercising judgment, (Exercising judgment on highstake tests can be dangcrous.) they are also very efficient. Yet consensus achieved through proceduralobjectivity provides no purchase on reality. It merely demonstrates that people can agree: we hope that forgood reasons, but what constitutes good reasons...is itself a matter of consensus. That might be all we everhave, but we ought to recognize it for what is. (p. 81)

The activity dimension of the cognitive model of portfolio assessment is illustrated in Figure 2. Itindicates that in assessing portfolios, the student and aggregator divide their attention across liveaspects of a portf olio: rationale (why the portf olio is being assessed), intents (what learning, ingeneral terms, the portfolio will demonstrate), contents (the materials that find their way into theportf olio), standards (how to describe good and not so good performance) and judgments (using thestandards, what the contents tell with respect to the rationale and intents).

In Figure 2 the contents column is specially shaded as a reminder that contents refers to thetangible materials that make up the portf olio, and that the selection of the contents is tied, explicitly orimplicitly, to rationale, intents, standards and judgments.

RATIONALE INTENTS CONTENTS STANDARDS JUDGMENTS==

Figure 2. The act ivity dimension of the cognitive model for assessing portfolios.

Page 8.

1 1


The historical dimension

The historical cEmLnsion puts the portfolio into temporal perspective. This is the dimension thatlooks at last year's portfolio and this year's portfolio from the perspective of next year's class.

Antecedents refer to baseline performance, the context, and learm. naracteristics. Theantecedents define the starting points and set the stage for the judgments that will be made aboutstudent learning and program effectiveness.

Transactions are the countless encounters that occur around the portfolio itself. The portfoliobecomes an organized depository for materials that reflect a large number of transactions.Transactions are any instruction, experience or exposure that brings about change. Transactionsinclude encounters between the lc and the thing being learned, and between the learner and adifferent stakeholder's set of rationales, intents, contents, standards, or judgments. "Transactions"equals knowledge-as-a-verb, the "knowing", the "doing", the "constructing", the "understanding". In thecognitive model, transactions account for the differences bdtween antecedents and outcomes and areof great interest to an aggregator.

Outcomes, traditionally, have been the major interest of most educational assessmett. In Glaser's1963 quote (op.cit.), they are the everything. The proposed model places outcomes in balance withthe rest of the program. The aggregator looks at outcomes, not just for their own merit, but also tocompare them with antecedents and to identify or infer the transactions that took place in between.

Figure 3 relates the historical and activity dimensions. In Figure 3, the student work being"celebrated" in the portfolio is represented by the intersection of contents and outcomes. Earlier workthat is exhibited to demonstrate progress is represented by the intersection of contents andantecedents. Early drafts of a work, conference notes, or students' reflections on their own learningare represented by the intersection of contents and transactions. Figure 3 also shows that historicalchanges in contents are irrevocably tied to historical changes in rationale, intents, standards andjudgments.

A

0

RATIONALE INTENTS CONTENTS STANDARDS JUDGMENTS

NTECEDENTS

I I

RANSACTIONS

UTCOMES

Figure 3. The activity and historical plane of the cognitive model for assessing portfolios.

Page 9.

I `I


An aggregator following a traditional, unidimensional assessment model would take inventory ofthe outcomes exhibitcd in a portfolio's contents, or measure the changes between the antecedents andoutcomes. The cognitive model of portfolio assessment does indeed allow such aggregation.However, aggregators using the multidimensional cognitive model rn also able to iu:or metacognitionany time there is evidence that a student has related outrome contents to any other cell in the model.Students show evidence of mc.tacognition any time they contrast recent work with cialier work(outcomes with antecedents); include early versions of finished products oi reflect on their learningstrategies (transactions); or relate their outcomes to some rationale, intenk, or standards. Thecognitive model allows aggregators to use portfolios as a unique v.:ay to assess metacognition.

The stakeholder dimension

Finahy, we have the third dimension of the cognitive model. The stakeholder dimension identif iesthose individuals with an interest in the portfolio. The concept of the stakeholder in evaluation hasbeen fully developed by Guba and Lincoln (1989) in a concept they call fourth generationevaluation. In fourth generation evaluation, different stakeholders are provided enough iii:ormationthat they can reach consensus or at least negotiate a compromise. CMAP facilitates such consensusand compromise by having all stakeholders announce their rationale, intents, standards, and judgmentprocesses at the outset and by making timely reports of their transactions and outcomes'.

In the development of portf olios, stakeholders may include the student, the teacher, the family, thedistrict and the society in general. The aggregator is a stakeholder who may represent an additionalperspective or a perspective already represented. We limit the present discussion to only the three:the student, the teacher and the aggregator.

Student. A portf olio developed by a student will probably integrate the rationale, intents,standards, and judgments of the teacher. However, the student as a stakeholder will have personalreasons for devel )ping a portlio. Students have personal goals and objectives and set standards thatpertain to them. Variation will exist across students. For examplt., it would be expected that a studentpreparing to enter college to study science world have rationales, intents, contents, standards, andjudgments different from a student preparing to enter the personal computer sales force or oneplanring a career in law enforcement.

Teacher. The portf olio project planned by the teacher would probably ink:grate the rationale,intents, standards, and judgment processes established by the district in its corricuium, modified by toreflect each teacher's personal philosophy and the perceptions of this year's cl:Iss.

Aggregator. The aggregator beings an outside viewpoint to :he portfolio. The rationales, intents,contents, standards, and judgments will be similar to those of th o. agency that contracted theaggregation and also be determined by the intended use of the aggregator's report.

Portf olio assessment starts at the student level with meti4zognition, the student making judgmentsabout the contents that form the portfolio. Portf olio assessment continues at the teacher level withteachers reviewing and reflecting upon the portf olio contents and making judgments. The process isconsistent at the two levels and mutually supportive. Each successive stakehoider enters the picture toreviev., refiect on, and make judgments. Portfolio assessment thus becomes a consistent, supportiveprocess at all levels of aggregation. A worthy object of review is the degree of compatibility or

he NWLA (1990c) guidehnes for portfolio assessment introduce the conccpt of a composite portf olio A teacher couldput together a por tfoho that represents an individual class l'his composite would contain all etc mcnts of an individualstudent portfolio including teacher's self-reflection. A -ernposite portfolio might be assembled at any stakeholder level

Page 10.

1 3


incompatibility between stakeholders with respect to rationales, intents, and standards. Thisaggregation plane is represented by Figure 4.

RATIONALE INTENTS CONTENTS STANDARDS JUDGMENTS:,..1t.

STUDENT

TEACHER

AGGREGATOR

Figure 4. The activity and stakeholder plane of the cognitive model for assessing portfolios.

ANTECEDENT TRANSACTIONS OUTCOMES

STUDENT

TEACHER

AGGREGATOR

Figure 5. The historical and stakeholder plane of the cognitive model of portfolio assessment.

The second area of interest is the impact of stakeholders on each other. See Figure 5. Will thestudents', teachers', and aggregators' knowledge of each other's rationale, intents, standards andjudgments have a positive or negative impact? Will any resulting consensus or compromise beeducationally desirable or undesirable? These questions are explored by using the plane dcpicted in

Page 11.

14


Figure 5 as a way to slice through the activity dimension, examining how stakeholders influence eachother's rationales, intents, contents, standards, and judgments.

The cognitive model: Implications for aggregation

The cognitive model for assessing portfolios makes no direct requirement that certain content bein place. If a teacher intends to teach X, and the standards permit judgment of whether or not studenthas demonstrated X, we can permit wide latitude on how the student (or teacher and student together)choose to demonstrate X. This may pose problems for aggregators who proceed with trad.tionalexpectations that student behavior will occur under controlled and carefully standardizedcircumstances. In this section we discuss how aggregators can proceed without detracting from theinstructional purposes of the portfolio, and even exploit them as evidence of students' capacity toreflect on their own learning.

As iney develop portf olios, students evaluate their own work; the aggregators evaluate the work ofmany students. Such dual evaluation of portfolios is simultaneously a strength and a vulnerability.The selective inclusion and exclusion of materials in a specific portfolio reflect a particular student'srationale, intents, and standards; assessment across por tfolios is based on the aggregator's rationale,intents, and standards. The two sets may be different, the cognitive model tokrates such diffcrences,thereby safeguarding portfolios as a context where students develop independence in the processes ofself-assessment.

In the event that an evaluation plan requires that the students and aggregators activities becompatible, or even identical, it is the aggregators' activities, not the students', that must be aligned.As a result, the aggregators may not have a choice in whether best work or process is described,whether single or integrated content is examined, or whether judgments are based on relative orabsolute standards. The students must not be asked to engage in activities that prevent them frompursuing their own rationale, intents, and standards. A major function of portf olios is to encouragestudents to become independent in all the activities of self assessment. The students' independencemust not be compr Dmised.

There are several ways that aggregation can proceed, even when students are given free rein indetermining rationale, intents, and standards. An obvious way is for aggregators to explain theirinterests bef ore students begin their portfolios, asking students to address both their personal and theaggregators intereos. Ideally, knowledge of the aggregators' interests will be instructive, providingstudents an alternative assessment model that makes them aware of the values held by otherstakeholders. for example, say the aggregator is looking for the correct use of punctuation. If thestudent has written a piece using a stream of ccnsciousness technique, knowledge of theaggregator'3 interest in assessing use of cont.entional punctuation will allow the student to callattention to t:-.e use of nonconventional punctuation and request that it not be judged usingconventional standards. It is imp ,rtant that the aggregators' activities be explained in sua a way thatthey do not devalue the student' personal choice of rationale, intents, and standards. The aggregatormight, for example, consider adding categories that record both the accurate use of conventionalpunctuation and the appropriate use of nonconventional punctuation. Thus, the collected work placedin the portfolios provides a context that may lead to a richer understanding of, or even differentconclusion about a student's abilities made with more traditional methods. Conceivably, this approachcould be used to provide a deeper appreciation of the performance of students from linguistic orethnic minority groups.

Page 12.

1 5

Paulson & Paulson How do Portfolios Mea.urc Up?

Portf olk,s are also repositories of metacognitive data that are unavailable in more traditionalsources. Aggregators have the opportunity to :P;port the frequencies with which students'metacognitive reflections fall within the various cells of the historical by activity dimensions (Figure3). For example, evidence of students' metacognitive aLtiv::ies may include students' description ofthe experiences that they believe account for differences between t..2-ly and recent work, ordescriptions of their activities as they proceeded to compile their portfolios and evaluate the contents.Students may report how personal standards have changed, how what they once considered asatisfactory level of performance is no longer. Aggr3gators also have an opportunity to observe thedegree to which students satisfy their own goals and the consistency and sophistication with whichstudents make judgments about their work.

Additionally, the portfolio can provide inf ormation useful for program evaluation. The programevaluator can report the degree io which there is evidence that student portfolios are consistent withprogram rationale (philosophy), intents (goals), content (curriculum), and standards. In other words,the aggregator can make statements regarding the match between what the students find valuable andthe goals that educational policy makers believe should be valuable.

In summary, the cognitive model is like the traditional models of assessment in that both setspecific criteria on what constitutes a portfolio. The traditional model focuses on structure: whatspecific components must be present to have a proper, scorable portfolio. The components must bestandardized. The cognitive model is constructivist: it focuses on the processes that must be present inorder for the student to produce a proper, describable portfolio. In assessing, the cognitive modelfocuses more on proce - than product. The cognitive model allows students options in how theyaddress their ov n or t aggregators' rationales, intents, and standards. It is our personal belief thatrequests for specific exhibits (e.g., outputs, checklists, assignments) can and should be avoided. Bydoing this, aggregators may be unable to report what students do not know, but have a uniqueopportunity to discover what they do know and how they choose to put this learning to use.

Assessment using judgment

Thus far, we have argued that portf olio assessment should focus on multidimensional, cognitiveprocesses rather than unidimensional achievement of facts and skills. We described an assessmentmodel that is descriptive and encourages the collection of complex and varied information. We tookthe stance that the ultimate outcome, the bottom-line of portfolio assessment, should be determinedby human judgment. Finally, we discussed how the model could be used to aggregate information.

There is a fundamental question that we must now confront in proposing assessment techniquesthat don't just allow diversity, but encourage it. That question is this: Does our view of assessmentleave us with a solipsistic world, one without rules in which "anything goes", where "what is theanswer?" is replaced by "what do you want the answer to be?" What happens to "objectivity" in thecognitive model of portfolio assessment? D.C. Phillips (1989) provides an answer that works for us.He writes

It turns out, then, that who: J crucial for the objectivity of any inquiry -- whether qualitative or quantitative-- is the critical spirit in which it has been carried out." (p, 35)

Thus, we call for portfolio assessment that respects the human capacity for judgment.

Instead of using test scores per se as the outcome indicators, our model uses judgments that havecriteria and are open to review. P. Paulson (1972) had judges refer to a taxonomy of possible content

Page 13.

1 6


found in student paragraph summaries in order to reliably identify commonalities even though theparagraph summaries had no words in common. F. L. Paulson (1974) used a judgment system to assessthe impact of instructional television on the cooperative behavior of preschool children.

Several portfolio projects base outcomes decisions on judgments. The Fort Worth pilot project inwriting portfolios (Lewis, 1990) uses structured teacher judgment to determine change both withinand between grades. The procedure focuses on variables to which the portfolio is particularlysensitive (e.g., increases in lenga. or complexity of pieces, self-initiated revisions, risk taking,metacognition). Medford (Oregon) High School employs panels of faculty and community membersto rate the performance of students participating in the Senior Project (Weiss,1989; The SeniorProject, undated). Vermont's statewide portfolio program (see Writing Assessment LeadershipCommittee, undated) asks citizens to participate in portfolio assessment in an effort to encouragediscussion and provoke thought about how well the schools are achieving their goals. According to theCommissioner of Education (Mills, 1989), "We are undertaking this massive project because we areinterested in real student's work, real performance, not the proxy delivered by standardized tests." (p.11) The Stanford University Teacher Assessment Project (a project with a strong portf oliocomponent) is developing judgment based criteria for assessing teacher competence (Hertel, 1990;Schulman, 1987), and Connecticut performance assessment program is developing classroomobservation instruments (Connecticut Competency Instrument) that are flexible and context sensitive(Suen & Davey, 1990; Hertel, 1990).

Methodology: judgment, description, and (yes!) numbers, too

Whatever the approach to making judgments, several research and evaluation methodologies areavailable that can make the process more rigorous. This section looks at several that are particularilypromising. One comes from psychometrics, ihe study of ways subjective human judgment may bequantified. Another is an extension of classical test theory that recognizes and estimates multiplesources of measurement error allowing for a more multidimensional approach to measurement.Others from ethnographic research provide clues on how to aggregate from case studies anddescriptivc data. Additionally, we speculate about the implications of chaos theory for educationalevaluation.

Psychometrics began with the pioneering work of Francis Galton and was continued by Thorndike(1948) and Stevens (1958). Under pressure of congressional legislation to protect environmental andecological resources, the U. S. Forest Service asked psychologist Terry C. Daniel (1990) to developways to use human judgment to to measure the quality of the natural environment. Daniel's chargewas to expand the narrow, bottom line, unidimensional thinking that represented environmentalvalues in economic terms (e.g., representing the value of trees as board feet) to a broader basis thatwould accommodate ways of representing natural scenic beaaty, wilderness experience,outdoor recreation, and visibility values. In a sense, his problem was to apply a new way ofaggregating the value of the Department of the Interior's portfolio.

Daniel had to deal with two constraints that often confront portfolio assessors. First, existingpolicies and pi oredures placed a premium on quantified, objective traits. Daniel had to translate thesubjective assessir .:nt of beauty into a standardized, "objective", quantifiable framework acceptable tohis audience.

Page 14.


Psychometric techniques used by Daniel use multiple judges (sometimes representing differentpoints of view) to make independent ratings of stimuli (in this case landscape examples). Theprocedure uses sums of judgments by all judges across landscape examples and the sum of landscapeexamples across all judges to develop an overall index of scenic beauty. What goes in are highly"subjective" human judgments of the beauty of different settings, what comes out are surprisinglystable estimates of environmental quality across different reference groups and environmentalsettings. Daniel's work is particularly appealing for portfolio assessment where students areencouraged to attend to self reflection, feelings, aesthetics, social significance, and other dimensionsthat are poorly measured using standardized techniques.

A second technique often used in judgment-based studies (see, for example, Paulson, 1972) comesfrom the use of the analysis of variance, a statistical technique that looks at test reliability from amultidimensional viewpoint (Cronbach, et.al., 1972; Feldt & McKee, 1958; Shavelson, Webb, &Rowley, 1990). Generalizability theory uses the analysis of variance (ANOVA) to attribute error ofmeasurement to multiple sources of variation. In their paper on performance assessment, Shavelson,Carey & Webb (1990) contend that

(*cause the complexities of such (performance assessment] measures will not match the assumptions ofcurrent psychometric theories (e.g., item response theory), creativity is...needed to exploit existingpsychometric theory (e.g., generalizability theory) and to develop new theories to make the new testingtechnologies efficient and cost-effective. (p. 697)

Generalizability theory is uniquely qualified to address issues identified by Suen and Davey (1990)in their analyses of Connecticut's performance assessment procedures. One issue is whether differentstandards for reliability of measurement apply for performance assessment and traditionalachievement tests. The more situations are allowed to vary (i.e.,remain nonstandardized) the morethe traditional measures of reliability and validity drop. In ANOVA terms, the more fixed andthe fewer random facets (i.e., the more standardized the test), the "better" the reliability and validity.Traditional paper and pencil measures maintain high reliability by fixing conditions, but pay the pricein reduced authenticity. The contrast between paper and pencil tests and authentic tests (includingportfolio assessment) can be viewed as a continuum of high to low standardization. Generalizabilitytheory is a technique that can yield a more refined analysis throughout this continuum.

Psychometrics and generalizability theory both grow out of the tradition of quantitative research inpsychology. Enthnographic research also provides approaches to aggregation used in anthropologyand sociology (Schofield, 1990). Ragin (1987) is developing a qualitative comparison method thatuses boolean algebra to make sense out of complex, multiple patterns found in case studies. Hissystem lends itself to social research because it allows the same cause may have different effects andthe same effect may have different causes. Noblit and Hare (1988) while rejecting the concept ofaggregation because it loses more than it keeps, offer a technique called meta-ethnology design( ' topreserve the interpretive nature of qualitative information across studies. Their technique is designedto develop a synthesis, a translation into a common language permitting comparisons across studies.

Our final example of analytic techniques with potential application to portfolios (at least in thelong term) comes from the emerging science of chaos (Gleick, 1987). Chaos theory is the study of thegrowth of complexity in nature. It is the study of patterns in natural events that had formerly beenthought to be random (test theories assume variation away from the unidimensional model is randomerror). Chaos theory uses the mathematics of nonlinearity in which simple relationships are replacedby complex patterns that accurately describe the behavior of many naturally occurring events. It hasalready found practical application, for example in medical treatment (see Gleick, 1987 p. 280ff).Techniques with colorful names, such as bootstrap and jackknif e, are already finding their way intothe behavioral sciences (Cronbach, 1988, p. 49, Lunneborg, 1988).

Page 15.

18


The methods reviewed come from both quantitative and qualitative research. Scene, likepsychometrics, and generalizability have been available to educational researchers for some time.Others like tho:,e used in ethnographic and qualitative research are begianing to find acceptance ineducational research and evaluation. The American Educational Research Journal, for example,has only recently accepted qualitative research manuscripts The more exotic techniques like chaostheory may remain in the wings for now but have a leading role in the next generation of educationalresearch.

Obviously there are a great many techniques that might be used in the cognitive model forassessing portfolios. We urge adoption of the pragmatic position recommended by both Patton (1990)and Stake (1963). Portfolio assessors and aggregators should adopt and adapt freely from all schoolsof research methodology but carefully remain free from the confines of allegiance to any singleapproach. In portfolio assessment, diversity in content should beget diversity of analysis.

Practical applications of the model

To this point, we have explored theoretical issues with profound practical implications forportfolio assessment, and we have outlined a conceptual model for assessing portfolios andaggregating the results. How might such a model look in practice?

Any or all subject areas may be represented. Within subject areas, students and teachers generallyhave wide latitude regarding material that finds its way into the portfolios. There may be more thanone set of rationales, intents, standards, or judgments. Multiple sets of these may represent theaggregator, teachers, students and other stakeholders. The teachers and students may compare theirsets to those of the aggregator, but not necessarily with any intention to make them the same.

Teachers as well as students may place material into the portfolios if ground rules are clear at theoutset. In principle, students should have complete freedom over what they may put in the portfolio,but they are encouraged to give their reasons for tlie selection. Teachers provide younger chile enmore models of the decision making process.

The aggregator's initial attention is to the extent to which the district has used a model thatpreserves the function of portfolios for teaching self -assessment. Aggregators determine the degreeto which students and teachers have related contents to district intents and evidence that students andteachers have modified or enhanced goals to reflect individual needs or interests. The ,.ggregatoridcntif ics which district goals and priorities are addressed in the portfolios and seeks confirmationthat individual students are free to demonstrate competence in whatever ways make the most sensepersonally. In addition, the aggregator ascertains that the district encourages students to contributetheir own personal goals. Finally, the aggregator judges the degree to which students are successful inmeeting their own standards.

At first glance, the "contents" of the portfolios are simultaneously an evaluator's nightmare and ananthropologist's bonanza. However, most of the material is carefully indexed and, the indexingprovides structure. The indexing may be to district goals, or according to the student's, teacher's, oraggregator's intents, or cross-indexed to several. Indexing to these goals and intents provides theaggregator a clearer pathway through the portfolio contents.

Page 16.

1 9

Paulson & Psvilson How do Portfolios Measure Up?

All portfolios also contain specific guidance on how performance is to be judged. For example, awriting sample is accompanied by a handbook on the district's analytical trait writing assessmentmodel. In mathematics, there is a list of criteria applied to problem-solving exercises (For examples,see Beaverton, 1986; California Mathematics Council, 1989, p. 19)

Contents are not standardized. The following examples give a flavor of the diversity of contentsthat could be indexed to a district mathematics objective; all student portf olios contain a forminstructing them on how to present and document evidence that they can perfol m long division butstudents have used innovative ways to document competence at long division. One student's portfoliocontains a copy of a standardized achievement test report with the number of items answered correctlyin long division highlighted by a yellow marker and the district goal number penciled in. This studenthas written a note that her dad is only interested in the bottom line! Another student's portf oliocontains materials developed as a part of a "Business in the Schools Project" sponsored by a localbusiness group. Although there are dozens of cross references to district objectives, the crossreference to long division is on the project's inventory reports; the student attached several examplesof long division necessary for completion of these reports. A third student includes a two page writtenproblem analysis of the solution to an arithmetic challenge problem (see California MathematicsCouncil, 1989, p. 12). An appended note points out how the solution required the accurate calculationof long division. At the end of the year, the portf olio review committee, applying standards stated atthe outset, will look at these various kinds of evidence to determine whether students satisfied thegoal.

Portfolio contents are unique in their attention to metacognitive processes. Students and teachersdon't just put material into the portfolio, they reflect upon the material, what they learned from theexercises, and why they are placing material into the portf olio.

There is an important strategy implicit in this description. The aggregation process is structured insuch a way that external impact on the portfolio contents is limited to the statement of instructionalgoals and standards. The contents are free to vary widely but with the challenge to the student andteacher to relate them to the goals and to discuss the outcomes in terms of the standards. This leavesthe program open to multidimensional thinking. One can approach the goals in a variety of ways usinga variety of media. There is minimal impact of the assessment on the thing assessed. Diversity isencouraged and the assessment is performed by judges using criteria that adapt to variability.

Example portfolio projects

The examples to follow illustrate how the cognitive model for assessing portfolios (CMAP) may beused to describe three existing but very different portfolio projects. One project uses portfolios fordeciding which students should be admitted into programs for the gifted. We illustrate how CMAPcould provide a structure for project evaluation. The second describes a program in which a classroomteacher uses the portfolio to promote student growth in writing. CMAP illustrates a structure forevaluating the instruction students receive in self assessment as well as the quality of their writing. Inthe third, we show how CMAP may be used to describe an example of metacognitive activity. CMAPprovides a structure for evaluating student's self assessment as reported and documented by thosestudents.

Page 17.

20


Example 1. The Pupil Product Portf olio

The Pupil Product Portf olio (PPP) is a system for helping districts identify gif ted children(Grades K 3) and for helping teachers make instructional decisions. It was developed by BertieKingore (1990a, 1990b & 1990c), a specialist in gifted children at Hardin Simmons University. ThePPP has been tested under her supervision with several hundred children, and reports are inpreparation (Kingore, 1990c).

The PPP is a collection of physical evidence of excellence in performance. It is assembled and keptby the teacher. The PPP may contain three categories of materials:

1. Spontaneous products from school and home that the child produces spontaneously andnot as a part of class assignment. Kingore describes them as things that "just happen."

2. Periodic Products collected from classroom work (e.g. writing journals) in order tonocument growth. (This category is used for instructional purposes only and not used foridentifying the gifted.)

3. Planned experiences from standardized situations collected on all children at the same timein order to give all children an equal chance to qualify for gifted programs. An example isDrawing Starts, ambiguous, partially drawn figures that children complete. The scoringfocuses on process, with no right or wrong answers.

To identify gif ted primary students, a judgment team reviews six examples of planned experiencesand three spontLneous products found in the PPP. Giftedness may be demonstrated in five categories:

1. In-depth understanding

2. Unique or unexpected idea

3. Exceeds grade or age level expectation

4. Advanced, complex organization

5. Resourceful and clever use of materials

Each judgment decision is on a binary scale: the child either does or does not demonstrateperformance consistent with one or more of the c tegories. The total score on PPP is the sum ofpoints awarded for the 6 planned experiences and 3 spontaneous products. Children who receiveeither 8 or 9 points arc classified as gifted. Kingore (1990c) reports relatively high (90% plus) levelsof agreement between judges.

An analysis of PPP using CMAP reveals almost immediately that PPP does not focus on thestudent as a self-assessor. However, the model can be directly applied as a structure forcomprehensive program evaluation. And while the PPP does not focus on self assessment, CMAPenables the aggregator to discover student metacognition as "serendipitous breakthroughs" within thecontents of the portfolios.

Applying CMAP to the PPP project, the active stakeholders clearly are the district and the teacher.The student is a passive stakeholder. Although the studelit is the person who supplies the contents and

Page 18.

21


is directly affected by the decisions, the student is not asked to actively participate in any of the otherassessment activities: formation of a rationale, intents, standards or judgments.

The activity dimensions arc readily identified for the district stakeholder and the teacherstakeholder. The district's rationale is to identify gifted students. The intents are for students todemonstrate giftedness as defined by the five categories. The portf olio contents are to be six plannedand three spontaneous student products, selected by the teacher. The standards are binary onesapplied to each each of the nine products. The judgment is giftedness, based on a score of 8 or 9points. An aggregator applying the cognitive model to this dimension would conclude that all theactivities of assessment were in place and that there was internal consistency along the activitydimension.

The teacher's rationale has two components, one being to meet a district requirement for selectinggifted students. The teacher would therefore have adopted the district's intents, standards, andjudgments as the basis for selecting student work. Program evaluation would look for consistency inthe way the teacher went about the district's business. The teacher's other rationale would be to guideinstruction. The PPP project does not specify the intents, standards or judgments in relation toinstruction. The aggregator would have to determine these through teacher interview and look forevidence of these in the contents of the student portfolios.

Of the three dimensions of the cognitive model, the historical one is least relevant to the PPPproject because, by definition, the Pupil Product Portf olio is specifically concerned with outcomes,not transactions or antecedents. Transactions that account for these outcomes are not a focus of thedistrict as it goes about selecting gifted students. It is assumed that transactions are of great concernto the teacher who helps students develop planned and periodic products and who uses these (andspontaneous products) to guide future instruction. While perusing the portfolios, the aggregatormight find evidence that the students were aware of their transactions (just as there might be evidencethat they engaged in activities of self-assessment) and such a discovery would enrich the aggregator'sreport. The final component of the historical dimension, antecedents, does not appear for anystakeholder or activity.

Example 2. A Writing Portfolio

A description of a writing portfolio activity in a fourth grade classroom appeared in the NWEAPortfolio Assessment Newsletter (NWEA, 1990b). It is an activity conducted by RhondaWoodruff, a fourth grade teacher in Beaverton. Oregon. The project can be described using the threedimensions of the CMAP.

Stakeholders.

The student's writing portf olio is used by three stakeholders, students, teachers, and parents.Students take an active role in sc.. .,cting material and maintaining the portfolio and exhibit a high levelof ownership in their portfolios. The teacher uses the portfolio to provide feedback to students,monitor the class, report to parents, and provide information for next year's teacher. The portfolio isavailable to parents and is a favorite source of information on progress in writing.

Page 19.

22

Paulson & Paulson How do Portf olios Measure Up?

Historical.

The historical dimension is represented primarily as transactions and outcomes. The transactionaldimension is represented in a least three ways. A record of published writing is kept over the entireyear. Students are encouraged to include any writing they feel proud of. The process of writing is aclear focus of daily instruction and learning. This would support a look at growth. Since the contentsare described as finished writing, the primary focus would appear to be on outcomes.

Activity.

The teacher described and modeled her rationale, intents, contents, standards, and judgments. Shekept a portfolio herself and taught the students how to engage in the same process. Excerpts from thearticle are accompanied by CMAP analyses to show just how the model can be used as a framework fordescribing existing program.

Analysis

In the analysis, excerpts describing the program appear on the left, the analysis of each excerptappears on the right.

Program Description

I n my fourth grade classroom, students learnwriting as a process. The use of writing stagesarc taught. The writing stages include: pre-writing, draf trng, revising, editing, publishing,and sharing.

In addition, my young writers are instructed inthe use of Analytical Trait Scoring. Analyticalscoring defines characteristics of writing. Itallows us to measure a writer's ability to dealwith individual components of writing.

The six traits we use are: ideas, organization,voice, word choice, sentence structure, andwriting conventions

Students learn how to assess writing to thesetraits, recognizing that a paper may bemechanically sound but weak in ideas, orperhaps strong in organization but rather weakin vocabulary. Students also receiveinstruction on how to revise papers using thesetraits. They learn how to make weak papersstrong. These six traits are my six writing unitsf or the year.

CMAP Analysis

Here the teacher conveys to the studentsher expectations regarding thetransactions that will occur as a part of theportfolio activity.

The teacher's reference to the AnalyticalTrait Scoring system indicates thestandards that will be applied to theportfolio. Transaction is implied by theteacher's :eference to specific instructionin the system.

The listing of the six traits themselves is anindication of intents.

This statement is transactional in whichthe teacher conveys to the student specificexpectations regarding their performance.

[Not stated in piece: Each student in thisdistrict has a copy of the district's writinghandbook that lists the standards used tojudge the district's writing assessment.Thus, the teacher transmits the district'sstandards to the student.]

Page 20.

23


Description (Concluded)

Writing portfolios are essential in helpingprovide writing success for my children.Portfolios are meaningful to students,teachers, and parents alike as they serve as aplace where we, as writers, can store part ofourselves. Our writing successes. Ourinspirations. Our hearts.

Every writer in my class...has a writingportfolio (with) ... (I) a list of books read bothin and out of class, (2) a list of writingspublished or shared during the year, and (3)xerox copies of writing treasures we, as writers,wish to save (the originals are in much demandat home).

Writing in the portfoGos may come fromjournals, class assignments, writing composedat home, on trips. Anywhere!

At the end of the school year, students choosefrom their treasures three samples of f avoritewritings to leave in their portfolios to be passedon to their fifth grade teachers. Students add tothe portfolios self -assessments of their progressin writing as well as fall and sin ing writingassessnients completed in class.

My wish is to not only send a portf olio on to thenext year with every student but to receive onefor each new student coming to my room in thef all. I would love to be able to tell every newstudent the very f irst day of school, "I may notyet know you by your face, but I know youalready by your words Your thoughts Yourinterests. f our successes. Welcome."

CMAP Analysis (Concluded)

Here is a succinct statement of theteacher's Rationale. It is stated in acollaborative mode (the reference tostudents, teachers, and parents) andimplies attention to metacognitiveprocesses.

Teacher establishes the general contentsof the portfolio. Students have widelatitude in terms of the contents theyselect to include.

This describes the student as stakeholder.It deals with the antecedents,transactions, and outcomes.

This year's outcomes become next ycar'santecedents.

Example 3. Thc Metacognitive Letter

Jill Marienberg (see NWEA 1990a), a Hillsboro, Oregon, high school teacher, provides examplesof letters that studcnts submit that describe their writing portfolios (Marienberg, 1990). These letters,which Marienberg calls rnetacognitive letters, demonstrate that thc students have actively engagedin all five activities of self assessment and are aware of many of the transactions critical to theirlearning.

In the example metacognitive letter that follows, the student states rationale and intents in thc firstparagraph, lists the contents, then discusses each piece of writing she has chosen to included in thcportf olio. In the discussion, the student specifies or implies the standards by which she judged herwork.

Apparently, the teachcr provided thc students with a rationale which might be paraphrased, "Thepurpose of the portfolio is to allow students to collect samples of their writing during the junior year

Page 21.

24

Paulson & Paulson How do Portf olios Measure Up?

that accurately represent their thinking and writing abilities." Students typically provide a version ofthis statement in their letters, e.g., in the example letter, "...what I consider to be my finest work....anoverview of my ability to write and and organize my thoughts."

In examining several metacognitive letters from Marienberg's classes, statements of intenls varyfrom student to student. The example letter lists "expository, descriptive and reflective pieces", as wellas evidence of writing and organizing "under pressure." Another student listed intents asdemonstrating the ability to produce "expository, informative, and creative pieces".

Contents also vary from student to student. Some students included more poetry than others, a fewinclude research pieces, and most include something written as a part of a major class assignment.Marienberg is clear that which pieces are selected is completely up to the student. The student whowrote the sample metacognitive letter was unique in including an example from the district's directwriting assessment because she wanted to show her ability to write "under pressure". It is interestingthat when students were asked to demonstrate how well they can write, the student who choose toinclude an example taken from the district's direct writing assessment made an explicit disclaimer thatit showed the ability to produce under nontypical circumstances.

According to the metacognitive letters, students clearly set standards by which to judge their work.These standards are specific not only to each piece of work but to the interests of each student. Acrossthe class, students' standards vary from holistic ones (e.g., "the 'art' of portraying the usual asunusual") to moderately analytic ones (e.g., "showing voice"), to highly analytic ("improving verbchoice"). Statements of standards may include aesthetics considerations (e.g., "startling imagery"), thedemonstration of the achievement of traditional goals (e.g. "writing within a certain structure"), orbreadth of skill (as in the above example where the student includes an example of writing underpressure to complement the other materials in the portfolio).

There is substantial evidence :hat students are aware of the transactions that account for theirgrowth during the year. The example letter includes these statements: "I believe that all of my writingshave Ix Jited from this assignment early in the year"; "This selection stretched me the mostintellectually"; "My knowledge of main characters in The Scarlet Letter had to be intimate..."; "Myorganization and logic skills enable me..."

Finally, the letter ends with an elaboration of the portfolio's rationale: "To assess my growth,competence, and college-level thinking."

A transcript of the metacognitive letter written by student S.D. follows.

To Whom it May Concern

Enclosed in my portfolio you will find five samples of what I consider to be my f inest written workduring my pinior year in high school. The portfolio contains writing samples from expository,descriptive, and reflective pieces, to give yau an overview of my writing ability. In all but one ofthese works! was allowed plenty of time f or revision and critiques. However one piece which! haveincluded is front a timed in-class assignment. I hope this example will provide you with an idea of myabiltty to write and organize my thoughts under pressure

Contents.

I "House Cleaning"2. Poetry "You and I"3. "My Present"

Page 22.

25


4. Literary Analysis of the Major Characters in The Scarlet Letter.5 Poetry: "Blue"

Discussion

"House Cleaning" is a short narrative written early in my junior year which f ocused on thedescription aspect of writing. I believe that I have captured the "art" of portraying the usual asunusual in this paper. Sonic of the sentences combined just the right verb choice with the correctdescription to produce startling imagery. The sentences also pick up a beautif ul f lowing quality. Ibelieve all my writings have benef ited from firs assignment early in th:year which taught me to beoriginal and leave state-of -b:ing verbs behina

The f irst poetry piece "You and 1" is included to show my capability of writi ng within a certainstructure. It uses several f igures of speech to depict the same old subject of love in a diff erent way.

The third writing which I have included, "My Present" is a piece done in class under o f iffy minutetime restriction as practice for the Hillsboro District Writing Assessment. Although we were free towrite on any subject I feel it is a good example of my ability to work alone under pressure. I wouldhave liked to have had more time to ref ine and rewrite this piece, however, I feel it has more voicethan any of my other works.

The fourth piece which I have submitted for you review is "A Literary Analysis of the Me!orCharacters inThe Scarlet Letter." This selection stretched me the most intellectually and was themost difficult to write. My knowledge of the main characters in The Scarlet Letter had to be intimateas well as my grasp of Lawrence Kohlberg's theory of moral development. I believe, however, that Irose to ;he occasion well. My organization and logic skills enables me to put together a smooth-f lowing, easy to f ollow paper, with specific supporting detail throughout. I am pleased with thefluency of the sentences and of the depth of knowledge this paper portrays.

The second poetry piece, "Blue" is included because of its startling sensory imagery. Written infree verse I feel it is more me than ony of my other works. This poem contains some intriguingexamples of personification and my observation skills are tested. I have given this paper a sense ofroundness by bringing "Blue eyes" back in the end.

With a wide variety ref lected in my f rye pieces, I hope you will be able to assess my growth,competence, and college-level thinking successf ully. I am proud to submit this collection as aref lection of my ability.

Respectfully submitted,

S.D.

Recap of the three examples

In the Pupil Produ ct Portfolio, CMAP clarifies that self-assessment is not a major purpose ofthis project. In fact, it is not clear whether students are even aware that the assessment is taking place.The historical dimension also plays a minor role. However, the stakeholder dimension is vet yinteresting and an aggregator could employ CMAP to clarify the relationships among the stakeholdersand to examine the consistency of their activities.

In the writing portfolio, CMAP reveals that a major purpose of this portfolio project, from theteacher's point of view, is to teach self-assessment. All five activity components of self-assessment aretaught, modeled, and practiced in collaboration. The historical dimension is important because theinstructional emphasis is on assessment processes. The example lends itself to aggregation in twoways: The aggregator can more easily judge the contents of multiple portfolios at the teacher levelbecause all have identical rationales, intents, and standards. The aggregator can also inspect the

Page 23.

26


ntents for evidence that the :.tudents have engaged in self-assessment, and that there have beenhistorical changes in the quality of the:r self-assessment activities.

In the metacognitive letter, each student provides written self-assessment as part of the portfoliocontent providing an aggregator with direct evidence of self-assessment. One way to aggregate acrossportfolios would be to use "evidence of metacognition" as the aggregato, 's rationale, and to describethe CMAP dimensions represented in the student's self-assessment. By comparing the student'sintroductory letter to other contents of the portf olio, the aggregator could make judgments regardingthe students' ability to self assess and their awareness of growth on the historical (antecedent,outcome) dimension. The aggregator could even look at awareness of the metacognitive transactionsthat arc responsible for this growth.

An aggregator interested in overall results could use one of the judgment-based system considere 1earlier (page 13ff) to produce information that could be aggregated at the school or district level. Forexample, in the example of the metacognitive letter, an aggregator might use teams of judgesrepresenting various stakeholder groups to score the overall quality of writing in the portf olios. Theaggregator could then write a comprehensive report to describe the program from the perspectives ofthe stakeholders while putting the quantitative results into a context that would make themmeaningful.

Conclusion

There is comfort in test scores. They arc units that can be counted and accounted. Test scores fitnicely the American value system of achievement measured by big bottom line. You can take themto the bank! The questions this paper raises are not about whether we measure outcomes inquantifiable units -- our "bottom line" oriented culture derAands it. Rather, we raise the concern thatwe examine carefully our currency before we start counting. We arc a diverse people who 'hink ofour society as a great melting pot where strength grows out of an amalgam of differences. We arethe descendants of pioneers, those risk takers who had great ideas and the initiative to carrythrAigh. And, maybe most American of all, when we are at our best we demonstrate know-how, thatfabled V,. i!'ee ingenuity that allows us to carry through. Look again at the multiple choice tests thatguide the education of our next generation of leaders. Are they telling us the real bottom line ofAmerican education? Arc we focusing on developing students with initiative, know-how, and awillingness to take risks, or are we focusing on those who know the "right answer" in its most limiteddefinition?

Portfolios provide a complex and comprehensive view of student performance that encourages usto look at learning as a complex and multiclinIznsional process. They allow us to define achievement inbroad, adaptive terms rather than narrow, restrictive ones. Portfolios also provide a complex andcomprehensive view in a context where instruction and assessment are inseparable. 3ecause assessingprofoundly impacts instruction, we urge that assessments be designed that support the instructionalvalue of the portf olio. We encourage an approach that f ccuses on the process of assembling and usinga portfolio rather than on standardizing what is placed into the portf olio.

The assessment community is at a juncture. We can view portf olio assessment as "business asusual" and impose existing, inadequate, procedures based on traditional models, or we can approachthe challenge from new and exciting perspectives. The choice is ours. In advocating the pluralistperspective, we echo Eisner's (1990) assertion that

Page 24.

27


...there is no single, legitimate way to make sense of the world. Different ways of seeing give us differentworlds. Different ways of saying allow us to represent different worlds. Helping people participate in aplurality of worlds...is what education ought to try to achieve....We need multiple voicns and we need peoplewho can understand them. (p. 111)

At the start of this paper we noted that the dictionary's definition of aggregate read, "a total orwhole: a group of distinct things gathered together". The dictionary offers a second definition as well,"the sand and pebbles used in making concrete." Together, these definitions p. ovide a metaphor tosummarize our argument. In the attempt to find the aggregate in portfolios, we should seek to uncoverthe be...Ay of the structures and the strength of the concrete, not count pebbles and sift sand.

(Meanwhile...after seven and a half million years, Deep Thought signaled that it had finished computing theanswer. The crowd gathered. The tension was unbearable.]

"do you have..."

"an answer for you?" interrupted Deep Thought majestically. "Yes I have. Though I don't thinkthat you are going to like it."

"h doesn't matter....We must know it. Now!"

"All right....The Answer to the Great Question. Of Life, the Universe, and Everything...is", saidDeep Thought, and paused.

"Yes..."

"Is Forty-two", said Decp Thought with inf mite majesty and calm.

It was a long time bef ore anyone spoke..,.. "Forty two...15 that all you've got to show for seven anda half na!lion year's work?"

"I checked a quite thoroughly," said the computer,"and that quite definitely is the answer. Ithink the problem, to be quite honest with you, is that you've never actually known what thequestion is....Once you know what the question actually is, you'll know what the answer means."

Douglas Adams, The hitchhikersguide to the galaxy

rz E ex- en c e s

Beaverton School District (1986). Analytical trait writing: Student Copy. Beaverton OR:Beaverton School District 48J.

Calif ornia Mathematics Council (1989). Assessment alternatives in mathematics, Berkeley CA:EQUALS, Lawrence Hall of Science.

Cronbach, Lee J. (1957). Two disciplines of scientific psychology. American Psychologist, 12,671-684.

Cronbach, Lee J. (1975). Beyond the two dis,..iplines of scientific psychology, AmericanPsychologist, 30, 116-127.

Page 25.

28


Cronbach, Lee J. (1988). Playing with chaos, Educational Researcher, 17(6), 46-49.

Cronbach, Lee J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability ofbehavioral measurement.;. Theory of ,7eneralizaHlity of scores and profiles. New York:Wiley.

Cronbach, Lee J. & Snow, R. E. (1977). Aptitudes and instructional methods: A handbook f orresearch on aptitude-treatment interactions. New York: Irvington.

Daniel, Terry C. (1990). Measuring the quality of the natural environment: A psychophysicalapproach. American Psychologist, 45 (5), 633-637.

Daniel, Terry C. & Boster, Ron S. (1976). Measuring landscape esthetics: The 6- enic beautyestimation method. (USDA Forest Service Research Paper RM-167). 'Ft coffins CO: RockyMountain Forest and Range Experimentation Station. (Out of print)

Donmoyer, Robert (1990). Gencralizability and the single-case study. In E. Eisner & A. Peshkin(Eds) Quantitative inquiry in education: The con:inuing debate. New York: TeachersCollege Press.

Eisner, Eliott (1990). Objectivity in educational research. Paper read at the annual meeting of theAmerican Educational Research Association, Boston MA. (The paper is from The enlightenedeye: On doing qualitative inquiry. New York: Macmillan, in press)

Feldt, L. S. & McKee, M. E. (1958). Estimation of the reliability of skills tests, The ResearchQuarterly, 29 (3), 279 - 293.

Gardner, Howard (1983). Frames of mind. New York: Basic Books.

Gardner, Howard & Hatch, Thomas (1989). Multiple intelligences go to school. EducationalResearcher, 18 (8 ),4-10.

Glaser, Robert (1990). The reemergence of learning theory within instructional research. AmericanPsychologist, 45( 1), 29 - 39.

Glaser, Robert (1971). Instructional technology and the measurement of outcomes: Some questions.In W. J. Popham (Ed), Criterion-referenced measurement: An introduction. EnglewoodCliff s, NJ: 197i. (Originally published in American Psychologist,1963,18, 519-521.)

Gleick, James (1987). Chaos: Making a new science. New York: Viking.

Guba, E. G. & Lincoln, E. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.

Hardison, 0. B., Jr. (1989). Disappearing through the skylight: Culture and technology in thetwentieth century. New York: Viking.

Hertel, Edward H. (1990). From expert opinions to reliable scores: psychometrics for judgment-basedteacher assessment, Draf t paper presented at the meeting of the American Educational ResearchAssociation, April 1990, Boston.

Page 26.

2 9


Joint AERA, APA, NCME Committee for the Review of the Standards for Educational andPsychological (1985). Standards f or educational and psychological testing. Washington,DC: American Psychological Association.

Kingore, Bertic (1990a). Appropriate programs f or gifted young children, Tempo, (Spring), 8 - 9.

Kingore, Bertie (1190b), The Kingore observation inventory (K01). Des Moines, Iowa:Leadership Pt blishers (P.O.Box 8385, Des Moines IA 50301)

Kingore, Bertie (1990c). Audio tape of teacher workshop KOI Training presented at MultnomahESD Feb 8-9, 1990. Kingore discusses the Pupil Product Portfolio in a f orthcoming book.

Lewis, Linda (1990). Pilot project f or portf olio assessment. Fort Worth Independent SchoolDistrict: Fort Worth, TX.

Lunneborg, Clifford E. (1988). Bootstrap applications for the behavioral sciences, AppliedPsychological Measurernene, 12(1), 105-106. (Software available in MS-DOS format)

Marienberg, Jill (1990). Unpublished examples of student portfolios. Hillsboro High School, 3285S.E. Rood Bridge Road, Hillsboro OR 97123.

McCordick, Pamela (1972). Machines who think. San Francisco: Freeman.

Mills, Richard P. (1989). Portfolios capture rich array of student perf ormance, The SchoolAdministrator, December, 8 - 11.

Minski, Marvin (1986). The society of mind. New York: Simon and Shuster.

Noblit, G. W. & Hare R. D. (1988). Meta ethnography: Synthesizing qualitative studies. SageUniversity Series Qualitative Research Methods, Vol 11. Beverly Hills: Sage.

Northwest Evaluation Association (1990a). Profile: Jill Marienberg: Language arts in Hillsboro,NW EA Portf olio Assessment Newsletter, Vol. I (I). 5 Centerpointe Dr. Suite 100, LakeOswego OR, 97035.

Northwest Evaluation Association (1990b). Profile: Rhonda Woodruff: Fourth grade teacher inBeaverton, NW EA Portf olio Assessment Newsletter, Vol. I (2). 5 Centerpointe Dr. Suite100, Lake Oswego OR, 97035.

Northwest Evaluation Association (1990c). NW EA white paper on aggregating portfolio data:Revised. (Prepared by Carol Meyer & Steven Schuman). 5 Centerpointe Dr. Suite 100, LakeOswego OR, 97035.

Papert, Se: mour. (1980). Mindstorms: Children, computers, and powerf ul ideas. New York:Basic Books, 190 - 207.

Patton, M. Q. (1990). Qualitative evaluation and research methods. Newbury Park (CA): Sage.

Paulson, F. L. (1972). Live versus televised observation of social behavior in preschool children,Proceedingsk 80th Annual Convention, APA,135 - 136.

Page 27.

30


Paulson, F. L. (1974). Teaching cooperation on television: An evaluation of Sesame Street socialgoals programs, A. V. Communicatiun Review, 22 (3), 229 - 246.

Paulson, Pearl R. (1972). Paragraph summarization by computer: A comparison with summaries madeby pupils. Unpublished doctoral disseration, Stanford University.

Phillips, D. C. (1990). Subjectivity and objectivity: An objective inquiry. In E. Eisner & A. Peshkin(Eds), Qualitative inquiry in education: The continuing debate. New York: TeachersCollege Press.

Ragin, C. C. (1987). The comparative method: Moving beyond qualitative and quantitativestrategies. Berkeley: University of California Press.

Resnick, L. B. & Resnick, D. P. (in press). Assessing the thinking curriculum: New tools foreducational reform. In B. R. Gifford & M. C O'Connor (Eds), Future assessments: Changingviews of aptitude, achievement, and instruction. Boston: Kluwer Academic Publishers.(Quoted in L. Sheppard (1990). Psychometricians' beliefs about learning. Paper presented at theannual meeting of the American Educational Research Association, Boston, April 17, 1990.)

Rose, Frank (1984) Into the heart of the mind: An Anerican quest f or artif icial intelligence.New York: Harper and Row.

Rottenberg, C. & Smith, M. L. (1990). Unintended effects of external testing in elementary schools.Paper read at the annual meeting of the American Educational Research Association in BostonMA.

Schank, Roger C. with Childers, Peter G. (1984). The cognitive computer: On language,learning, and artif icial intelligence. Reading (MA): Addison-Wesley.

Schofield, Janet Ward (1990) Increasing the generalizability of qualitative research. In E. Eisner & A.Peshkin (Eds) Qualitative inquiry in education: The continuing debate. New York:Teachers College Press.

Schulman, L. S. (1987). Knowledge and teaching: Foundations of the new reform. HarvardEducational Review, 57, 1-22.

Scriven, Michael (1972). Pros and cons about goal-free evaluation, Evaluation Comment, 3, 1-7.

Shavelson, R. J.-. Carey, Neil B. & Webb, Noreen M (1990). Indicators of science achievement:Options for a powerful policy instrument. Phi Delta Kappan, May, 692 - 697.

Shavelson, Richard J., Webb, Noreen M., & Rowley, Glenn L. (1989). Generalizability theory.American Psychologist, 44 (6).922-932.

Shepard, Louie A. (1990) Psychometricians's beliefs about learning. Paper presented at the annualmeeting of the American Educational Research Association, April 17, 1990.

Stake, Robert (1975). fo evaluate an arts program. In R. Stake (Ed) Evaluating the arts ineducation. Colun_bus OH: Merrill,13-32.

Page 28.

3 1

4Paulson & Paulson How do Portfolios Measure Up?

Stake, Robert (1967). The countenance of educational evaluation. Teachers College Record, 68(7 ), 523-540.

St.:Yens, S. S. (1958). Problems and methods of psychophysics, Psychological Bulletin, 55(4), 177- 196.

Suen, fbi K. & Davey, Bruce (1990). Potential theoretical and practical pitfalls and cautions of theperformance assessment design. Paper read at the annual meeting of the American EducationalResearch Association, Boston MA.

Swing, S.R., Stoiber, K.C., & Peterson. P.L. (1988). Thinking skills vs learning time: Effects ofalternative classroom-based interventions on students' mathematics problem solving. Cognitionand Instruction, 5 (2), 123-191.

The Senior Project (undated). The senior project. Information brochure published by Far WestEdge, Inc., 1817 Woodlawn Ave., Medford OR 97504.

Thurstone, L.L. (1948) Psychophysical methods. In T. G. Andrews (ed) Methods in psychology.New York: Wiley, 124 - 157.

Tukey, John (1972). Data analysis, computation, and mathematics, Quarterly of AppliedMathematics, 51-65.

Weislogel, R. L. & Schwartz, P. A. (1955). Some practical and theoretical problems in situationaltesting. Educational and Psychological Measurement, 15,39-46.

Wiggins, Grant (1989). A true test: Toward more authentic and equitable assessment. Phi DeltaKappan,May 1989,703-713.

Weiss, Stefanie (1989). No more business as usual. NEA Today, 7(7), 4 - 5.

Wolf, Robert L. (1975). Trial by jury: A new evaluation method, Phi Delta Kappan, 57(3). 185-187.

Writing Assessment Leadership Committee (undated). Vermont writing assessment program.Montpelier VT: Vermont Dept. of Education.

Valencia, Shiela (1990). A portfolio approach to classroom reading assessment: The whys, whals, andhows. The Reading T eacher January 1990, 338-340.

Page 29.

32

Appendix 16

END

U.S. Dept. of Education

Office of EducationResearch and

Improvement (OERI)

ERIC

Date Filmed

March 29, 1991

author paulson, leon f.; paulson, pearl r. title assessing … · 2013-08-02 · portfolio has the...

Documents