Portfolio Assessment: A literature review

Philip Smyth

English Centre

The University of Hong Kong

Overview

Historical perspectives Defining the portfolio Purposes of portfolios Issues in portfolio assessment Research in portfolio assessment My own possible research avenues

Historical perspective

Portfolios widely used for many years Late 80s interest in portfolios for

assessment (Belanoff and Dickson 1991)

90s saw advent of eportfolios A shift in emphasis away from

assessment to learning?

Definitions

“collection of student work that demonstrates achievement or improvement” (Stiggins 1994)

“a portfolio is a collection of evidence that is gathered together to show a person’s learning journey over time and to demonstrate their abilities” (Butler

2006)

Definitions

“…student writing over time, which contains exhibits showing the stages in the writing processes a text has gone through and the stages of the writer’s growth as a writer, and evidence of the writer’s self-reflection on her/his identity and progress as a writer” (Hamp-Lyons 1996)

Definitions

portfolios are “…prepared with a particular audience in mind”, “…are selective” and “call for judgments” (Calfee

and Freedman 1996)

Definitions

“…a purposeful collection of student work that illustrates efforts, progress, and achievement in one or more areas [over time]. The collection must include: student participation in selecting contents, the criteria for selection, the criteria for judging merit, and evidence of self-reflection” (The Northwest Evaluation Association cited in Barret 2005)

Definitions – main characteristics They are collections of work, different from

a single timed impromptu essay or a class essay carried out over a semester.

They are purposeful in that they “demonstrate”, “exhibit” or provide “evidence” of “achievement”, “improvement”, “the writer’s self reflection”, “the writing process” and “the writer’s growth”.

The degree to which these characteristics are evidenced in portfolios largely depends on their purpose.

Types of portfolio

a process portfolio a showcase portfolio an assessment

portfolio A dossier portfolio A reflective portfolio A classroom portfolio

A positivist portfolio A constructivist

portfolio A personal portfolio A structured portfolio An employment

portfolio A working portfolio

Where are portfolios used?

Primary and secondary classrooms In tertiary settings:

Teacher education Medicine Nursing Engineering Dentistry Psychology ESP/EAP writing classes

How are portfolios used?

In a class Across more than one class Statewide Across a university curriculum

Purposes

Accountability; evaluating program or curriculum effectiveness

Evaluating individual student progress; grading, certifying student accomplishment

Diagnosing students’ needs; informing classroom instructional planning

Purposes

Encouraging teacher efficacy; encouraging reflective practice at the school and classroom levels; supporting teachers’ professional development

Encouraging student efficacy; promoting student self-assessment; motivating student performance

(Herman, Gearhart and Acshbacher, 1996)

Issues

Different audiences (who is the portfolio for?)

Grading – who grades? (fairness) Learning and reflection get lost in

drive to measure competency (Herman and Winter 1994)

Issues

Gap between psychometrics and collaborative nature of the revision process (Song and August 2002,Hamp-Lyons and Condon 2000 )

Time needed for both teachers and students (Callahan 1995, Herman and Winters 1994)

Common problems

Purposes – clear to teacher and student? (Callahan 1995)Mismatch between assessment

criteria and goals of programme of study

Student anxiety and confusion (Butler 2006)

How do portfolios function best?

Research in portfolio assessment

A slim collection?

Research in portfolio assessment

Validity and reliability Fairness Impact

Validity

Vermont programCorrelation ranging from .47 to .58

between writing portfolio scores and direct writing assessments

Similar correlation between portfolio scores and multiple-choice maths test scores

(Koretz 1993 cited in Herman and Winter 1994)

Validity

Gearhart and others (1993) cited in Herman and Winter (1994) found: No relationship when comparing writing

portfolios with standard writing assessments Two thirds of students classified as

“masters” on the portfolio assessment would not have been so classified on the standard assessment.

Validity

Gearhart and others (1993) cited in Herman and Winter (1994) also found: When portfolios were scored in two different

ways (holistic and individual pieces scored) correlations were in the .6 range

Half the students who would have been classified as masters on the single portfolio score would not have been so classified when individual pieces were averaged

Which assessment best represents an enduring capability?

Validity

CUNY (Song and August 2002) 2 groups, 1 assessed by portfolio and writing

test, the other only a writing test Students twice as likely to move to the next

course when evaluated by portfolio At the end of the next course the pass rate

and grade distribution for the two groups were nearly identical

Reliability

Vermont interrater reliability of .28 to .60

Pittsburgh portfolio system ranged from .6 to .7

Herman et al. (1993)found correlations of .82 in an elementary school portfolio containing final drafts of writing

Reliability

Little work on other sources of portfolio reliabilityScore stability over timeStability across different rater groupsThe portfolio set in which a particular

portfolio is rated(Herman and Winter 1994, pg. 51)

Reliability

Heller, Sheingold and Myford (1998) think-aloud protocol on portfolio raters to see if they fit process model of portfolio rating

Found score validity was threatened when a major process was omitted or extraneous assessment criteria were applied

Reliability

Nystrand, Cohen and Dowling (1993) found reliability could be significantly improved if:Raters scored each task in response

to a prompt before moving to the next task and

Raters read several examples together to decide how they were to be rated

Impact

Herman and Winter (1994) based on self-reports from teachers and others implementing portfolios appears to have positive effects on instruction

Vermont principals affirmed that the portfolio assessment program had beneficial effects on curriculum and instruction

Impact

Aschbacher’s (1993) action research cited in Herman and Winter 1994 suggests teacher’s instructional practices and their attitudes towards students changed. Reported ways they thought about their own

teaching Two-thirds of teachers expected higher level

of performance from students

Impact

Hirvela and Sweetland (2005) used 2 case studies showing the 2 students did not strongly endorse the portfolios as used in 2 different courses. Seemed to need more explanations of what

portfolio approaches were meant to achieve Even with a 5% final course grade students

saw the portfolio as essentially summative in nature

Impact

Richardson (2000) study involved classroom observations teacher and student interviews and examination of student writing and teacher response. Found that students regard teacher

responses as directives. Were not prepared to make independent judgments largely because of the threat of grades

The future?

More technical quality Issue of fairness needs to be

addressed More research on impact A move away from psychometric

measurement?

Potential research

Equity of portfolio assessment Validity v reliability Nature of feedback Portfolios for employers

References

Abrami, P. C., & Barrett, H. (2005). Directions for Research and Development on Electronic Portfolios. Canadian Journal of Learning and Technology, 31(3).

Belanoff, P., & Dickson, M. (Eds.). (1991). Portfolios : process and product. Portsmouth, N.H.: Boynton/Cook Publishers.

Calfee, R. C., & Freedman, S. W. (1996). Classroom Writing portfolios:Old, New, Borrowed, Blue. In R. C. Calfee & P. Perfumo (Eds.), Writing Portfolios in the Classroom. MahWah, N. J.: L. Erlbaum Associates.

Callahan, S. (1995). Portfolio expectations: Possibilities and limits. Assessing writing, 2(2), 117-151.

Case, S. H. (1994). Will mandating portfolios undermine their value? Educational Leadership, 52(2), 46-47.

References

Desmet, C., & Cummings, R. (2004). Negotiating the Teaching-Assessment Cycle in Writing Programs with XML. Paper presented at the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2004 Washington DC.

Frederiksen, J. R., Sipusic, M., Sherin, M., & Wolfe, E. W. (1998). Video Portfolio Assessment: Creating a Framework for Viewing the Functions of Teaching. Educational Assessment, 5(4), 225-297.

Hamilton, S. J. (2006). A Principle-Based ePort Goes Public (and Almost Loses its Principles). In A. Jafari & C. Kaufman (Eds.), Handbook of Research on ePortfolios (pp. 434-446). Hershey: Idea Group Reference.

Hamp-Lyons, L. (1990). Second language writing: assessment issues. In B. Kroll (Ed.), Second language writing : research insights for the classroom. Cambridge: Cambridge University Press.

References

Hamp-Lyons, L., & Condon, W. (2000). Assessing the Portfolio: Principles for Practice Theory and Research. Cresskill, NJ: Hampton Press.

Heller, J. I., Sheingold, K., & Myford, C. M. (1998). Reasoning about Evidence in Portfolios: Cognitive Foundations for Valid and Reliable Assessment. Educational Assessment, 5(1), 5-40.

Herman, J. L., Gearhart, M., & Aschbacher, R. (1996). Writing portfolios in the classroom : policy and practice, promise and peril. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in the classroom : policy and practice, promise and peril

(pp. x, 374 p.). Mahwah, N.J.: L. Erlbaum Associates. Herman, J. L., Gearhart, M., & Baker, E. L. (1993). Assessing

writing portfolios: Issues in the validity and meaning of scores. Educational Assessment, 1(3), 201-224.

References

Herman, J. L., & Winters, L. (1994). Portfolio research: A slim collection. Educational Leadership, 52(2), 48-55.

Hirvela, A., & Sweetland, Y. L. (2005). Two case studies of L2 writers' experiences across learning-directed portfolio contexts. Assessing writing, 10, 192-213.

Holt, D., & Baker, N. W. (1991). Portfolios as a follow-up option in a proficiency-testing program. In P. Belanoff & M. Dickson (Eds.), Portfolios: process and product. Portsmouth NH: Boynton/Cook.

Jafari, A., & Kaufman, C. (Eds.). (2006). Handbook of Research on ePortfolios. Hershey: Idea Group Reference.

Murphy, S., & Camp, R. (1996). Moving towards systemic coherence: A discussion of conflicting perspectives on portfolio assessment. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in the classroom: policy and practice, promise and peril. Mahwah, N.J.: L. Erlbaum Associates.

References

Nystrand, M., Cohen, A. S., & Dowling, N. M. (1993). Addressing reliability problems in the portfolio assessment of college writing. Educational Assessment, 1(1), 53-70.

O'Brien, K. (2006). ePortfolios as Learning Construction Zones. In A. Jafari & C. Kaufman (Eds.), Handbook of research on ePortfolios (pp. 74-82). Hershey: Idea Group Reference.

Ostheimer, M. W., & White, E. M. (2005). Portfolio assessment in an American college. Assessing writing, 10, 61-73.

Pullman, G. (2002). Electronic Portfolios Revisited: The efolios Project. Computers and Composition, 19, 151-169.

Richardson, S. (2000). Students' conditioned response to teachers' response: Portfolio proponents, take note! Assessing Writing, 7, 117-141.

References

Song, B., & August, B. (2002). Using Portfolios to Assess the Writing of ESL Students: A Powerful Alternative? Journal of Second Language Writing, 11, 49-72.

Spalding, E., & Cummins, G. (1998). It was the best of times. It was a waste of time: University of Kentucky students' view of writing under KERA. Assessing Writing, 5(2), 167-199.

Stiggins, R. J. (1994). Student-centred classroom assessment. New York: Merrill.

Wagner, M., & Lamoureaux, E. (2006). Implementing an Outcome-Based Assessment ePortfolio. In A. Jafari & C. Kaufman (Eds.), Handbook of Research on ePortfolios (pp. 539-550). Hershey: Idea Group Reference.

Weigle, S. C. (2002). Assessing writing. Cambridge, UK Cambridge University Press.

References

Yancey, K. B. (1996). Portfolio as genre, rhetoric as reflection: Situating selves, literacies, and knowledge. WPA 19(3), 55-69.

Yao, Y., Thomas, M., Nickens, N., Downing, J. A., Burkett, R. S., & Lamson, S. (2008). Validity evidence of an electronic portfolio for preservice teachers. Educational Measurement: Issues and Practice, Spring 2008, 10-24.