e-portfolios and nationally standardized tests: validation or depreciation?

8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?

1/19


2/19

External Pressures for Assessment

Perceived Federal Interest: Spellings Report (2006)

Increased Attention to Higher Education in Ohio:University System of Ohio (Voluntary System of

Accountability, emphasizing the Collegiate Learning

Assessment test; subsidy tied to metrics-student

retention)

North Central Associations Higher LearningCommission: emphasizing student learning outcomes


3/19

Dual Pilot study at the University ofCincinnati:

To consider the applicability of learningportfolios (and especially e-portfolios)

alongside the CLA among a group of

first-year Honors students

http://tinyurl.com/uceportfolioproject


4/19

The Voluntary System of Accountability

(VSA)

Result of a partnership between the American Association of

State Colleges and University (AASCU) and the National

Association of State Universities and Land-Grant Colleges

(NASULGC), 2007Includes:

Student and Family Information

Student Experiences and Perceptions

Student Learning Outcomes (page 4)

Measure of Academic Proficiency and Progress (MAPP)

Collegiate Assessment of Academic Proficiency (CAAP)

Collegiate Learning Assessment (CLA)

Institution Specific Measures

http://www.voluntarysystem.org


5/19

The Collegiate Learning Assessment Developed by the Council for Aid to Education (CAE) in

2004 www.cae.org/cla

A standardized and nationally normed test that holistically

measures the institutional contributions (value added) tovery specific learning gains made by students

Direct assessment of student learning (claims face

validity)

critical thinking

analytical reasoning / problem solving written communication

Most frequently used of the VSA tests


6/19

AAC&U VALUE Metarubrics

AAC&U collaboration with faculty and student affairsprofessionals

Developed rubrics for 15 essential learning outcomes

Encouraged institutions to revise to accommodate individualneeds

Development is evolving among institutions

UC employed the Critical Thinking and Written Communicationmetarubrics:


7/19

Methodology Study cohort of 111 first-year Honors students

Average ACT is 31

Representing a range of UC colleges and programs

Briefed on experiment

Follow cohort over four-year time frame Administer the CLA as a course requirement

Half complete make an argument and critique an

argument, with the other half completing the problem-solving

performance task

Student post-test survey also administered at testing site During the same month, students completed assignments

designed to measure critical thinking and written communication

Graded by faculty using a variation of AAC&U meta-rubrics

,


8/19

Primary Research Questions

How do the CLA results compare to results produced via

VALUE-rubric scoring? In terms of

Measures of individual student learning

Measures of institutional-level value added

As a tool for guiding program improvement

As a response to accountability


9/19

Some Findings from the StudentSurvey

Usefulness of the test (waste of time or only mildlyuseful=50%,useful=40%, very useful=10%)

Describe the test experience (unpleasant=37%, indifferent48%, pleasant=15%)

Effort exerted (little or none=8%, some=40%, a great deal52%)

Incentives to take the test seriously (would do it for free=8%,

self-improvement=39%, financial incentive required=43%) Experiences that prepared students to do well on the test:

prep courses for ACT and SAT


10/19

Side-By-Side ComparisonCLA VALUE Metarubrics

Grading Human and computer Human

Inter-rater Reliability Systematically achieved Achieved high IRR

through norming

Test Method Timed, open-ended

response

Open ended, over time

Test Incentive Voluntary Homework

Relevance Real world application Academically connected


11/19

Side-By-Side Comparison

CLA VALUE Metarubrics

Scoring Holistically Trait-by-trait

Test Time 90 minutes 10 Weeks

Turnaround time 8 months As needed

Motivation Add on Embedded in context

with course


12/19

Institutional CLA Data

Mean CLA

Score

Unadjuste

d % Rank

Deviation

Score

Adjusted

% Rank

Total CLA Score 1352 98 1.4 95

Performance Task 1338 98 1.7 96

Analytic Writing Task 1366 98 0.8 79

Make-an-Argument 1348 98 0.3 63

Critique-an-Argument 1383 98 1.4 92


13/19

Metarubric Assignments

N Avg Std Dev Avg team

rater

correlation

Overall

Correlation

Written

Communication

109 22.7 3.47 0.717 0.696

Critical Thinking 55 18.98 3.91 0.772 0.824


14/19

Correlation


15/19

Interpretation and Explanation The lack of statistically significant correlation between the CLA

task scores and the e-portfolio assignment scores indicates a

difference in evaluating student work.

Although both processes measure the application of intellectual

and practical skills, each is evaluating student outcomes atdiffering levels of context.

The level instruments provide very different levels of

aggregation. While the VALUE rubrics provide trait-level analysis

of students, the CLA holistic score considers student

performance.

Differences in the administration of the two instruments must be

considered ( context, incentives, timing, the ability to draft).


16/19

Conclusions

The CLA is useful for INSTITUTIONAL assessment

National benchmarking & comparisons for CLA are likely to

be more valid than the more nuanced results of VALUE.

The CLA does not appear to provide data that is particularly

useful for STUDENT diagnostic feedback.

Program assessment will require significantly greatersampling than required by VSA.


17/19

Conclusions The Value Rubrics are useful for providing a

consistent framework for the evaluation of student

artifacts. They can help transform grading into the

assessment of specific Student Learning Outcomes Well received by faculty, appropriate assessment for

accreditation and program improvement

Useful in providing some comparability across faculty but

only of limited value for institutional-level reporting because

the rubrics are designed to be modified to meet specificneeds

Provides more timely and detailed feedback


18/19

Bottom Line Institutions should employ a complex and varied assessment

strategy to understand achievement of student outcomes

Understand achievement in relation to student outcomes

Demonstrate the value of the educational experience

Assure that students are well-prepared for lifelong learning

and intellectual inquiry Provide a framework for program feedback and improvement


19/19

Recent Analyses Trudy Banta, The Search for a Perfect Test Continues, in Assessment

Update (November-December 2007)

Joan Hawthorne, Accountability & Comparability: Whats Wrong with the

VSA Approach? in Liberal Education (Spring 2008)

Paul Basken, Electronic Portfolios May Answer Calls for MoreAccountability, in The Chronicle of Higher Education (April 17, 2008)

Trudy W. Banta, Assessment for Accountability: A Cautionary Tale,

presented at The Ohio State University on June 19, 2008

Klein, Liu, Sconing, 2009. Test Validity Study (TVS) Report .

http://www.voluntarysystem.org/docs/reports/TVSReport_Final.pdf

Schulenberger, D., Keller, C. (2009) Interpretation of Findings from the TVS

for the VSA.

http://www.voluntarysystem.org/docs/reports/VSAabstract_TVS.pdf

e-portfolios and nationally standardized tests: validation or depreciation?

Documents