Implementing theAAC&U Value Rubrics

in SakaiSean Keesler and Janice Smith

Three CanoesJune 15, 2010

Presenters• Sean Keesler, Three Canoes• Janice Smith, Ph.D., Three Canoes

http://threecanoes.com

Three Canoes MissionThree Canoes Consulting wants your institution to become self-sufficient in using Sakai and the Open Source Portfolio (OSP) for teaching, learning, and assessment. We offer on-site and virtual services for assessment planning, ePortfolio design, OSP customization, and documentation and training for your use of Sakai. We are fellow educators with years of experience working with higher ed and K-12 institutions and have been a part of the Sakai and OSP communities since their inception.

Session Outline

• The AAC&U VALUE Rubrics• Implementing the AAC&U VALUE Rubrics in Sakai• Three Canoes OSP Contribution• Technical Specifications• Discussion

The AAC&U VALUE Rubrics

AAC&U • Association of American Colleges and Universities

VALUE• Valid Assessment of Learning in Undergraduate

Education

Association of American Colleges and Universities

VALUE Project

• Part of AAC&U’s Liberal Education and America’s Promise (LEAP) initiative

• Project Dates: May 2007 through April 2010• Staffed by Sakai’s own Wende Morgaine, January

2008 to January 2010

Association of American Colleges and Universities

VALUE Project

• Project involved teams of faculty and other academic and student affairs professionals.

• Iterative process took place over eighteen months.• Teams gathered, analyzed, synthesized, and drafted

institutional level rubrics.• Rubrics address 15 essential learning outcomes.• Each VALUE rubric contains the most common criteria for

judging student work in for the outcome. • The VALUE rubrics reflect U. S. faculty expectations for

essential learning regardless of institutional type, mission, size or location.

VALUES Project Assumptions• To achieve a high-quality education for all students, valid assessment

data are needed to guide planning, teaching, and improvement; • Colleges and universities seek to foster and assess numerous essential

learning outcomes beyond those addressed by currently availablestandardized tests;

• Learning develops over time and should become more complex andsophisticated as students move through their curricular and cocurriculareducational pathways toward a degree;

• Good practice in assessment requires multiple assessments, over time;well-planned electronic portfolios provide opportunities to collect datafrom multiple assessments across a broad range of learning outcomeswhile guiding student learning and building self-assessment capabilities;

• ePortfolios and assessment of work in them can inform programs andinstitutions on progress in achieving expected goals.

From AAC&U VALUE Website: http://www.aacu.org/value/index.cfm

AAC&U Essential Learning OutcomesIntellectual and Practical Skills• Inquiry and analysis • Critical and creative thinking • Written and oral communication • Quantitative literacy • Information literacy • Teamwork and problem solving Personal and Social Responsibility• Civic knowledge and engagement—local and global • Intercultural knowledge and competence • Ethical reasoning and action • Foundations and skills for lifelong learning Integrative and Applied Learning • Synthesis and advanced accomplishment across general and specialized studies

Inquiry and Analysis VALUE Rubric

Creative Thinking Rubric

Implementing the AAC&U VALUE Rubrics In Sakai

• Sakai doesn’t come with a default portfolio experience.

• There is well researched prior art.• You can springboard your implementation.• Use the AAC&U VALUE rubrics to start the

conversation at your institution about portfolio purpose and assessment.

Possible Sakai Portfolio WorkflowStudents • Choose associate learning artifacts with matrix cells according to essential

learning outcomes that the artifacts address.• Use a reflection form to evaluate themselves according to the relevant VALUE

Rubric.Faculty• Use an evaluation form containing the same VALUE Rubric to evaluate student

artifacts in each matrix cell.Students• Compare their self-reflection with evaluation by their faculty• Use a portfolio template to share the content of the matrix with others as an

indication of their learning

1211th Sakai Conference - June 15-17, 2010

Inquiry and Analysis Evaluation Form

Completed Evaluation Form

Student View of Evaluation Form

Calculation of Average Rating

Values Matrix

VALUE Portfolio Template Ideas

• Create a “matrix”portfolio to harvest data from the VALUES matrix

• Ask faculty evaluators to sign off on each VALUE indicating student mastery

• VALUEs that have been mastered would appear on the portfolio with checkmarks

• Provide themes for students to select or alternatively encourage students to creatively illustrate their portfolios with images and captions

Live Demo of VALUE Rubrics

Three Canoes OSP Contribution

• Matrix and 15 VALUE Rubric forms• Go to Open Ed Practices• http://openedpractices.org

Search for Three Canoes

Or• http://openedpractices.org/resource/aacu-essential-lear

ning-outcomes-matrix-and-value-rubrics• http://openedpractices.org/practice/assessing-student-le

arning-using-aacu-value-rubrics

Technical Specifications

• Rubric renderers• Four non-zero choices• Criteria: Ennumerated string values• Keys• Weighted averages• Standard Fields

Rubric Renderer

• “Standard” form renderers don’t produce a form that looks like a rubric.

• They are useful starters for developers to modify

• Your own rubrics can be implemented as Sakai forms with a similar renderer.

• The renderer has some limitations and assumptions incorporated into the design.

2211th Sakai Conference - June 15-17, 2010

Four Non-Zero Choices

• This renderer• Creates a table with four columns• Can be modified

• If an evaluator doesn’t choose one of the four choices, the score is a ZERO for that criteria.

• You can make the renderer capable of handling any size table of values.

11th Sakai Conference - June 15-17, 2010 23

Criteria - Enumerated String Values

• This renderer expects each field with enumerated string values to be a row of the rubric.

• There may be a need for other “non-rubric” fields on the form.

11th Sakai Conference - June 15-17, 2010 24

Keys

• Each criteria is assigned a key (sakai.key).• Keys:• Match criteria rows in the rubric table with the

text input for that criteria.• Match a criteria to it’s rubric weight.• Determine the sort order of the table rows and

text input fields for each criteria

11th Sakai Conference - June 15-17, 2010 25

Weighted Averages

• The result of completing the rubric form is a weighted average of the criteria scores.

• Setting the sakai.rubricWeight for each criteria element will allow different emphasis on each criteria.

• Inter-rater reliability can be achieved by automatically calculating the final rating.

11th Sakai Conference - June 15-17, 2010 26

Standard Field Names

• The final calculated score is called “rating.”• The evaluator comments are called

“comments.”

11th Sakai Conference - June 15-17, 2010 27

Our Questions for You

• How would you use the AAC&U VALUE Rubrics in your institution’s instance of Sakai?

• Do you have ideas for improving the rubric forms or the VALUES matrix?

• What characteristics should the VALUEs portfolio template have?

• Do you have any ideas about how to move the VALUE Rubrics toward portability across systems?