Welcome!

WILU May 2014

Rubric Assessment (à la RAILS) for

Your Library’s Instruction Program

Claire Holmes Carroll Wilkinsoncholmes@towson.edu cwilkins@wvu.edu

Agenda for Today :

• Background on Assessment,

RAILS & Rubrics

• Norming & Rating Sessions

• Reflections & Questions

Assessment…• Knowing what you are doing

• Knowing why you are doing it

• Knowing what students are learning as a result

• Changing because of the information

(Debra Gilchrist, Dean of Libraries and Institutional Effectiveness, Pierce College, from Assessment: Demonstrating the Educational Value of the Academic Library, ACRL Assessment Immersion, 2011)

Identify learning

outcomes

Create and enact

learning activities

Gather data to

check for learning

Interpret data

Enact decisions

to increase learning

Information Literacy

Instruction Assessment

Cycle (ILIAC)

Oakleaf, Megan. "The Information Literacy Instruction Assessment Cycle: A Guide for Increasing Student Learning and Improving Librarian Instructional Skills." Journal of Documentation. 65.4. 2009.

The Institute of Museum and Library Services is the primary source of federal support for the nation’s 123,000 libraries and 17,500 museums. The Institute's mission is to create strong libraries and museums that connect people to information and ideas.

Megan Oakleaf, founder of all things RAILS. (more @ www.railsontrack.info)

www.railsontrack.info

RAILS Project Purposes

• Investigated an analytic rubric approach to IL assessment in higher education

• Developed a suite of IL rubrics

• Investigated rubric reliability & validity

• Developed training materials for training/ norming/ scoring

• Explored indicators of rater expertise

RAILS Participants’ Purposes• Professional development opportunity

• Develop rubrics for use on campuses

• Identify opportunities for assessment within the curriculum

• Gain experience in norming

• Assess student work to learn about their information literacy skills

New IL framework…

Our shift to examining and assessing student learning through the lenses of meta-literacy & threshold concepts will require time and consideration…

Framework for Information Literacy in Higher Education

• A growing and developing document that introduces a new definition for IL and provides an explanation of why and how IL has changed since 2000.

• May be adopted by the ACRL Board in August, 2014 after further open discussion and final revision.

IL in 2014

• Meta-literacyexpansion of the scope of traditional information skills

• Threshold conceptsthe core foundational ideas that, once grasped by the learner, create new perspectives

Teaching Values: A new approach?

DispositionsValues behind the development of information literate capabilities

Persistence and adaptability

Critical thinking Value of intellectualcuriosity

Understanding by Design1. What do you want students to learn?

(outcome)

2. How will you know that they have learned it? (assessment)

3. What activities/assignments help them learn, offer evidence of that learning and also provide assessment data?

(teaching method & assessment)

(Wiggins & McTighe, 2006)

Performance/Integrated AssessmentStudents reveal their learning when they are provided with:

complex,

authentic

LEARNING ACTIVITIES

to explain, interpret, apply, shift perspective, empathize

and self-assess.

What we assess.What they learn.

(Megan Oakleaf, Assessment: Demonstrating the Educational Value of the Academic Library, ACRL Assessment Immersion, 2011)

5 Questions for Assessment Design:1. Outcome What do you want the student to be able to

do?

2. IL Curriculum What does the student need to know in order to do this well?

3. Pedagogy What type of instruction will best enable the learning?

4. Assessment How will the student demonstrate the learning?

5. Criteria for evaluation

How will you know the student has done well?

(Lisa Hinchcliffe, Student Learning Assessment Cycle. ACRL Assessment Immersion, 2011)

Evidence of “authentic” student learning:

For instance, the research worksheet in your packet that asks students to break down and practice sequential steps in the search process.

Brainstorm… What other possible examples of evidence of student learning do we collect? What could we collect?

Brainstorm ideas…

Evidence: Possible examples of authentic student learning…• Research journal• Reflective writing• “think aloud”• Self or peer evaluation• Works cited page• Annotated bibliography• Posters• Multimedia presentations• Speeches• Open-ended question

responses

• Group projects• Performances• Portfolios• Library assignments• Worksheets• Concept maps• Citation maps• Tutorial responses• Blogs• Wikis• Lab reports

1. What are our expectations of students completing this assignment?

2. What specific learning outcomes do we want to see reflected in the completed assignment?

3. What evidence can we find that will demonstrate learning success?

4. Are there clear differences between the various levels of student work in this assignment?

Why create and use a rubric?

Norm!

• 2 dimensions1. criteria

2. levels of performance

• grid or table format

• judges quality

• translates unwieldy data into accessible information

(Image: thefirstgradediaries.blogspot.com)

Norming is Crucial

• “I know it when I see it” does not mean “I can articulate it.”

• Norming is critical for establishing shared understanding of the rubric and achieving greater inter-rater reliability.

Reasons why we norm…

Rubrics are powerful tools that provide structure and consistency to assessment. If more than one rater uses a rubric, norming is crucial. Norming facilitates agreement, identifies misunderstandings, and minimizes measurement errors.

Holmes, C. & Oakleaf, M. (2013). The Official (and Unofficial) Rules for Norming Rubrics Successfully. Journal of Academic Librarianship, 39(6), 599-602.

Inter-rater Reliability

• From the Cambridge Dictionary of Sociology“Measurement gives rise to consideration of the issues of reliability and validity. Reliability refers to the ability to

repeat the results of a measurement accurately (common forms include inter-rater reliability; test-retest reliability; and measures of internal consistency, including split-half and coefficient alpha). 

 

 • From Sage’s Encyclopedia of Survey Research Methods:

“The concept of inter-rater reliability essentially refers to the relative consistency of the judgments that are made of the same stimulus by two or more raters…. An

important factor that affects the reliability of ratings made by a group of raters is the quantity and the quality of the training they receive.)

SAMPLE RAILS RUBRIC (green handout in your packet)

 Performance Level 3

Student:Performance Level 2

Student:Performance Level 1

Student:Performance Level 0

Student:

 1.

Determines Key Concepts

Determines multiple key concepts that reflect the research topic/thesis statement accurately. 

Determines some concepts that reflect the research topic/thesis statement, but concept breakdown is incomplete or repetitive.

Determines concepts that reflect the research topic/thesis statement inaccurately.

Does not determine any concepts that reflect the research question/thesis statement. 

2. Identifies synonyms

and related terms Identifies relevant synonyms and/or related terms that match key concepts.

Attempts synonym (or related term) use, but synonym list is incomplete or not fully relevant to key concepts.

Identifies synonyms that inaccurately reflect the key concepts.

Does not identify synonyms.

3. Constructs a search

strategy using relevant operators

Constructs a search strategy using an appropriate combination of relevant operators (for example: and, or, not) correctly.

Constructs a search strategy using operator(s), but uses operators in an incomplete or limited way.

Constructs a search strategy using operators incorrectly. 

Does not use operators.

4. Uses evaluative

criteria to select source(s)

Uses evaluative criteria to provide in-depth explanation of rationale for source selected.

Uses evaluative criteria to provide a limited/superficial explanation of rationale for source selected.

Attempts to use evaluative criteria, but does so inaccurately or incorrectly.

Does not use evaluative criteria.

5. Uses Citations

Uses an appropriate standard citation style consistently and correctly. 

Uses an appropriate standard citation style consistently (bibliographic elements intact), but with minimal format and/or punctuation errors.  

Attempts an appropriate standard citation style, but does not include all bibliographic elements consistently or correctly.

Does not include common citation elements or does not include citations.

Rubric Norming Process1. Facilitator thinks aloud through scoring several examples.

2. Raters independently score a set of examples that reflects the range of artifacts

3. Raters come together to review their scores to identify patterns of consistent and inconsistent scores.

4. Discuss and then reconcile inconsistent scores.

5. Repeat the process of independent scoring on a new set of examples.

6. Again, raters come together to review their scores to identify patterns of consistent and inconsistent scores.

7. Discuss and then reconcile inconsistent scores. This process is repeated until raters reach consensus about applying the scoring rubric. Ordinarily, two to three of these sessions calibrate raters’ responses.

Workshop Norming Practice

Round 1• For first student work sample, Claire will

“norm aloud.”

• Participants will rate 1 work sample individually.

• Group discussion: Can we reach consensus for what constitutes evidence for each performance level?

Norm, norm, norm!

Keep in mind…

• An info lit skills rubric does not score discipline content; it scores information literacy skills.

• You can only score what you can see.

Norming: Round 2

• Participants will rate 2-3 more work samples individually.

• Group discussion: Are we closer to consensus?

• Do we establish rating ground rules?

• Does the rubric need to be modified?

Access Needed Info: Original Rubric

Advanced Developing BeginningDetermine Key Concepts Student:

Determines keywords/subject /subheadings that describe the research question/thesis fully including relevant variants

Student:

Determines keywords/subject /subheadings that describe the research question/thesis partially

Student:

Does not determine keywords/subject /subheadings that describe the research question/thesis

Access the Needed Information

Student:

Accesses information using effective, well-designed search strategies.

Demonstrates persistence and ability to refine search

Student:

Accesses information using simple search strategies

Student:

Accesses information randomly

Retrieves relevant information (Determine the extent of information needed)

Student:

Retrieves information sources that fit search parameters, relates to concepts or answers research question

Student:

Retrieves information sources that partially fit search parameters, relates to concepts or answers research question

Student:

Does not retrieve information that fits search parameters, relates to concepts or answers research question

What do norming revisions look like? Advanced Developing BeginningDetermine Key Concepts

Student: Determines keywords/subject

/subheadings that fully describe the research question/thesis fully including relevant variants

Student: Determines

keywords/subject /subheadings that partially describe the research question/thesis partially

Student: Does not determine

keywords/subject /subheadings that describe the research question/thesis

(pharm disease state & drug; nursing multi-faceted, omit shortage)

Access the Needed Information

Student: Accesses information using effective, a

logical progression of advanced search strategies such as limits, Boolean searches, or combined searches

Demonstrates persistence and ability to refine search

Student: Accesses information

using simple search strategies

Accesses information using advanced search strategies, such as limits, Boolean searches, or combined searches

Student: Accesses information

randomly Accesses information

using only simple search strategies

Retrieves relevant information (Determine the extent of information needed)

Student: Retrieves information sources that fully fit

search parameters and relate to concepts or answer research question

Student: Retrieves information

sources that partially fit search parameters or relate to concepts answer research question

Student: Does not retrieve

information sources that either fit search parameters or relates to concepts or answer research question

Norming/Rating Discussion

• How do we achieve consensus?

• What was challenging?

Rubrics – Benefits

Learning

• Articulate and communicate agreed upon learning goals

• Provide direct feedback to learners

• Facilitate self-evaluation

• Focus on learning standards

More benefits of a (normed) rubric…Data• Facilitate consistent, accurate, unbiased scoring

• Deliver data that is easy to understand, defend, and convey

• Offer detailed descriptions necessary for informed decision-making

• Can be used over time or across multiple programs

Other• Are inexpensive ($) to design & implement

Rubrics – Limitations• Possible design flaws that impact data quality

• Require significant time for development

• Sometimes fail to balance between holistic and analytic focus

• May fail to balance between generalized wording and detailed description

• Can lack differentiation between performance levels

• Start with established partners, existing librarian/disciplinary faculty collaborations

• Evaluate a skill relevant to many campus partners (ex. use information legally and ethically)

• Include those who can help disseminate results and promote IL assessment efforts across campus

• Meet with stakeholders regularly to review and improve assignment and rubric.

RAILS Lessons

• Explicit, detailed performance descriptions are crucial to achieve

inter-rater reliability.

• Raters appear to be more confident about their ratings when student artifacts under analysis are concrete, focused, and shorter in length.

• The best raters “believe in” outcomes, value constructed consensus (or “disagree and commit”), negotiate meaning across disciplines, develop shared vocabulary, etc.

More RAILS Lessons

Identify learning

outcomes

Create and enact

learning activities

Gather data to

check for learning

Interpret data

Enact decisions

to increase learning

Information Literacy

Instruction Assessment

Cycle (ILIAC)

Oakleaf, Megan. "The Information Literacy Instruction Assessment Cycle: A Guide for Increasing Student Learning and Improving Librarian Instructional Skills." Journal of Documentation. 65.4. 2009.

Using Assessment Results…

Improvements within the library• Instruction • Relationships• Impact on learning

Growth within the field • Conferences• Publications

ReferencesArter, J. (2000). Rubrics, scoring guides, and performance criteria:

Classroom tools for assessing and improving student learning. Retrieved from

http://eric.ed.gov/?id=ED446100

Bresciani, M., Zelna, C. & Anderson, J. (2004). Assessing student learning

and development: A handbook for practitioners. Washington, DC:

NASPA-Student Affairs Administrators in Higher Education.

Wiggins, G. P., & McTighe, J. (2006). Understanding by design. Upper Saddle

River, NJ: Pearson Education, Inc., 2006.

Wiggins, G. P. (1998). Educative assessment: Designing assessments to inform

and improve student performance. San Francisco, CA: Jossey-Bass.

Selected Readings:

 

Diller, K. R., & Phelps, S. F. (2008). Learning outcomes, portfolios, and rubrics, oh my! Authentic assessment of an information literacy program. Portal: Libraries and the Academy, 8 (1), 75-89.

 

Fagerheim, B. A., & Shrode, F. G. (2009). Information literacy rubrics within the disciplines. Communications in Information Literacy, 3(2), 158-170.

 

Holmes, C. & Oakleaf, M. (2013). The Official (and Unofficial) Rules for Norming Rubrics Successfully. Journal of Academic Librarianship, 39(6), 599-602.

Knight, L. A. (2006). Using rubrics to assess information literacy. Reference Services Review, 34(1), 43-55.

 

Oakleaf, M. (2007). Using rubrics to collect evidence for decision-making: What do librarians need to learn? Evidence Based Library and Information Practice, 2(3), 27-42.

 

Oakleaf, M. (2009). The information literacy instruction assessment cycle: A guide for increasing student learning and improving librarian instructional skills.  Journal of Documentation, 65(4), 539-560.

 

Oakleaf, M., Millet, M., & Kraus, L. (2011). All together now: getting faculty, administrators, and staff engaged in information literacy assessment.  Portal: Libraries and the Academy, 11(3), 831- 852.

 

Stevens, D. D., & Levi, A. (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Sterling, VA: Stylus Publishing.

WILU: May 2014

Rubric Assessment (à la RAILS)

for Your Library’s Instruction Program

Claire Holmes Carroll Wilkinson

Towson University West Virginia Universitycholmes@towson.edu cwilkins@wvu.edu

SlideShare URL:

Thank you!