building a culture of assessment: what i've learned from research, trial, and (a lot of) error

BUILDING A CULTURE OF

ASSESSMENT WHAT I'VE LEARNED

FROM RESEARCH, TRIAL, AND (A LOT OF) ERROR

Meredith Farkas Portland Community College

ABOUT ME

➤ Faculty Librarian at Portland Community College (OR)

➤ Previously:

➤ Head of Instructional Services, Portland State University

➤ Head of Instructional Initiatives, Norwich University

➤ Significant research and publishing on what it takes to build an assessment culture

➤ Former psychotherapist interested in what it takes to really change culture

“A culture of assessment is an organizational environment in which decisions are based on facts, research, and analysis, and where services are planned and delivered in ways that maximize positive outcomes and impacts for customers and stakeholders.

-Amos Lakos, Shelley Phipps, and Betsy WilsonLakos, A. & Phipps, S. E. (2004). "Creating a culture of assessment: A catalyst for organizational change." portal: Libraries and the Academy, 4(3), 345-361.

MY DEFINITION

➤ Assessment is the norm - a regular part of practice

➤ User-focused culture

➤ Learning and curiosity driven

➤ Assessment is done for improvement

➤ Changes and decisions are made based on what is learned

➤ Planning for new services always includes planning for assessment

ASSESSMENT AT MANY INSTITUTIONS

➤ Focused on doing what is needed to satisfy accreditors or administrators

➤ Coercive and top-down

➤ Little support

➤ Coalition of the willing

➤ Data is gathered, but rarely used

➤ Not seen as an integral part of teaching and learning

➤ Often a lot of cynicism around assessment

MY EXPERIENCE: ADVENTURES WITH

COORDINATOR SYNDROME

“The word ‘Coordinator’ in a position title. Beware it, especially in combination with a new or non-traditional job niche…

The problem with these jobs is that as often as not, there’s nothing actually to coordinate. No budget. No dedicated staff. No established service. ‘Coordinate’ all too often means ‘try to establish a beachhead by begging your new colleagues to vouchsafe you a few minutes of their time now and then, knowing that their supervisors won’t tell them to and you have no authority whatever to demand anything of them.’

Oh, but the buck will stop with you, Coordinator, on all matters regarding your service. Be aware, also, that because what you’re doing is new, you must hit a home run with it, quickly; slow and steady gains will not do. If you don’t, two damaging assumptions arise: that the service is a waste of effort, and that you’re a lousy librarian. You’re a coordinator now; congratulations!

-Gavia Libraria, “The C-word"

YOU SAY YOU NEED SUPPORT? I’VE GOT YOUR SUPPORT RIGHT HERE.

➤ Farkas, Meredith G. "Building and sustaining a culture of assessment: best practices for change leadership." Reference services review 41.1 (2013): 13-31.

➤ Farkas, Meredith G., and Lisa J. Hinchliffe. "Library faculty and instructional assessment: Creating a culture of assessment through the high performance programming model of organizational transformation." Collaborative Librarianship 5.3 (2013): 177-188.

➤ Farkas, Meredith Gorran, Lisa Janicke Hinchliffe, and Amy Harris Houk. "Bridges and barriers: factors influencing a culture of assessment in academic libraries." College & Research Libraries (2014).

➤ Farkas, Meredith. "Accountability vs. Improvement: Seeking Balance in the Value of Academic Libraries Initiative." OLA Quarterly 19.1 (2014): 4-7.

“For a scholarship of assessment to thrive, we must align faculty culture, institutional structures, and leadership for change.

The importance of this point cannot be overstated. A meaningful assessment program is more than just a new activity to be undertaken, it is a change in how we think about what we do in higher education.

-Don Haviland “Leading Assessment, from Faculty Resistance to Faculty Engagement.”

THESE STUDIES SUGGEST SPECIFIC FACILITATING/HINDERING FACTORS

1. Oakleaf, M. & Hinchliffe, L. J. (2008). Assessment cycle or circular file: Do academic librarians use information literacy assessment data? In Proceedings of the 2008 Library Assessment Conference: Building Effective, Sustainable, Practical Assessment.

2. Ndoye, A. & Parker, M. A. (2010). "Creating and sustaining a culture of assessment." Planning for Higher Education, 38(2), 28-39.

3. Hiller, S., Kyrillidou, M., & Self, J. (2008). "When the evidence is not enough: Organizational factors that influence effective and successful library assessment." Performance Measures and Metrics, 9(3), 223-230.

4. Wright, S. & White, L. (2007). SPEC Kit 303, Library Assessment. Washington, D.C.: Association of Research Libraries.

FACILITATING FACTORS SUGGESTED IN THE LITERATURE

➤ Assessment is a library priority (4)

➤ Commitment from library leadership (1, 3, 4)

➤ Commitment from faculty/staff (2, 3, 4)

➤ Organizational culture (3, 4)

➤ Staff with needed expertise and time (1, 3, 4)

➤ Staff development (4)

➤ Evidence-based decision-making (1, 2, 3, 4)

DEMONSTRATING THAT ASSESSMENT IS A LIBRARY PRIORITY

➤ Shared understanding of assessment exists within the organization

➤ Clear expectations or concrete goals for assessment

➤ Building an assessment plan

COMMITMENT FROM ADMINISTRATION

➤ Articulates the value of assessment

➤ Assessment is built into ALL planning

➤ Assessment is built into reward structures

➤ Supports library faculty/staff engaging in assessment

➤ Uses assessment data to inform decisions

EXPERTISE AND TIME

➤ Staff need dedicated time to learn, try, fail, and learn more

➤ Find ways to capitalize on expertise outside of the library

STAFF DEVELOPMENT➤ Don’t just send one person to

a conference

➤ Opportunities for staff to learn together

➤ Build a culture of reflective practice and learning communities

WHY DON’T LIBRARIANS USE ASSESSMENT RESULTS?

➤ Too busy with other work ➤ Won’t be rewarded for assessment

work ➤ Lack of expertise ➤ Lack of centralized support within

the library ➤ Not valued by their library

Oakleaf and Hinchliffe. (2008). “Assessment Cycle or Circular File: Do Academic Librarians Use Information Literacy Assessment Data?” Proceedings of the 2008 Library Assessment Conference.

FACILITATING FACTORS - EXPANDED

1. Clear expectations for assessment in the library

2. Assessment is a priority of administration

3. Library has adopted learning outcomes

4. Library has an assessment plan

5. Library has a shared understanding of the purpose of assessment

6. Library faculty/staff are supported in doing assessment

7. Education and training about assessment are supported

8. Library has assessment expertise in-house

9. Library has systems/technologies to support assessment

10. Library leadership offers explicit support to get staff doing assessment

11. Data is available to interested parties

12. Leadership uses assessment data in decision-making

13. Librarians use results to improve practice

14. Library culture is user-focused

ULTIMATELY, THIS IS ABOUT CHANGING

CULTURE

JOHN KOTTER’S LEADING CHANGE: THE ORGANIZATIONAL CHANGE BIBLE

CREATE A SENSE OF URGENCY

➤ Urgency must be driven by opportunities, not by fear

➤ Appeal to the things that your audience cares about (student learning, being better instructors, etc.)

➤ Sometimes urgency comes from outside (mandates from administrators, outside agencies, etc.)

FORM A GUIDING COALITION (NOT JUST A COORDINATOR)

➤ Form a group to lead and support assessment efforts

➤ Membership based on skills, interest, and politics

➤ They will need admin support

➤ Examples:

➤ Northwestern University libraryassessment.org/bm~doc/4macayeallightingtalk.pdf

➤ Portland Community College www.pcc.edu/edserv/learning-assessment/LACAboutUs.html

http://libraryassessment.org/bm~doc/4macayeallightingtalk.pdf

http://www.pcc.edu/edserv/learning-assessment/LACAboutUs.html

MacAyeal & Ryan. (2014). Support System: Establishing, Sustaining,and Growing a Frameworkfor Assessment. Library Assessment Conference.

PCC LEARNING ASSESSMENT COUNCIL

CREATE AND COMMUNICATE A VISION

➤ Common understanding of the purpose of our assessment program

➤ Develop an assessment plan

OR

➤ Develop concrete expectations for what progress in assessment looks like

EMPOWER OTHERS TO ACT ON THE VISION

➤ Remove barriers

➤ Time

➤ Lack of knowledge/education on assessment

➤ Fear of failure

➤ Incentivize assessment

➤ Grants, rewards, recognition

➤ Give staff/faculty the freedom to determine what and how they will assess

➤ Demonstrate that assessment results will be used

CREATE SHORT-TERM WINS (OR “THE PERFECT IS THE ENEMY OF THE GOOD”)

➤ Start small and simple

➤ Assessments that are easy to do but yield valuable learning

➤ What are we curious about?

➤ Debrief together and learn from each experience

➤ Talk about changes made based on assessment results

CONSOLIDATE IMPROVEMENTS AND INSTITUTIONALIZE NEW APPROACHES

➤ Develop formal mechanisms for reporting and sharing assessment results

➤ Build mechanisms for discussing assessment results within the library

➤ Demonstrate evidence-based decision-making

➤ Build assessment into planning

➤ Build assessment into reward structures (tenure?)

THERE ARE LIMITATIONS WITH THE LITERATURE WE RELY ON TO MAKE CLAIMS ABOUT BUILDING

AN ASSESSMENT CULTURE

1. Oakleaf, M. & Hinchliffe, L. J. (2008). Assessment cycle or circular file: Do academic librarians use information literacy assessment data? In Proceedings of the 2008 Library Assessment Conference: Building Effective, Sustainable, Practical Assessment.

2. Ndoye, A. & Parker, M. A. (2010). "Creating and sustaining a culture of assessment." Planning for Higher Education, 38(2), 28-39.

3. Hiller, S., Kyrillidou, M., & Self, J. (2008). "When the evidence is not enough: Organizational factors that influence effective and successful library assessment." Performance Measures and Metrics, 9(3), 223-230.

4. Wright, S. & White, L. (2007). SPEC Kit 303, Library Assessment. Washington, D.C.: Association of Research Libraries.

Survey sent to an infolit listserv

Based on their experience with research libraries that want to build an assessment culture

Sent to an assessment listserv

Survey of ARLs only

Our goal:

A systematic survey of culture of assessment and investigation of factors that facilitate or hinder libraries.

A survey of libraries, not librarians

THE SURVEY

➤60 Question Survey ➤10 descriptive questions about the institution or the

individual responding for the institution ➤47 yes/no questions ➤inquiry about presence/absence of 16 facilitating factors

named in other publications ➤does the presence/absence facilitate or hinder culture of

assessment ➤3 open-ended questions about what most facilitates or hinders

creation of a culture of assessment ➤Respondents presented with different questions depending on

their answers to presence/absence of facilitating factors (total questions for each respondent = 41 or 42)

FACILITATING FACTORS

1. Campus-wide assessment initiative

2. Library involvement in campus-wide assessment initiative

3. Clear expectations for assessment in the library

4. Assessment is a priority of administration

5. Library has adopted learning outcomes

6. Library has an assessment plan

7. Library has a shared understanding of the purpose of assessment

8. Library faculty/staff are supported in doing assessment

9. Education and training about assessment are supported

10. Library has assessment expertise in-house

11. Library has systems/technologies to support assessment

12. Library leadership offers explicit support to get staff doing assessment

13. Data is available to interested parties

14. Leadership uses assessment data in decision-making

15. Librarians use results to improve practice

16. Library culture is user-focused

Stage 1: Winter 2013

Libraries at all BA/MA/PhD-granting institutions

Received a 42% response rate

Results presented at ACRL 2013 and published in College & Research Libraries

Farkas, Meredith Gorran, Lisa Janicke Hinchliffe, and Amy Harris Houk. "Bridges and Barriers: Factors Influencing a Culture of Assessment in Academic Libraries." College & Research Libraries 76.2 (2015): 150-169.

BUT COMMUNITY COLLEGES ARE DIFFERENT

Stage 2: Spring 2014

Libraries at all U.S. public community colleges

Received a 32% response rate (n=292)

WOULD YOU DESCRIBE YOUR LIBRARY AS HAVING A CULTURE OF ASSESSMENT, WHERE ASSESSMENT IS A REGULAR PART OF YOUR INSTITUTIONAL PRACTICE?

THIS IS HIGHER THAN AT FOUR YEAR AND ABOVE INSTITUTIONS?

IS THERE A CAMPUS-WIDE ASSESSMENT INITIATIVE AT YOUR COLLEGE?

Also higher than at 4 year and above institutions (84.9%)

Only 80% of these libraries are involved in the campus-wide assessment initiative

IS ASSESSMENT A PRIORITY OF LIBRARY ADMINISTRATION?

Higher than at 4 year and above institutions (77.8%)

IS THERE A SHARED UNDERSTANDING OF THE PURPOSE OF ASSESSMENT IN YOUR LIBRARY?


ARE THERE CLEAR EXPECTATIONS FOR ASSESSMENT IN YOUR LIBRARY?


DOES YOUR LIBRARY HAVE AN ASSESSMENT PLAN?


ARE LIBRARY FACULTY/STAFF ADEQUATELY SUPPORTED IN THEIR ASSESSMENT WORK?


DOES YOUR LIBRARY HAVE THE NECESSARY SKILLS IN-HOUSE TO DEVELOP AND CONDUCT MEANINGFUL ASSESSMENTS AND ANALYZE THE RESULTS?

Higher than at 4 year and above institutions (56%)

DOES THE LIBRARY LEADERSHIP USE ASSESSMENT DATA SYSTEMATICALLY IN DECISION-MAKING AND PLANNING?


DOES THE LIBRARY STAFF USE ASSESSMENT DATA SYSTEMATICALLY IN DECISION-MAKING AND PLANNING?

Higher than at 4 year and above institutions (71%)

ON WHICH FACILITATING FACTORS DID COMMUNITY COLLEGES SCORE SIGNIFICANTLY (P<0.05) HIGHER?

➤Is there a shared understanding of the purpose of assessment in your library?

➤Are library faculty/staff adequately supported in their assessment work?

➤Does the library leadership use assessment data systematically in decision-making and planning.

WHAT FACTORS ARE ASSOCIATED WITH HAVING A CULTURE OF ASSESSMENT?

REGIONAL ACCREDITING ASSOCIATION IS ASSOCIATED WITH A CULTURE OF ASSESSMENT

TENURE STATUS AND FACULTY STATUS IS ASSOCIATED WITH A CULTURE OF ASSESSMENT

p=0.02 This was not the case in the study of four year and above institutions

HAVING A CAMPUS-WIDE ASSESSMENT INITIATIVE IS ASSOCIATED WITH A CULTURE OF ASSESSMENT

P<0.001

ALL OF THE FACILITATING FACTORS ARE SIGNIFICANTLY ASSOCIATED WITH A CULTURE

OF ASSESSMENT

FACTORS WHOSE PRESENCE IS MOST STRONGLY ASSOCIATED WITH A CULTURE OF ASSESSMENT

➤Clear expectations for assessment in the library (90% who have it have an assessment culture)

➤Assessment plan (82%) ➤Shared understanding of the purpose of assessment in

the library (81%) ➤Library leadership uses assessment data (78%) ➤Library leadership offers explicit support (77%)

FACTORS MOST STRONGLY ASSOCIATED WITH NOT HAVING A CULTURE OF ASSESSMENT

➤Assessment is not a priority of administration (77% who report this do not have an assessment culture)

➤Librarians do not use assessment data (72%) ➤Assessment data is not available to interested parties

(74%) ➤Library leadership does not use assessment data

(70%) ➤No shared understanding of the purpose of assessment

(69%)

WHAT FACTORS ARE MOST OFTEN PERCEIVED AS FACILITATING?

➤ Clear expectations for assessment in the library (94% who perceive that this facilitates their assessment culture have an assessment culture)

➤ Library has an assessment plan ➤ Librarians use assessment data to improve practice ➤ Library leadership uses assessment data ➤ Existence of learning outcomes

OPEN-ENDED RESPONSES: WHAT ARE THE MOST IMPORTANT FACILITATING FACTORS?

➤Institution-wide emphasis on assessment ➤Administrative priority and/or support ➤Accreditation ➤Organizational culture ➤Information literacy as a campus-wide outcome

OPEN-ENDED RESPONSES: WHAT MOST HINDERS CREATING AN ASSESSMENT CULTURE?

➤Lack of sufficient staffing ➤Lack of time ➤Faculty or staff have not made assessment a priority –

either due to lack of interest or resistance to assessment

➤Lack of expertise/access to training ➤Institution doesn’t prioritize assessment

REFLECTIONS

“CULTURE OF ASSESSMENT” IS IN THE EYE OF THE BEHOLDER

http://www.flickr.com/photos/kylemay/1573158163/

WITHOUT AN INSTITUTIONAL COMMITMENT TO ASSESSMENT, A CULTURE OF ASSESSMENT IS FAR LESS LIKELY TO EXIST

Institutions with a campus-wide assessment initiative

66% of libraries report having an assessment culture

Institutions with no campus-wide assessment initiative

36% of libraries report having an assessment culture

THERE ARE SIGNIFICANT ASSOCIATIONS WITH TENURE AND FACULTY STATUS THAT ARE NOT PRESENT AT FOUR YEAR AND ABOVE INSTITUTIONS

HAVING A PLAN IS CRITICAL

Fewer than half of libraries... ➤ Have clear expectations for assessment (52%) ➤ Have an assessment plan (45%)

Of libraries reporting clear expectations and an assessment plan, 92% report having a culture of assessment (versus 32% of those who report the absence of both)

HOW ADMINISTRATION PROMOTES ASSESSMENT MAKES A DIFFERENCE

Of those indicating assessment is a priority of administration

74% report having a culture of assessment Of those indicating administration uses assessment data in decision-making 78% report having a culture of assessment Of those indicating administration adequately supports faculty/staff in assessment work 69% report having a culture of assessment Of those indicating a library has all three factors 88% report having a culture of assessment

HAVING INSTRUCTION AND ASSESSMENT LEADERS ALSO HELPS

Of those academic libraries with an instruction coordinator

74% report having a culture of assessment Of those with an assessment coordinator 87% report having a culture of assessment Of those with an instruction committee 71% report having a culture of assessment Of those with an assessment committee 87% report having a culture of assessment

Of those with none of these positions 57% report having a culture of assessment

MOST COMMUNITY COLLEGE LIBRARIES ARE

ASSESSMENT LEADERS

CONSIDER

➤ What has helped your library move toward (or achieve!) a culture of assessment?

➤ What has hindered your library in moving toward a culture of assessment?

➤ Which of these facilitating factors seem most important for you to work toward at your institution?

FUTURE RESEARCH?

Questions? Comments? Challenges?

Meredith Farkas Portland Community College Email: meredith.farkas at pcc.edu Twitter: librarianmer