Assessing Library Contributions to University Outcomes

Assessing Library Contributionsto University Outcomes9th Northumbria International ConferenceUniversity of York, EnglandJoe Matthews

August 2011

Universities under increasing pressure to demonstrate value and accountability

Google images Accountability2.jpg

business-strategy-innovation.com

2Indirect Measures

Indirect measures of student learningGraduation rates, retention rates, employment rates, student surveys, graduate school exam resultsData is easy to obtain and report to outside stakeholders

3National Survey of Student EngagementAcademic challenge

Opportunities for collaborative learning

Interactions with faculty

Enriching extra-curricular experiences

Supportive environment for learningStudents are asked to report their perceptions of campus life in 5 areas4NSSE & LibrariesLibrary use & educational purposeful activities are correlated at small liberal arts colleges

Larger universities no correlation

Students who use the library more likely to work harder meet faculty expectations

George Kuh and Robert Gonyea (2003) C&RL article5Library ExperiencesDo not lead to gains in information literacy

Do not lead to gains in student satisfaction

Do not lead to what students gain overall from collegeGeorge Kuh and Robert Gonyea. The Role of the Academic Library in Promoting Student Engagement in Leaning. College & Research Libraries, July 2003, 256-282. 380,000 respondents

6

NSSE survey 50% or more of survey respondents indicated that they never entered or used a library service during their undergraduate yearsGoogle images UC Berkeley library

7Book Use

Goodall & Pattern (2011)UK Library Impact ProjectAlmost 50% of students never borrowed a book and another 10% only borrowed 1-5 books during a year8

eResources

Library visitsAbout 40% of students have not been in the library and another 6-9% only visited 1-4 times9Direct Measures

Direct measures - Exams in the pre-test and post-test model to determine contribution of the university

Google images achievement.jpg

allposters.com

10Student LearningThe contribution of the university

in assessing student learning

is indirect, at best.It is difficult to separate the contribution of the university from other factors the peer group, family, work experiences, the Internet

11Assess LearningThe Collegiate Learning Assessment (CLA)

The Collegiate Assessment of Academic Proficiency (CAAP)

The Measure of Academic Proficiency and Progress (MAPP) Direct measuresAnother strategy use an examination to assess broad measures of of critical thinking, communication and decision-making12Collegiate Learning AssessmentCritical thinking

Judgment

Analytical reasoning

Problem solving

Writing skills Essay exams that ask students to reflect on information that is provided and then write an essayDoes not attempt to assess domain-specific knowledgeThe goal is to determine the value added by a college education13Entering

Student

CharacteristicsGraduating

Student

CharacteristicsCampus EnvironmentProgramsInstitutionalCharacteristicsFellow StudentsPlace of ResidenceFacultyLibrary ServicesAstins IEO ModelClassesStudent Learning FrameworkDemonstrates the complexity of the educational process14

Shavelsons Student LearningOutcomes ModelClearly, what is learned and how well it transfers to new situations depends on the natural aptitudes and abilities that students bring with them to the work place. These aptitudes and abilities are the product of their education (in and out of school) along with their inherent capabilities. Many compelling distractions that lead to the total collegiate experience

15Total Collegiate Experience

The challenge for any student is finding the right balance between the demands for learning and the time spent on extracurricular activities both on and off campus. For many, finding this balance can be tricky.

Google images sleepreplacementdrug.jpg

deepamohandas.blogspot.com

16Time Spent Studying19642004Philip Babcock and Mindy Marks. The Falling Time Cost of College: Evidence from a Half Century of Time Use Data. NBER Working Paper No 15954. Washington, DC: The National Bureau of Economic Research, April 2010.Decline from about 40 hours per week (attending class and studying) to between 26-28 hours per week

17Disengagement Compact

Rather than rolling up their sleeves and getting to work, students today are gaming the college environment by carefully shaping schedules, carefully selecting instructors, and working to limit workload. George Kuh has suggested that a disengagement compact has been struck between students and faculty. This compact can be briefly summarized, as Ill leave you alone if you leave me alone.

Google images ComputersandLecture.jpg

tusb.stanford.edu

18

Megan Oakleafs recent report19Areas of ImpactStudentFacultyUniversityEnrollment

Retention & graduation

Success

Achievement

Learning

Experiences, attitudes & perceptions of quality

Research productivity

Grants

TeachingInstitutional reputation & prestigeLimitationsMicro-level studies

Inward looking

Small samples sizesNeed Demonstrationsof Value

One ModelSchool libraries & standardized test scores

Controlled for school & community differences and found high correlations with use of library & test scores

20 studies in different states

Broad-based Data Analysis

What we need -23Library Data Farm

Lots of data silosJoe Zucca Univ of Penn

24ProcessesLoadCleanNormalizeAnonymize

AnalysisExport

Libraries need a metric management system that allows for the integration of library data plus campus dataSuch a system needs to:25

Assessment Management SystemsCounting OpinionsCloud-based services26Expand Data SetsIn addition to library data

Partner with the Office of Institutional ResearchNCESIPEDSNSSECLACampus surveysStudent registrar data (enrollment, grades)NCES = National Center for Educational StatisticsIPEDS = Integrated Post Secondary Data SystemNSSE = National Survey of Student EngagementCLA = Collegiate Learning Assessment

27Anonymity & privacyare not incompatible

Library Needs to Support Assessment

Collections & Services SpaceVirtual SpaceCommunity SpaceNew approaches needed to develop a better understanding who is using the library and why

29Collections & Services SpaceILS dataIn-library use dataILL dataUse of IT servicesReference servicesInstructional servicesOther

Recently an analysis of data at the Huddersfield University library in the U.K. found that students who read more, measured in terms of borrowing books and accessing electronic resources, achieve better grades while students who make less or no use of the library attain lower grades Goodall & Pattern 2011

Google images HuddersfieldUniversity.jpgbbc.co.uk

30Library Use & GPA

A study at the Hong Kong Baptist University library found a positive relationship between grade point average (GPA) and the use of the librarys collections. 8,700 + pairs of students C&RL latest issue

Google images 318a450x320.jpg218.188.25.8431Virtual Space

Authentication logs

32

For each individual - # of times log in and what resources are being used

33Community Space

Need to identify specific individuals scan ID cards and learn how they are using the libraryMeeting rooms (individual and group), labs, IT services, library collections, library services

34Combine the Data

Data Jockey35Library Assessment ConferenceBuilding Effective, Sustainable, Practical AssessmentBaltimore, Maryland 2010David ShulenburgerDavid Shulenburger, Vice-President for Academic Affairs for the Association of Public and Land-Grant Universities Kalamazoo College faculty - Collegiate Learning Assessment data

36

PrivacyInstitutional Review BoardPartneringChallenges37Broad-based Data AnalysisEnables a library to prepare a credible analysis of the librarys impact in the lives of

Students

Faculty

ResearchersThe Goaluntil libraries know that that student #5 with major A has downloaded B number of articles from database C, checked out D number of books, participated in E workshops and online tutorials, and completed courses F, G, and H, libraries cannot correlate any of those student information behaviors with attainment of other outcomes. Until librarians do that, they will be blocked in many of their efforts to demonstrate value. Megan OakleafBooks

Print journals

Special collectionsIntellectual developmentIntangibleTangibleProductAssessment

= GradeSuccesseJournals

eBooks

eResourcesUseUseLibrary Impact ModelNeed to develop a more complex model to better understand the impact of the libraryUse Structural Equation Modeling - 40The GoalGet a better handle on:

Who is using the library?

Why are they using the library?

What impact does library use have in their life?The solution its the data! A metric management system that eliminates the data silos and integrates the data at the student level so we can understand and demonstrate the value of the library!41Questions?

www.joematthews.org

joe@joematthews.org

Inheritance xAccumulatedExperienceAbstract, ProcessOrientedIntelligenceGeneralFluidCrystallized

General ReasoningVerbalQuantitativeSpatialExample: Graduate Record Examination

Broad AbilitiesReasoningCritical ThinkingProblem SolvingDecision MakingCommunicatingIn Broad DomainsDisciplines - Humanities, Social Services, SciencesAnd Responsibility Personal, Social, Moral, and CivicExample: Collegiate Learning Assessment

Concrete, Content-Oriented

Direct ExperienceKnowledge, Understanding, and ReasoningIn Major Fields and Professions (Business, Law, Medicine)Example: ETSs Major Field Tests