Lorna Dodd
College Liaison LibrarianUCD Library | Dublin 4. email: [email protected] Tel: +353 (0)1 716 7074 www.ucd.ie/library
The Information Literacy Impact factor: How to Measure Value
Outline
Why conduct a survey of module coordinators?
Review of the Literature
Rationale
How the survey was conducted
Survey results
Implications for the future
Conclusions
Lorna Dodd: LILAC 2013
Why?
Lorna Dodd: LILAC 2013
Why?Looking Back
& Looking Forward
Lorna Dodd: LILAC 2013
Looking Back
Large University
No single strategy for the development of Information Literacy
Heavy load of information skills delivery
Driven by individuals
Sometimes evaluated but rarely measured
No clear indication in meeting learning outcomes(Dodd, L. & Kendlin, V., 2010)
Lorna Dodd: LILAC 2013
Looking Forward…
New Library Strategic Plan 2010-2014
“monitor, measure and evaluate the Library’s teaching and learning strategy and
activities.” (UCD Library, 2010)
Lorna Dodd: LILAC 2013
Looking forward
Realignment of staffing structure From approx 17 Liaison Librarians to 6 College
Liaison Librarians
Move from module to programme approach
Move from responsive to consultation
Measure impact on student learning
Lorna Dodd: LILAC 2013
Looking Forward…
Lorna Dodd: LILAC 2013
Literature Review (1)
A lot available discussing why it is important to measure impact
Much of what has been written focuses on Library use rather than instruction.
(Stone 2011; Poll & Payne 2006; Schilling & Applegate 2012).
Limitations of traditional organisational approaches at measuring success and meeting targets.
Most libraries focus on process and output indicators rather than measuring impact.
Measuring impact on student learning requires systematic evaluation of training
(Markless & Streatfield 2006; Schilling & Applegate 2012))
Lorna Dodd: LILAC 2013
Literature Review (2) Several problems arise when measuring impact.
Data protection rules Differing value of services for specific groups Difficulty in measuring long-term impact Time consuming nature of measuring impact of Library
instruction.
Important to use results of any study conducted to measure the impact of Library services Benchmarking activities Improving services Justifying resources used for services Campaigning for increased funding.
Common/Successful ways to measure impact of Library instruction Surveys; Pre and post-tests; Self-assessments; Behavioural observations.
(Poll & Payne 2006)
Lorna Dodd: LILAC 2013
Literature Review (3)
2006 LIR/SCONUL measuring impact initiativeDeveloped impact process
Identified key performance indicators
Used to evaluate the effectiveness of interventions supporting teaching and learning
Research provided information regarding emerging issues throughout the process.
Results included: Improved relationships with academic community A raised profile for Library staff Better understanding of how the Library can support
academic programmes.
(Markless & Streatfield 2006; Blagden, 2005)Lorna Dodd: LILAC 2013
Literature Review (4)
2012 CONUL Information Literacy SurveyGathered feedback on the impact of information literacy instruction
Found library instruction had a positive impact on students’ skill development
Information Literacy often assessed
Targeted respondents
Academics already fully engaged
First year students only
Small sample from each institution
(CONUL Advisory Committee on Information Literacy, 2013)
Lorna Dodd: LILAC 2013
We needed to:
Make academics aware of new approach
Create a strategy to get academics to embrace new approach
Identify what has been successful to date
Collect feedback from those not actively engaged with the Library
Identify what methods would be useful going forward
Problems:
Measuring impact retrospectively
Perceived impact is subjective
External factors influence development of information skills
More likely to opt for options that are familiar
Rationale
Lorna Dodd: LILAC 2013
Survey
Seeking information on modules
Two parts:1. Evaluating information skills delivery to date2. Identifying what would be useful in the future
Survey Monkey 11 questions Mainly quantitative Multiple choice questions Measurement/ranking questions
Lorna Dodd: LILAC 2013
Survey
Introduction:
Level of module – undergraduate/taught postgraduate
If there has been Library instruction
Part 1:
Kind of instruction
Impact of instruction on students’ abilities
Part 2:
Should all students have library instruction
When should library instruction happen?
Which services should be developed
Free Question
Lorna Dodd: LILAC 2013
Promotion
Approx. 700 module coordinators
Only possible to target academic staff
Not all module coordinators are academic staff and not all academic staff are module coordinators
Targeting email – approx 1,900
College Liaison Librarian contacts
Three weeks
Lorna Dodd: LILAC 2013
Survey Results
Type of instruction
Impact of instruction
When and who?
Preferred services for the future
Qualitative feedbackLorna Dodd: LILAC 2013
Type of Library Instruction
Lorna Dodd: LILAC 2013
Impact of Library
Instruction
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Mostly Taught Postgraduate: EndNote Special Collections PBL One-on-One Consultation
“Students have always received instruction to date
and so difficult to answer the comparison questions below”
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Should ALL students have Library Instruction?
Lorna Dodd: LILAC 2013
Should ALL students have Library Instruction?
Lorna Dodd: LILAC 2013
Should Library Instruction happen in every year?
Lorna Dodd: LILAC 2013
Should Library Instruction happen in every year?
Lorna Dodd: LILAC 2013
Which Stage?
“Instruction in Year one should be built on in subsequent years
to enhance skills by year”
Lorna Dodd: LILAC 2013
Preferred Services for the
FutureLorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Influence of instruction had on preference?
Instruction had
First Choice Second Choice Last Choice
Tour Subject Specific Workshops
Generic workshopTutorialOnline Video
Tour
Lecture Subject Specific Workshops
Generic WorkshopTutorial
Tour/Online Video
Workshop Subject Specific Workshops
Generic WorkshopTutorial
Tour
No Instruction Subject Specific Workshops
Subject Specific Workshops
Tour
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
Lorna Dodd: LILAC 2013
“One of the best approaches would be to create materials that are reusable and for any student, regardless of level.
Just because a student moves up a level doesn't mean they've acquired all of the
learning they should have”
“…really good online video materials, from complete intro to specific and
advanced, and available on demand is the way to go”.
“Tutorials (tutors would be trained by Library staff)”
Lorna Dodd: LILAC 2013
Qualitative Feedback
Lorna Dodd: LILAC 2013
Comments Themes
20% (53) of those who completed the survey made comments
In line with our new strategy
Online options
Constructive comments & suggestions on current and future services
Complimenting services to date
Comments on the survey
Lorna Dodd: LILAC 2013
Comments
5 respondents made comments inline with our new strategy Programme Approach Reusable Learning object and Online tools
6 respondents made comments about proposed future instruction
Lorna Dodd: LILAC 2013
Comments 12 respondents made comments about current
services and changes
Problems with current service:
“…same 'introductory' talks by librarian using lecture slots of various modules for
at least three times (the third time he skipped as he found it ridiculous)
“Students just skip the guest lecture.”
Lorna Dodd: LILAC 2013
Comments
Benefits of current service:
“…requires personal contact with a friendly face early on.
“The current reduction in library services is a real retrograde step especially with the explosion in available information
sources”
Lorna Dodd: LILAC 2013
Comments
Over half (28) complimented current strategy
“we really love our librarians; thank you”
‘Campaigning’ for traditional serviceAcknowledging individualsRecognising value of information literacy to students
Lorna Dodd: LILAC 2013
“…it makes a noticeable difference in the quality of assignment students produce. The Library Tutorial is typically rated as one of the most important lectures they have in that Stage 1 module.”
Lorna Dodd: LILAC 2013
Implications for the Future
Lorna Dodd: LILAC 2013
Impact of Instruction
Definite benefit in Library instruction
Active learning environment is preferred
Stronger evidence in the discovery and use of resources
Remains difficult to identify the ‘real’ impact
Lorna Dodd: LILAC 2013
Timing of Instruction
Most module coordinators feel all students should have Library instruction
Instruction for students at every stage is desirable
Taught Postgraduates Beginning and middle of Semester 1
Undergraduates Stage 1 Stage 3
Lorna Dodd: LILAC 2013
Preferred Services Active learning sessions tailored to subject needs
most preferred
Tours perceived as ineffective
Questions:
Asked to identify how effective instruction was at improving students’ information literacy skills. When they request instruction, do they usually
consider this? Or do they feel students need to know about the
library rather than what they need in order to transition from 2nd level, develop researching skills etc.
Lorna Dodd: LILAC 2013
Questions
Did respondents choose workshop environment over online version because they are more familiar?
If so, then why did the majority of those who had no instruction choose workshop?
If delivering workshops on this scale is unsustainable, what can we do ensure effectiveness?
What work do we need to do in terms of marketing?
Lorna Dodd: LILAC 2013
Conclusion
Shared understanding between Library & Academic in:
Benefit of Instruction Taking a programme approach Ensuring all students get an equal
opportunity (core modules)
Difference in: Most appropriate method of delivery
Lorna Dodd: LILAC 2013
References
Blagden, P. (2005) ‘The LIRG/SCONUL Measuring Impact Initiative: Overview of phase 1 impact projects’, Library & Information Research (LIR) vol. 29 (91)
Dodd, L. & Kendlin, V. (2010) ‘Damned if we do and Damned if we don’t: How to address sustainability in the delivery of information literacy components in UCD’: Librarians' Information Literacy Annual Conference (LILAC), March 29-31, Limerick, Ireland.
Markless, S & Streatfield D. (2006) ‘Gathering and applying evidence of the impact of UK university libraries on student learning and research: A facilitated action research approach’, Journal of Information Management vol. 26 pp. 3-15
Poll, R & Payne, P. (2006) ‘Impact measures for libraries and information services’, Library Hi Tech vol. 24 (4) pp. 547-562
Schilling, K. & Applegate, R. (2012) ‘Best methods for evaluating education impact: a comparison of the efficacy of commonly used measures in library instruction’, Journal of the Medical Library Association, vol. 100 (4) pp. 258-269
Stone, G. et al (2011) ‘Does Library use affect student attainment? A preliminary report on the library impact data project’ Liber Quarterly vo. 21(1) pp. 5-22
UCD Library (2010) ‘UCD Library Strategic Plan 2010-2014’
http://www.ucd.ie/library/news_publicity/showcase_strategy/
Lorna Dodd: LILAC 2013