Learning about learning 2.0: evaluating the New South Wales public library Learning 2.0 program

Download Learning about learning 2.0: evaluating the New South Wales public library Learning 2.0 program

Post on 30-May-2018

215 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • 8/14/2019 Learning about learning 2.0: evaluating the New South Wales public library Learning 2.0 program

    1/23

    LEARNING ABOUT LEARNING 2.0: EVALUATING THE NEW SOUTHWALES PUBLIC LIBRARY LEARNING 2.0PROGRAM

    Ellen Forsyth, Mylee Joseph and Leanne Perry

    Ellen Forsyth, State Library of New South Wales, Sydney 2000. E-mail:eforsyth@sl.nsw.gov.au. Mylee Joseph, State Library of New South Wales, Sydney2000. E-mail: mjoseph@sl.nsw.gov.au. Leanne Perry, State Library of New SouthWales, Sydney 2000. E-mail: lperry@sl.nsw.gov.au.

    Presented at Research Applications in Information and Library StudiesConference 5 (RAILS5) University of Technology, Sydney 23 January, 2009.

    AbstractThe New South Wales Learning 2.0 training program was made available to

    the entire public library work force of over 2300 people in 2008 - 2009. Thispaper is a snapshot of the impact the Learning 2.0 course at September 2008,five months after the course was launched. The paper explores how thetraining has impacted on the level of skill, knowledge, and confidence relatingto web 2.0 tools.

    Aim

    The aim of this evaluation of the New South Wales public library Learning 2.0program is to measure the impact of self paced web 2.0 training on individualsworking in New South Wales (NSW) public libraries. In this paper the focus is

    on how the training has impacted on the level of skill, knowledge, andconfidence relating to web 2.0 tools.

    Background to the Learning 2.0 Program

    2,344 staff are employed in 363 local government public libraries and onboard 22 mobile libraries throughout New South Wales1. From discussionsand anecdotal evidence, it became apparent to Public Library Services staff atthe State Library that a significant proportion of public library staff did notpossess the knowledge or technical skills required to fully participate in therapidly changing library service environment, including engagement with web2.0 technologies, which is emerging to meet changing client expectations inan increasingly interactive online environment.

    The New South Wales public library Learning 2.0program (Learning 2.0)2 isan online self-paced training program to allow public library staff across NewSouth Wales to learn more about emerging technologies on the web. Theprogram was written by Ellen Forsyth and Mylee Joseph. They work in thePublic Library Services branch at the State Library of New South Wales. Thisteam is responsible for promotion and development of public library servicesin New South Wales. The course was based on Learning 2.0 programs from

    Orange County Library System in Florida

    3

    and King County Library System inWashington4. Learning 2.0 is a twelve week online self-discovery training

    mailto:eforsyth@sl.nsw.gov.aumailto:mjoseph@sl.nsw.gov.aumailto:mjoseph@sl.nsw.gov.aumailto:lperry@sl.nsw.gov.aumailto:eforsyth@sl.nsw.gov.aumailto:mjoseph@sl.nsw.gov.aumailto:lperry@sl.nsw.gov.au
  • 8/14/2019 Learning about learning 2.0: evaluating the New South Wales public library Learning 2.0 program

    2/23

    program designed to encourage library staff to explore new web-basedtechnologies like blogs, wikis and podcasts. Each week there is a module towork through which introduces participants to new web 2.0 tools. There willoften be a video to explain the tool, participants will see examples of goodpractice library and museum implementations, and there will be an exercise to

    try tools for themselves. The course included setting up a blog, exploringFlickr, tagging and Creative Commons; RSS and RSS aggregators, Wikipediaand other wikis; online video including Youtube and Google video,folksonomies, delicious and LibraryThing, answer boards, social searching,podcasts and audio, mashups, online applications and social networks.

    Each participant maintains a learning blog where they write about what theyhave discovered each week. Participants are encouraged to comment oneach others blogs. The original program was designed by Helene Blowers forthe Public Library of Charlotte and Mecklenburg County5. It has a creativecommons licence which has allowed other programs to be developed,

    including the programs at Orange County Library System in Florida6 and KingCounty Library System in Washington and the State Library of New SouthWales. The training program is completely built on web 2.0 technologies thatare freely available on the Internet. These sites include: Blogger, Flickr,YouTube, PBWiki & Bloglines.

    The New South Wales public library Learning 2.0 program was publicisedextensively through e-mail and meetings. All public library staff wereencouraged to participate. It was presented as a course in which everyonecould participate. There were no barriers to participation from the StateLibrary. Barriers to participation occurred due to local council policies onaccess to some sites (particularly YouTube and online e-mail), speed ofconnections (broadband is not everywhere) and the level of interest ofparticipants.Course material is accessed via the internet allowing staff in rural and regionalcentres equal access to the program. As there is no prior knowledgerequired, staff with any skill level can participate. The program is self paced,allowing flexibility as to when and where it is undertaken, and how long aperson takes to complete the course. The program is built using the sameweb 2.0 tools included in the course. It is published by the State Library of

    New South Wales with a Creative Commons licence allowing it to be easilyshared with other agencies.

    Background on the learning 2.0 evaluation what is happeningelsewhereWhile there are more than 250 individual 23 things or Learning 2.0programs7 either completed or currently underway in almost 500 participatinglibraries worldwide, there is limited information available around the evaluationand outcomes of these programs. Where information is available it is clearthat the evaluation of the success of programs has been largely measured vianumbers of completions and feedback received via participants through blog

    entries.

  • 8/14/2019 Learning about learning 2.0: evaluating the New South Wales public library Learning 2.0 program

    3/23

    Some programs have actively sought feedback via surveys at the end of theprogram, typically an online evaluation in the last week of the program forexample the Minnesota Multitypes 23 Things on a Stick program.8Generallysurveys have focussed on the participant assessment of the program,including reactions to the program, both initially and at the end of the program,

    favourite and least favourite parts of the program, what was most challengingabout the program, possible improvements for the program and ways web 2.0applications could be used in the participants library service.

    Recently there has been a move to establish clear evidence of the impact ofthe Learning 2.0 programs on library services. Helene Blowers, the designerof the original Learning 2.0 program, 23 things9 has initiated a survey forLearning 2.0 program co-ordinators. The survey seeks to evaluate howsuccessful the program has been on a global level and what are the keyfactors for success including issues such as organisational support andincentives. The survey also attempts to measure some of the broad impacts

    of the program rather than just the program itself, by including questions onthe outcomes for staff and the development of new services as a result of theprogram. Blowers survey of more than 700 course coordinators around theworld was released in August 200810 and the survey results were released inJuly 2009.11 This was a very high level survey which mainly asked aboutmotivation for participation, openness of participation, use of incentives andmanagement support

    The NSW Public Library Services Learning 2.0evaluation included a range ofstrategies which sought to measure both satisfaction of participants with theprogram and the impact of the program on knowledge and use of web 2.0tools at an individual and library service level. As with other Learning 2.0evaluations feedback received via program and participant blogs were a keysource of information for evaluating the success of the program. However,unlike most other evaluations we undertook to measure the change in skilllevel of participants in web 2.0 technologies over the course of the programand at nine month post program completion. Although Mayo Clinic LibrariesLearning 2.0 program have also undertaken some evaluation of the impact oftheir program on individual participants skill levels there is little evidence thatthis level of evaluation has been undertaken in a public library Learning 2.0program context.12

    Research methodologyThere are six components to the research methodologyCompletion ratesProgress of participants through the program was monitored and recorded bythe project team leaders. This information included the number of peoplestarting, the number who reached mid point and the number who completed.

    Self assessment of skills and knowledgeEach participant completed an anonymous self assessment survey at thebeginning of the training program. These were deliberately anonymous as it

    was important for people to be able to answer honestly. The survey wasdesigned to give a realistic snapshot of participant skills and understanding. It

  • 8/14/2019 Learning about learning 2.0: evaluating the New South Wales public library Learning 2.0 program

    4/23

    was considered more important to have a state wide overview rather thantracking changes to individuals.

    Mid point surveyThe mid point survey was included in the activities for week six to check how

    participants were managing online learning. The survey was designed toidentify issues with the program and online learning, what was working andwhat wasnt. It was also designed to find if the current level of support beingprovided was effective.

    Completion surveyThe survey, at the end of the program, repeated the questions from the midpoint survey and the self assessment. The purpose of this survey was toidentify the changes in participants perceptions of their skills and knowledgeof web 2.0 tools, and to comment on the content of, and support for, theprogram. This survey may be skewed by the high achievers completing and

    others falling by the wayside and not completing the survey.

    Nine month post completion surveyA nine month completion survey will be undertaken with all participants whocomplete the program. This survey will explore the longer impact of theLearning 2.0 program on library services. The survey will include looking atthe tools people are using in their libraries, what policies have changed andhow skill levels have been maintained or enhanced. This survey wasundertaken in March and the results are being analysed.

    Feedback on blogsThe program was designed to elicit comments and feedback on both theprocess and progress of learning about web 2.0 tools. All participants wererequired to establish a learning diary blog which was publicly visible, linked tothe course blog roll but did not disclose their identity or library. The progressof individuals was monitored by comments and feedback posted on theirlearning blogs, and comments posted on the administrators blog.

    Evaluation results

    1. Program participants, progress and completion rates

    More than 10% of the public library workforce in New South Wales havesuccessfully completed the program to date. While this is reasonable, initialexpectations of completion rates were much higher. Over one third of theNew South Wales public library workforce started this program. This was thelargest number of people to ever to participate in training provided by theState Library. A snapshot of the variety of roles is useful in assessingparticipation across the profile of the NSW public library workforce.

    Table 1 Role of people undertaking the program

    Roles No. enrolled

    Library Manager 52Team leader / coordinator 91

  • 8/14/2019 Learning about learning 2.0: evaluating the New South Wales public library Learning 2.0 program

    5/23

    Branch librarian / Branchlibrary officer

    96

    Specialist [eg. children's,reference, local studiesetc.]

    176

    Customer service officer 17Mobile library driver 4

    Library officer / technician 165

    Library assistant 240

    Other 76

    Total 917

    The following table shows the numbers of people undertaking the program, bytype of public library at the start, midpoint and completion of the program.

    Table 2 Completion rates

    Start Mid point Completion

    Branch Library 331 107 87

    Central Library 446 149 110

    Regional headquarters 67 29 18

    Mobile library 6 4 2

    Joint use library 7 1 3

    Other 36 6 6

    Total 893 296 226

    Note the different totals in the tables are because not everyone chose to

    provide information about the kind of library they work in.

    The mix of library workplaces was useful in assessing the accessibility of thecourse and any barriers to participation and completion. The results indicatedthat staff working at a branch library were just as likely to complete the courseas those working in central and headquarters locations, ie. there were norelevant geographical barriers and staff at branch locations were equally selfmotivated to complete the course. This is useful information for planningfuture professional development activities for the NSW public library network.

    2. Self assessment of skills and knowledge

    The self assessment undertaken by participants at the mid point andcompletion of the program provides a clear indication of an increase inparticipant skill and knowledge level of the web 2.0 tools. As illustrated inTables 3 and 4, a clear shift in skill level was indicated in all web 2.0 tools.The shift followed a consistent pattern of moving up one stage in terms of themajority of participants skill levels. In all cases where the majority ofparticipants indicated that they were unfamiliarwith a web 2.0 tool prior to thecommencement of the course the majority of participants moved to the newuser skill and knowledge levels. Similarly where the majority of participants

    indicated that they were new users prior to the commencement of the courseat the completion of the course the majority of these participants had moved

  • 8/14/2019 Learning about learning 2.0: evaluating the New South Wales public library Learning 2.0 program

    6/23

    up one level to userskill and knowledge levels. By the completion of thecourse at least 82% of participants indicated they were at new user level orabove for all web 2.0 tools.

    Table 3.Self assessment prior to commencing the Learning 2.0 program

    0%

    10%

    20%

    30%40%

    50%

    60%

    70%

    80%

    90%

    100%

    Blogs

    ollaboration

    oftware[eg.

    agging[eg.

    el.icio.us,

    Online

    project

    Graphic

    organizer

    RSS

    Podcasts

    and

    Wikisand

    Wikipedia

    Social

    networking

    Instant

    messaging

    nlinevideo

    [eg.

    igitalonline

    usic[eg.

    Digital

    ictures[eg.

    Mashups

    Answer

    boards[eg.

    Internet

    safety

    Power U

    User

    New Use

    Unfamilia

    Table 4. Self assessment at completion of the Library 2.0 program

    0%

    10%

    20%

    30%

    40%

    50%

    60%

    70%

    80%

    90%

    100%

    Blogs

    ollaboration

    oftware[eg.

    agging[eg.

    del.icio.us,

    Online

    project

    Graphic

    organizer

    RSS

    Podcasts

    and

    Wi...