A Meta-Assessment of Statewide Program

EvaluationsJeffrey PomerantzCarolyn HankSchool of Info. & Library ScienceUNC Chapel Hill<pomerantz, hcarolyn>@unc.edu

Charles R. McClureJordon Andrade

College of InformationFlorida State University

<cmcclure, jca07d>@fsu.edu

Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931

http://is.gd/1eJQ

Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931

Research questions

1. What methodologies are used in LSTA evaluations?

2. How do states’ goals map to LSTA goals?3. What, if any, correlations are there between

methodologies used and states’ and LSTA goals?

4. How successful are different methodologies in providing useful evaluation data about library programs?

Background on IMLS and LSTA2008: IMLS awarded ~$161 M under Grants to

States program

LSTA funds < 15% of total state library fundingBut ≈ 97% of all federal funding to state libraries

IMLS encourages grantees to use Outcome Based Evaluation methods

Rendon’s LSTA goal categoriesA: Establish or enhance electronic linkages among or between libraries.B: Electronically linking libraries with educational, social, or information

services.C: Assisting libraries in accessing information through electronic networks.D: Encouraging libraries in different areas, & encouraging different types of

libraries to establish consortia & share resources.E: Paying costs for libraries to acquire or share computer systems and

telecommunications technologies.F: Targeting library & information services to persons having difficulty using a

library & to underserved urban & rural communities, including children from families with incomes below the poverty line.

The Rendon Group (2003). National Profile: Analyses of the Five-Year Evaluations submitted to the Institute of Museum and Library Services by the State Library Administrative Agencies under the Grants to States program of the Library Services and Technology Act. Washington, DC: Institute of Museum and Library Services.

Methodology

Collected 5-year plans & evaluation reports for 28 states for 1998-2002 & 2003-2007.

From 5-year plans:1. State’s 5-year goals,2. Connection between state’s goals and LSTA’s goals.

From evaluation reports:1. Methodologies and data collection instruments used,2. Stakeholder groups that provided data,3. Accomplishment of LSTA goals,4. Recommendations.

Methods used in LSTA evaluations

Methodologies used

Number of methods used

Data collected from whom

States’ goals mapping to Rendon

Accomplishment of states’ goals

Accomplishment of Rendon goals

Methods used to evaluate goals

Discussion

Quality and readability of eval reports varied considerably.

Need more explicit mapping between state & LSTA goals, and eval methodologies.

Only 39% of goals accomplished completely.

Recommendations

• To libraries:– Need for clearer goals & measurable objectives.– Need for more appropriate methods.

• To IMLS:– Not just OBE.– Simplify the process.

• To both:– Need for more consistent use of terms.– Need for planning pre-eval & re-program implementation.

Answers to research questions1. What methodologies are used in LSTA

evaluations? A lot of surveys.2. How do states’ goals map to LSTA goals?

Not very well.3. What, if any, correlations are there between

methodologies used and states’ and LSTA goals? Not many.

4. How successful are different methodologies in providing useful evaluation data about library programs? Not very.

Thank you!

Jeffrey PomerantzCarolyn HankSchool of Info. & Library ScienceUNC Chapel Hill<pomerantz, hcarolyn>@unc.edu

Charles R. McClureJordon Andrade

College of InformationFlorida State University

<cmcclure, jca07d>@fsu.edu

Tapping the vast reservoir of human knowledge --Louis Round Wilson, founder, 1931