This guide is intended as a basic primer to the steps in program evaluation. There are many useful resources for program evaluation that can offer you in-depth guidance in evaluation design.

General Evaluation Planning Resources

Planning a Program Evaluation booklet and worksheetUniversity of Wisconsin-Extension Program Development and EvaluationThis easy-to-use worksheet (available in an editable Word document format) and booklet walk you through planning for the six steps in program evaluation. A version of this worksheet has been used by multiple Texas Mother-Friendly Worksites in planning their worksite lactation support program’s evaluation.http://www.uwex.edu/ces/pdande/evaluation/index.html

A Framework for Program EvaluationCenters for Disease Control and PreventionThis is a practical tool that summarizes the steps and standards for evaluating public health programs.http://www.cdc.gov/eval/framework/index.htm

Evaluation Handbook W.K. Kellogg FoundationThis handbook provides a framework for thinking about evaluation as a relevant and useful program tool.http://www.wkkf.org/knowledge-center/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx

The Fundamentals of Evaluation: Why, What, How, & When You Should Evaluate Your Wellness ProgramWellness Council of AmericaThese free resources, including The Fundamentals of Evaluation, are specific to the process of planning effective, sustainable programming for worksite wellness. This report presents the fundamentals for evaluating worksite wellness programs. http://www.welcoa.org/freeresources/index.php?category=8

Program Evaluation Planning Primer | 1

Developing an Evaluation PlanThe Community Tool Box Offered by the University of Kansas as a “resource for free information on the essential skills for building healthy communities,” this website includes a variety of information and tools for planning and implementing an evaluation.http://ctb.ku.edu/en/tablecontents/section_1352.aspx

Evaluation Planning Resources Specific to Worksite Lactation Support Programs

Lactation Support Program EvaluationCenters for Disease Control and Prevention, Healthier Worksite InitiativeThis website provides guidance and ideas for evaluating your worksite lactation support policy, breastfeeding support services, lactation room use, and the overall program.http://www.cdc.gov/nccdphp/dnpao/hwi/toolkits/lactation/evaluating.htm

Why Do We Need to Evaluate Our Program?

To ensure ongoing program success and quality improvement, it is important to evaluate your program’s process and outcomes on a regular basis. Evaluation is key to the sustainability of your program because it allows the value of the program to be recognized and celebrated. It also helps to continually assess the program direction to make sure that the components that are implemented continue to meet the needs of the employees and employer. Evaluation also allows program planners to assess the program’s return on investment (ROI) and shape future direction.

Key Functions of Program Evaluation Provide evidence of your department’s need for a worksite lactation support program. Affirm the initial decision to develop a program. Provide ongoing measurements of the various program components. Help you to make any necessary adjustments to the design of the program. Provide the data necessary to demonstrate success in achieving the program objectives. Facilitate the continuation or possible enhancement in funding.

Source: California Department of Public Health’s Guide to Develop and Implement a Work Site Health Promotion Program. Available from: http://www.dpa.ca.gov/benefits/health/wellness/guide/guidetoc.htm

2 | Program Evaluation Planning Primer

What are the Six Steps for Planning and Implementing a Program Evaluation?

1. Engage Stakeholders Stakeholders are those groups or individuals who are affected by the program, including program implementers, partners, participants, and decision makers. If you have developed a Mother-Friendly committee, you will already have representation from most or all of these groups. During this step, you will think about who will use the results of the evaluation to make decisions and what types of information they might need based on their role within the program.

2. Describe the ProgramMuch of this work is completed during your program’s assessment and planning phases. During the assessment phase, you have learned more about the context of the program and the current environment of your worksite. You have also learned about your employee characteristics and the different audiences that will be affected by the program, including potential program users, managers and supervisors, and other program “customers” or audiences.

During development of the Operating Plan in the planning phase, you have identified the program components you will be implementing; the resources, processes, and timeline you will use to implement them; and the anticipated outcomes of each.

Often in evaluation planning, information that describes the program is used to develop a systematic and visual tool called a logic model to demonstrate the relationship between the work you have planned and the results that you expect to occur through implementation of your program. This type of tool can help you understand and communicate how you believe your program will work.

Resources:

Logic Model Development GuideW.K. Kellogg Foundationhttp://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx

Developing and Using a Logic ModelCenters for Disease Control and Preventionhttp://www.cdc.gov/DHDSP/programs/nhdsp_program/evaluation_guides/logic_model.htm

3. Focus the Evaluation In this step of program evaluation, you will begin to home in on what aspects of your program that you will want to assess. Looking at your logic model can help you think about all the different questions you may have about your program. Did things occur as planned? Did the

Program Evaluation Planning Primer | 3

relationships between the parts of your program and the anticipated outcomes occur and if so, did they occur as anticipated? There are an unlimited number of questions you may ask, so focusing the evaluation will help you to arrive at those questions that will be most helpful for you to have answered.

The purpose of your evaluation should be determined by your committee’s understanding of:

Who needs to know about the program’s progress (implementers, partners, participants, and decision makers)?

What information key stakeholders will be most interested in?

What they will use the information for (to inform future program planning, to justify continued maintenance of the program, to solicit funding, etc.)?

The major areas of focus for a program evaluation are:

To measure the effectiveness of how the program was administered—was the program managed in a way that could reasonably support the objectives you set out to achieve?

To measure program outcomes—What was the impact of your program and the outcomes of your objectives and goals? Did your program result in the changes you intended?

Evaluation Questions: Evaluation questions are broad questions that express what you want to know about your program’s implementation.

Some evaluation questions you may consider are:

Program Administration Effectiveness

Were the target audiences aware of the program?

Did the target audiences understand what you were asking them to do?

Did they participate in some or all of the components?

To what extent did they participate?

Were they satisfied with the program components?

Were there any competing influencers or barriers to their participation in the program?

Were the benefits of participating in the program a greater influence than the real or perceived barriers to participation in the program?

Were you successful at implementing program activities or strategies according to your plan?

Did the audience have easy and convenient access to any tangible products or services you offered them?

Were the program activities or messages effectively placed to reach the target audiences?

4 | Program Evaluation Planning Primer

How many people were exposed to your program’s communication messages?

Did communication/promotion reach the right people (your target audiences)?

Was your policy implemented, and by what process?

To what extent has the policy been implemented?

Once implemented, how and to what extent is the policy maintained?

Program Outcomes

How and to what extent have attitudes and/or behaviors changed? Did they change in the anticipated direction (better, or worse)?

Because behavior change is a long-term outcome that is difficult to measure, how and to what extent have the precursors (intention, knowledge, skills, self-efficacy) to behavior changed?

Has there been an increase in perceived or actual access to worksite lactation support?

Has there been:

o Reduced absenteeism?

o Improved retention?

o Improved morale?

o Increased breastfeeding?

o Improved health outcomes?

o Return on investment?

To help you prioritize the questions you may ask, consider the following:

What resources are available to you (time, expertise, personnel, data, financial, control over other influencing variables, etc.)?

What resources would be needed to answer your question?

Assuming you are able to answer the question, would you be willing and/or able to make changes to the program based on the answer?

In addition to your reasons for asking the question, are there other ways that the answers could be used? If so, are these ways acceptable to your committee and to your organization?

Do you already know with confidence the answer to the question? (e.g., is the answer well established in literature or through prior experience?)

Program Evaluation Planning Primer | 5

Timeline for Program Evaluation

In addition to determining the scope of the evaluation, it is also important to establish a realistic timeline for gathering the data. Some of your program processes and objectives can be accomplished quickly, while others may take a much longer period. Because one of your objectives for program evaluation is to continue to engage your executive champions, it is very important that your evaluation plan include short-, intermediate-, and long-term measures; that realistic timeframes be projected for the given measures; and that incremental progress is communicated.

TIP: When planning your program’s evaluation, think about how the information may be used in decision making. Focus on the most important parts of the program; only evaluate the elements of your program that you are able and willing to change.

4. Gather Credible Evidence After you have determined the focus of the evaluation, you will need to gather credible evidence to answer your evaluation questions. This process is similar to the one you used during the assessment phase (Step 3: Assess Your Site, sometimes known as “formative evaluation”) of your Mother-Friendly program planning. You will need to consider:

Who will answer the questions?

How (by what method) will the questions be answered?

What is the data source?

When and how will the data be collected, collated, and analyzed?

Who, what, when, where, why, and how will data be reported?

Resources for considering methods of evaluating a worksite lactation support program:

Investing in Workplace Breastfeeding Programs and Policies, Chapter 6: Methods of Measuring SuccessNational Business Group on HealthThis chapter describes simple suggestions for measuring your worksite lactation support program’s success.http://www.businessgrouphealth.org/benefitstopics/breastfeeding.cfm

Lactation Support Program EvaluationCenters for Disease Control and Prevention, Healthier Worksite InitiativeThis website provides guidance and ideas for evaluating your worksite lactation support policy, breastfeeding support services, lactation room use, and the overall program.http://www.cdc.gov/nccdphp/dnpao/hwi/toolkits/lactation/evaluating.htm

6 | Program Evaluation Planning Primer

Tools for measuring components of a worksite lactation support program

Feedback Form for Supervisors and ColleaguesNational Business Group on HealthUse this feedback form to gather feedback from supervisors and colleagues. http://www.businessgrouphealth.org/healthtopics/breastfeeding/docs/BF_toolkit_sect_8-%20Lact%20Supp%20Supervisors.pdf

In addition to tools listed above, many of the assessment tools used in Step 3: Assess Your Site may also be used—as-is or in a modified format—as program evaluation tools. You have already assessed a baseline or current status with these tools, and you may use them after the program has been implemented to measure resulting changes.

Good record collection and retention during your program implementation will allow you to reference administrative records and activity logs and evaluation sheets (such as pre- and post-surveys) from program activities. Likewise, good evaluation planning will help guide you in determining the type, frequency, and amount of administrative documentation you will need to collect as you implement the program.

5. Justify Conclusions During this step, you will analyze, interpret, and develop reports from your evaluation methods. This is the time for you to compare your results to your performance indicators (the standards you set for success) to see how successful you were. You will want to compile the data you have gathered and critically examine it with varying perspectives to consider any alternative explanations for your results. Report your methods, findings, and any alternative explanations in a format that can be easily understood—and used—by the program’s stakeholders.

6. Ensure Use and Share Lessons Learned It is not sufficient to collect and analyze evaluation data. The information you have synthesized and lessons learned must be communicated to the stakeholders whom you have identified during your evaluation planning, and the information should then be used to lead to further action.

Use your results to improve the program as it is implemented.

Promote and build upon your successes.

If results weren’t positive, examine why they weren’t. Is further qualitative assessment needed to identify what factors were in play?

Work with your committee to strategize whether a different approach is needed to achieve your intended outcomes.

Program Evaluation Planning Primer | 7

Whether or not your results were as intended, share your findings to contribute to our collective knowledge about what is and is not effective for programs of worksite lactation support. Consider:

o Emailing or calling the Texas Mother-Friendly Worksite Program or posting the lessons you learned at www.TexasMotherFriendly.org.

o Submitting an article for a professional journal or trade magazine.

o Presenting specifics about your program at a professional meeting, conference, or tradeshow of your peers.

The Texas Mother-Friendly Worksite initiative would appreciate hearing from you about the successes and challenges of your program. By learning about what worked, what didn’t, and why, we can continue to improve our services to you. Our objective is that through increased access to quality, evidence-based worksite lactation support programs like yours, barriers to breastfeeding will be reduced for Texas families. With improved breastfeeding outcomes, Texas employees and their families can experience greater health and productivity throughout their lifetime.

8 | Program Evaluation Planning Primer