THIS SECTION IS VERY IMPORTANT: IT FOCUSES ON

MEASUREMENT AND THE LOGIC OF THE PROGRAM. IT ENDS WITH A DIAGRAM THAT YOU WILL USE

BUT WHAT IS EVALUATED????

MUCH OF THE FOLLOWING IS TAKEN DIRECTLY FROM THE WEINBACH ONLINE READING

Mission, goals and objectives

These describe the purpose and focus of the program. Keep in mind that every program addresses specific problems with specific populations. However there is usually a focus to or emphasis on how they address it. While mission and goals state the overall purpose and focus, the objectives state what should happen specifically to the client as a result of the program; it should state what clients who have been successful in the program should now be doing. Objectives state the specific results the program hopes to achieve

Inputs Inputs are the resources expended for a program. Every program has its costs.

Often, inputs are a combination of resources contributed from outside the organi zation and from the organization that houses the program. The input that comes from the program’s organization is often referred to as the “match,” and reflects its commitment to, and willingness to share in, the expenses of the program. Many inputs of a program are reflected in its budget; some are less easily expressed in monetary terms. Many are tangible; others are less tangible.

Some inputs are standard in all programs; others are unique to a program or at least to a particular kind of program. Examples of inputs in all programs would be funding, staff (paid and volunteer), and facilities—a program needs money and people to run it and it has to be located somewhere. When we get much beyond those three, inputs tend to depend on the program. For example, in one after-school program, inputs might include art supplies, printed educational materials, transportation, first-aid materials, and parent volunteers. In another after-school program, computer software and sports equipment might constitute additional inputs, but art supplies, transportation, and parent volunteers might not.

21

Constraints Very few, if any, programs are totally free to do whatever they wish. Their constraints take many

forms, including laws, mandates, regulations, rules, procedures, or funding requirements. They share a common characteristic—they limit the nature and extent of a program. In a less rigid way, the mission of the a program is housed is often another constraint. A program needs to provide services that are consistent with the organization’s mission. For example, if the organization seeks to serve low-income families, our after-school program cannot be offered free of charge to just anyone. However, we might be able to open it to a limited number of children from more affluent homes on a pay-for-services basis, if we can justify the decision on the basis that their fees (an input) would help subsidize the cost of the program and thus make it available to more low-income children.

Many programs must meet licensure requirements. The requirements for becoming and remaining licensed are constraints. In an after-school program, there might be requirements regarding staff-child ratios, provision of meals, presence of a registered nurse or other health care professional, credentials of the program director and other staff members, and so forth. Other constraints related to health and safety might include requirements related to room temperatures, availability of potable drinking water, isolation of children with infectious diseases, and so forth.

In a program evaluation, we do not evaluate a program’s constraints—they constitute a “given.” However, fairness dictates that a program’s services, what it offers or does not offer, its accomplishments, and so forth should be evaluated within the context of the constraints under which the program operates. Thus, good program evaluations often tend to qualify any judgments of a program with reference to its constraints.

ActivitiesActivities are what a program’s staff does. It is how it uses its inputs to try to produce desirable

outcomes. Activities are implemented in such a way that, it is believed, they will address some problem of the program’s clients, either directly or indirectly. Those activities designed to directly serve clients are its ser vices (defined earlier). However, some other activities (especially those offered by managers and administrators) may be designed to support staff members who offer those services and the program in general. They only indirectly serve clients. For example, a program’s director may be involved in activities such as public rela tions, fund raising, or recruitment of volunteers. While these activities are not client services per se, they make services possible and increase the likelihood that they will be successful. Thus, they are a focus of a program evaluation.

In an after-school program, activities of the program might include individual counseling, social-

skills-building groups, tutoring, educational field trips, or planned recreation. However, other important activities that are not direct services to clients would include publicizing the program in a variety of settings, screening of applicants, seeking corporate donations, and holding Friday after noon meetings for staff to voice their concerns about the program.

OutputsOutputs are the products of program activities. Outputs can usually be quantified. They may be the

number of units of service delivered, the number of clients served, the number of educational pamphlets distributed by a program, and so forth. Out puts can be used to suggest the efficiency of a program or to compare the efficiency of two programs. The more outputs a program can produce (per monetary unit) the more efficient the program. For example, if program A serves 100 clients at the same cost as program B which serves 50 clients, program A (at least on the surface) appears to be twice as efficient as program B. Or, if we look only at outputs, a pro gram that had no budget increase but served 20 percent more clients this year, had a better year. Historically, outputs have been the primary indicator of program suc cess, and as Chapter 9 shows, that has been, and continues to be, a problem. Out puts in our after-school program might be the number of children who actively participated in the program, the number of field trips completed, the number of individual counseling sessions offered, or the number of children who received tutoring services.

Outcomes Outputs alone are no guarantee of program success or value. For example, just because a certain number of clients completed a program, that does not guarantee they have benefited from

it. The degree to which a program has achieved its desired outcomes is what really suggests whether a program has been successful. In most programs , a central outcome is some benefit for clients who complete or fully participate in the program.; often a change such as a reduction in or elimination of some problem that they have. In programs that serve clients whose main problems can not go away or reflect improvement (for example, people with Alzheimer ‘s disease or those who have a terminal illness) change of some form may still be a realistic outcome objective. For example, a desired outcome of the program might be improvement in some aspect of quality of life, such as increased independence or increased participation in family activities. If even an outcome like that is not real istic, an outcome objective might be to reduce or slow the onset of negative changes in the program’s clients, for example, to maintain the current level of client functioning for an extended period of time, or to slow the rate of clients’ decline in functioning. Or, it might be the prevention of some problem for all or most of the people the program is designed to benefit, for example to reduce unwanted teen pregnancy or to reduce the spread of sexually transmittable diseases.

Outcomes are the degree to which a program has benefited those who are supposed to be its beneficiaries, generally its clients. They are the primary yard stick by which program success should be measured. Examples of outcomes are changes in behavior, knowledge, attitudes, skills, values, conditions, and so forth. In our after-school program example, outcomes would be the degree to which chil dren who participated in the program benefited in some specific ways from their participation, the degree to which some present or potential problem diminished because they were enrolled. Obvious examples might be better grades or improved social skills.

Outcomes are sometimes classified as short-term, intermediate-term, and long-term. Short-term outcomes are how clients (or sometimes a larger system, such as a community) benefited directly from a program. For example, in an after-school program, other short-term outcomes might include having a safe place to be while parents are at work, having a quiet place to complete homework, learning “cooperative play,” positive use of free time, or learning positive alternatives to handling conflicts. Intermediate-term outcomes often entail continued growth and application of learning following program completion. In an after-school pro gram, intermediate-term goals might be continued grade improvement, use of positive alternatives in conflict situations, or continued growth in use of social skills. Long-term outcomes are the ultimate effects of a program that participants may carry through life. Like intermediate-term outcomes, documenting success in achieving long-term outcomes requires follow-up of participants over time (after they leave a program) and is more difficult. In an after-school program, long-term outcomes might include maintenance of good grades, completion of high school, admission to a college of choice, avoidance of substance abuse problems, lack of legal difficulties, and a high level of social functioning.

Impacts Impacts are sometimes synonymous long-term outcomes. In describing how a

program affected its clients over this is pro ably c rect. However, the impact of a program can extend beyond those effects experienced by just those clients who participated in a program. For example, some programs, while helping a limited number of individual clients, can still manage to affect how members of society perceive their problem. A program also can affect other programs within an organization, other similar programs within the community, its staff, and so forth. Impacts can be either positive or negative. For example, an after-school program could reduce the need for parent-teacher conferences, could increase attendance at school social activities, could reduce the number of children in nearby religious instruction programs, or could cause a problem of staff morale among those not offered employment in it.

It is important to note that, while some impacts of a program are more or less predictable, some are not. Despite the best planning efforts, some unexpected impacts almost always occur. This should not surprise anyone who understands systems theory. No program operates in a vacuum. Its implementation will invari ably impact on the larger system in unpredictable ways, and not all of them will be desirable.

Indicators As a group, outcomes (especially intermediate and long-term ones) can be difficult to

measure, certainly more difficult than outputs. They often cannot be measured directly, so we use the next best thing, indicators. In are the measurements of behaviors, attitudes, and so forth that suggest to what come_~~ have been achieved. In evaluation research, the term indicator is used similarly to the way it is used in basic research. The degree of reliability and validity of an indicator is always a major issue. When they are used in a given situation or with a given group of people and how they are used, both affect the credibility of any conclusions of an evaluation.

For example, in an after-school program, short-term indicators might be per cent of homework completed—that would be a valid, straightforward indicator. However, indicators of, for example, knowledge of cooperative play or of positive alternatives for conflict resolution would be “softer,” measured perhaps in the form of staff observations of children’s behaviors. Similarly, the ability to find a valid indicator of achievement of intermediate and long-term outcomes would vary. For example, high school or college graduation rates would be good indica tors of academic success. However, “avoidance of negative behaviors” would be more difficult to operationalize. And, of course, with the passage of time, it would be increasingly difficult to prove that it was the program and not something else that produced the desired outcomes reflected by the selected indicators.

SOME CLARIFICATION:1. OUTPUTS ARE DIFFERENT FROM OUTCOMES

An output is about quantity; an outcome is about quality. Outputs refer to how much and how many.

For example the output of an M.S.W program is number of graduates, the outcome would be number of graduates who can competently perform a multi-dimensional assessment. Another example: number of articles published by social wo=rk faculty is an output; number of articles published in “top rate social work journals” is an outcome. See the difference? Outputs are what and who comes out of a program; outcomes are the final condition or state of what and who comes out of a program.

2. GOALS ARE DIFFERENT FROM OBJECTIVES (REVIEW)

Goals are broad objectives are narrow.Goals are general intentions; objectives are precise.Goals are intangible; objectives are tangible. Goals are abstract; objectives are concrete.Goals can't be validated as is; objectives can be validated.

Goal: knows about the human body.Objective: name all of the bones in the human body as stated in the medical textbook "The Human Body".

Objectives can be achieved through specific measurable behaviors. . Program objectives answer the question: What should clients be able to do at end of program

Examples Hospice bereavement programProgram Goal: 1. To help people with the grieving processProgram objectives:1. to identify and address risk factors associated with complicated bereavement2. to return survivors to pre-morbid level of functioning within 24 months3. to reduce unusual or grief-related isolation4. to identify and address dysfunctional coping mechanism that begin or increase during the grieving process5. to educate clients about the processes of uncomplicated bereavement and complicated bereavement

Example: Mental health program that serves people with chronic and severe diagnosesProgram Goal: 1. to help people live independent and satisfying lives in a non-stigmatizing environmentProgram objectives:1. reduce the number of psychiatric episodes that require short term alternative housing (hospital, respite care,

jail etc)2. reduce the number of medication changes for clients3. find and maintain, stable, affordable, clean housing4. identify and remediate crises in their early stages5. Clients will self-identify their own needs pro-actively6. Clients quality of life satisfaction will be at least satisfactory as indicated by

- Scores on the quality of life index - a reduction in the number of high risk contacts with social agencies (arrests, overdoses, suicide attempts,

unplanned hospitalizations, fights etc.) Objectives either suggest or describe a way in which they can be measured and also

suggest a ‘standfard of achievement’ for that measure

3. OBJECTIVES ARE DIFFERENT FROM OUTCOMES –ALTHOUGH THEY ARE RELATED.

Outcomes come out of objectives. Objectives specify what the program –specifically desires or plans to achieve. (expected results). Outcomes are the actual measures of the objectives. In other words outcomes cannot be measured until people are actually going through the program and have opportunity to achieve the objectives.

4. OUTCOMES ARE DIFFERENT FROM INDICATORS –ALTHOUGH THEY ARE RELATED,

Outcomes can and should be measured directly. They are direct measurements of objectives. Or said another way, they are the direct measurements of the program results. Sometimes there are times when outcomes may not be directly measurable.In those cases, we might measure phenomenon that –while not a direct measure of the outcome – might INDICATE that the outcome was accomplished.

Example: I have a program to reduce ‘teen pregnancy’ in highschool. However, the school (like so many) does not keep actual records on the number of pregnant teens. So I go to the literature and find out that prenant teens 1. have high rates of truancy, 2. have lower grades. 3. drop out of school at higher rates. So I would then measure these three indicators before my program, start my program then measure them again.

An indicator cannot tell me with certainty whether I have changed the problem; only that I might have!