Evaluation BasicsPurpose of Evaluation

As adults and youth design and implement their evaluation, there are several important principles that will help them be more effective…

EvaluationBasicsPrinciples of Evaluation

Planning for quality and impact (program development) helps youth leaders…

Develop programs strong and focused enough to make a difference

Develop programs with long-term impact

Develop and evaluate programs in stages, rather than trying to do “too much, too soon”

Evaluation BasicsPrinciples of Evaluation

“Dosage,” or the right combination of program activity to produce change in knowledge, skill, or behavior, requires leaders to plan programs with appropriate…

Intensity: level of difficulty and amount of content;

Frequency: sequence of learning experiences, and

Duration: length of time

…to bring about the change they aim to achieve

Evaluation BasicsPrinciples of Evaluation

The importance of intensity, frequency, and duration are illustrated in this example:

Pre-teens given 2 hrs/day of guided hands-on practice with a GPS device in a week-long Day Camp learned basic skills for operating the device; Similar peers who shared time on GPS devices did not learn basic skills

Note on intensity: Extensive guidance and

personal practice is more likely to produce results than more limited guidance and practice (but 4 hrs/day might be too much!)

Evaluation BasicsPrinciples of Evaluation

The importance of intensity, frequency, and duration are illustrated in this example:

Overweight teens who were trained and tracked to follow a weight-reduction diet, changed eating habits and decreased weight during the six-week project but did not sustain results three months later

Notes on duration:Results suggest that the

first phase was long enough to produce expected results

Whatever happened after the second phase was not enough to sustain results

Evaluation BasicsPrinciples of Evaluation

Programs are most likely to make a change in knowledge, skill, or behavior, programs through…

Evidence-based curricula, shown to produce specific results with a similar audience, or

Research-based practices, shown to promote generally positive effects with a similar audience…

…but program fidelity, or consistency with the original model is critical for achieving similar outcomes

Evaluation BasicsPrinciples of Evaluation

Key questions in choosing curricula and practices:

What is the research evidence for this program?

How does that evidence match my program goals, audience, or conditions?

Evaluation BasicsPrinciples of Evaluation

Key questions in using curricula and practices:

How closely am I following curriculum or practice guidelines (e.g., for intensity, frequency, duration)?

If I am not following the recommended guidelines do I expect the advertised results?

…program fidelity or consistency with the original model is critical for achieving similar outcomes

Evaluation BasicsPrinciples of Evaluation

Programs are most likely to alter knowledge, skill, or behavior when targeted to the learner’s readiness for change, as described in these stages:

Pre-contemplation: Never thought about changeContemplation: Aware, but not considering changePreparation: Willing to try change; experimentingAction: Trying to make changes for 3-6 monthsMaintenance: Sustaining changes beyond 6 monthsRelapse: Breakdown of change efforts; need to re-try

Evaluation BasicsPrinciples of Evaluation

A dropout prevention program targeted to the right level of change for teens interested in automobiles…

Youth may be prepared for Automotive program but have not contemplated a Plan-for-College program Positive hands-on experience with a caring, expert automotive trainer may lead to contemplation of science, technology, and math courses (and staying in school)

Evaluation BasicsPrinciples of Evaluation

Middle School youth with strong science skills may be ready for applied programs that maintain or extend knowledge and skills……but science buffs may consider a

(pre)contemplation “Making Science Fun” program too simple and boring

Two-way communication in programs is critical to balancing challenge and support and tapping youths’ readiness to learn, serve, and lead

Evaluation BasicsPrinciples of Evaluation

Planning for quality and impact (program development) helps youth leaders…

Develop long-term (high-impact) Programs rather than focus on short-term (lower-impact) activities

Evaluation BasicsPrinciples of Evaluation

A long-term, multi-faceted Program typically produces deeper and broader results than one or more isolated “programs” or activities

A year of planned science, service, and life skills experiences usually results in more learning and applied practice than a series of unplanned events

Repeated participation in the same event, such as Presentation Contest or camp promotes more learning than random participation in unrelated events

Evaluation BasicsPrinciples of Evaluation

Programs mature over time, and so should evaluations:

Newly-initiated programs benefit from Formative Evaluation tracking audience, goals, and quality

Established programs can begin Summative Evaluation including pre/post measures of outcomes

Mature programs are able to conduct more extensive evaluations of quality and outcomes, comparing results with matched groups

Evaluation BasicsPrinciples of Evaluation

Program maturation refers to the level of operation achieved in a youth program. Evaluation goals and methods vary depending on the needs and capacities of program staff, participants, and stakeholders at any given time.

Evaluation BasicsPrinciples of Evaluation

Keeping in mind the basic principles for program and evaluation success, leaders of youth programs can begin learning and using evaluation practices…