general guidelines - milwaukeerecreation.netmilwaukeerecreation.net/mps-recreation/resources/... ·...

MPS Partnership for the Arts and HumanitiesOutcome Data Collection Guide

October 2014

Instrument ContentsGeneral Guidelines.....................................................................................................5

Choosing a method.................................................................................................5Timing and who to assess.......................................................................................5

Guidelines for administering self-report surveys or short answer questions............6

Guidelines for observations.......................................................................................8

Increase in self-discipline and self-direction.............................................................9

Increased understanding of one’s responsibility to and impact on the wider

community...............................................................................................................14

Increased ability to communicate one’s self through a specific medium (such as

visual art, writing, performance, etc.).....................................................................16

Increased understanding of differing cultures and how they connect to one’s own

................................................................................................................................. 18

Increase in self-confidence to perform a task..........................................................20

Increased ability to reflect upon one’s own abilities and performance...................23

Increased ability to assess situations from multiple perspectives and consider

multiple solutions to one problem...........................................................................26

Increased cooperative and collaborative behaviors to successfully complete a

shared task...............................................................................................................29

Frequently Asked Questions....................................................................................31

Wording of surveys...............................................................................................31Scale items............................................................................................................31Youth having a “bad day”.....................................................................................31We have existing outcome measures....................................................................31I saw change in my students last year, but surveys didn’t pick it up...................31Multiple sessions and/or programs.......................................................................32

2

Matching students on surveys..............................................................................32Outcome achievement..........................................................................................32Ceiling effect.........................................................................................................32

3

General Guidelines

The Center for Urban Initiatives and Research at UWM developed indicators and measurement tools to help assure consistent program outcome measurement across sites. The following guide provides suggestions for facilitating appropriate, simple, and reliable data collection.

Choosing a method All instruments are intended to assess each student individually.

It is strongly recommended that agencies choose only one method to measure outcome indicators. Three methods for collecting data are provided for each outcome:

o Close-ended survey questions Appropriate for intermediate and advanced readers

(approximately 7th grade and above). To be administered via paper and pencil.

o Open-ended short answers Can be administered via paper and pencil or through oral

interviews, where the interviewer takes careful notes. Appropriate for approximately grade 3 and above

(interviews) or 7th grade and above (paper and pencil).

o Observations Appropriate primarily for younger students. Some items may require brief inquiry, but most do not.

Timing and who to assess The pre should be completed near the beginning of your program, but

not during the first few days while everyone is getting settled and acclimated.

The post should be completed as near the end of the program as possible.

Only analyze results from those students who were in your program long enough to reasonably expect they would achieve the outcome (e.g., 50% or more attendance).

4

o You might consider only doing post-tests with this group and not assessing the other students.

Guidelines for administering self-report surveys or short answer questions

Create a code that will allow you to match the pre- to the post-test for self-report surveys or short answer questions. You could try a combination of the following:

o First initialo Last initialo Month borno Day borno “B” for boy or “G” for girl

You may modify the survey or short answer questions:o Specifics may need to be added about the art medium or to be

more relevant (e.g., change generic “task” to the relevant activity).

o Some questions may not be applicable.o Some vocabulary might not be appropriate for your students.o You may add other questions to measure other outcomes your

agency is interested in.

You may modify survey response categories to be more age-appropriate. Some examples include:

o Strongly Disagree, Disagree, Agree, Strongly Agreeo NO!, no, yes, YES!o Not like me, A little like me, Usually like me, A lot like meo

For surveys or written short-answer instruments, the following procedure is recommended:

Schedule time for completing the instrument during programming.

Gather students in one area. Provide an alternative activity for students whose parents have requested their students not be involved and/or for when students are finished.

5

Tell students that you are collecting information that will be used to help explain the impact of arts programming on students in the organization’s program.

Inform students that their participation is voluntary and in no way affects their participation in the program. Anyone who does not want to answer a particular question may skip that question, and those students who do not want to complete the instrument may turn in a blank document.

Explain that they should not write their names anywhere on the instrument. Explain how to create a unique identifier using the formula (see above). Go over each part of the identifier together to help assure accuracy and completion.

Then explain that there are no “right” answers, only answers that are close to how they feel and think. Ask them to choose the answers that are most like them.

Stay in an area where you cannot see responses so students don’t feel any pressure to answer a certain way.

Providing help with questions:o Depending on reading ability level or English comprehension,

some may need help with understanding words. You may assist a youth by explaining what a word means, providing examples or using a synonym.

o Sometimes youth will not know what to include or not include in their response, which can make it harder to give help. You may have to put it back on the youth to decide if they want to respond or not.

o If a student is requiring extensive help, it may not be appropriate to continue. If the amount of help you need to give is enough that the questions are being altered, their responses will not be a relevant comparison.

Have students place completed (or blank) instruments in an envelope at the front of the room.

Remember to thank them when data collection is complete.

6

Guidelines for observations Observations are labor-intensive for staff, as you will need time to

observe each student individually. Keep this in mind when scheduling observations.

Do not refer to pre-test observations when completing the post-test. We suggest you not even have the pre-test scores with you.

Observation instruments use frequency. Guidelines for frequency:o Never or seldom = less than 10% of the time.o Occasionally = between 10% and 25% of the time.o Often = between 26% and 75% of the time.o Almost always = more than 75% of the time.

Remember, CUIR researchers are available to provide guidance as you facilitate data collection!

7

Increase in self-discipline and self-direction

Note: Self-discipline and self-direction are closely related, but not necessarily co-emergent. Arts and humanities program providers will need to determine if one or both are appropriate, given your program activities and emphasis.

Survey ItemsSelf-disciplineStudent IDProgram/Session NameDatePlease choose the response that is most like you.

Not like me at all

A little like me

Mostly like me

A lot like me

When problems get in the way, I still stick to my goals. ○ ○ ○ ○When I make plans, I follow through on them. ○ ○ ○ ○I have problems focusing on completing tasks. ○ ○ ○ ○I put off doing things I WANT to do in order to finish what I NEED to get done.

○ ○ ○ ○Scoring—Circle the number and add (student should not see this)When problems get in the way, I still stick to my goals

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

When I make plans, I follow through on them.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I have problems focusing on completing tasks.

Not like me at all4

A little like me3

Mostly like me2

A lot like me1

I put off doing things I WANT to do in order to finish what I NEED to get done.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

8

Total: _________

Self-directionStudent IDProgram/Session NameDatePlease choose the response that is most like you.

Not like me at all

A little like me

Mostly like me

A lot like me

I have goals and plans for the future. ○ ○ ○ ○I am good at planning ahead. ○ ○ ○ ○I can solve most problems on my own. ○ ○ ○ ○I am good at organizing my time. ○ ○ ○ ○Scoring—Circle the number and add (student should not see this)I have goals and plans for the future.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I am good at planning ahead.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I can solve most problems on my own.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I am good at organizing my time.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

Total: _________

9

Short-Answer Questions (written or oral)

Self-discipline

Student ID: ______________Program/Session Name: _________________Date: ______________

Think about a project or activity you recently completed. Name the steps you had to take to complete it, any challenges you faced, and how you worked through the challenges.

Scoring—Circle the rating that best fits the response.(student should not see this)Cannot name any steps or challenges. 1Can name at least one step or challenge. 2Can name at least one step and one challenge and how it was overcome.

3

Can name more than one step and at least one challenge and how it was overcome.

4

Self-directionStudent ID: ______________Program/Session Name: _________________Date: ______________

Think of a goal you have for your future—it could be a goal for school, in sports or arts, or another goal you have for yourself. Name the goal you have in mind and the steps you need to take to reach your goal.

10

Scoring—Circle the rating that best fits the response.(student should not see this)Cannot name a realistic goal. 1Can name a goal, but no steps. 2Can name a goal and one step. 3Can name a goal and more than one step to achieve it. 4

Observations

Self-disciplineCircle One Pre-test Post-testStudent NameProgram/Session NameDateCheck one box per row for each behavior observed on the date above.

Never/ Seldom

Occasionally

Often Almost always

1. Stays on task. □ □ □ □2. Completes tasks. □ □ □ □3. Has disruptive behavior. □ □ □ □Scoring—Circle the number and add1. Stays on task. Never/

Seldom1

Occasionally2

Often3

Almost always4

2. Completes tasks. Never/ Seldom1

Occasionally2

Often3

Almost always4

3. Has disruptive behavior. Never/ Seldom

Occasion Often

Almost always

11

4 ally3

2 1

Total: _________

Self-directionCircle One Pre-test Post-testStudent NameProgram/Session NameDateCheck one box for each behavior observed on the date above.

Never/ Seldom

Occasionally

Often Almost always

1. Starts projects without needing direction or very little direction.

□ □ □ □2. Tries to solve problems without asking for help. □ □ □ □Scoring—Circle the number and add1. Starts projects without needing direction or very little direction.

Never/ Seldom1

Occasionally2

Often3

Almost always4

2. Tries to solve problems without asking for help.

Never/ Seldom1

Occasionally2

Often3

Almost always4

12

Total: _________

13

Increased understanding of one’s responsibility to and impact on the wider community

Survey ItemsStudent IDProgram/Session NameDatePlease choose the response that is most like you.

Not like me at all

A little like me

Mostly like me

A lot like me

I speak up for other people when they need help. ○ ○ ○ ○I believe that I can make a difference in my neighborhood. ○ ○ ○ ○I understand how I can contribute to my neighborhood. ○ ○ ○ ○I feel connected to my neighborhood. ○ ○ ○ ○I am important to my neighborhood. ○ ○ ○ ○Being actively involved in neighborhood issues is everyone’s responsibility, including mine.

○ ○ ○ ○

Scoring—Circle the number and add (student should not see this)I speak up for other people when they need help.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I believe that I can make a difference in my neighborhood.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I understand how I can contribute to my neighborhood.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I feel connected to my neighborhood.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I am important to my neighborhood.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

14

Being actively involved in neighborhood issues is everyone’s responsibility, including mine.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

Total: _________



Name an issue you feel strongly about that impacts others. What could you do to make the situation better?

Scoring—Circle the rating that best fits the response.(student should not see this)Cannot name an issue. 1Can name an issue but no solution OR can name an issue but not how it impacts others.

2

Can name an issue and either how it impacts others OR at least one solution.

3

Can name an issue, how it impacts others, and at least one solution.

4


Do you think the decisions you make and actions you take impact others? Why or why not?

Scoring—Circle the rating that best fits the response.(student should not see this)Does not think decisions impact others. 1Thinks decisions impact others but cannot say why. 2Thinks decisions impact others and can give one reason or example.

3

Thinks decisions impact others and can give more than one reason or example.

4

15

Observations (N/A)

Increased ability to communicate one’s self through a specific medium (such as visual art, writing, performance, etc.)

Note that the specific art medium needs to be specified where [brackets] indicate.

Survey Items

Student IDProgram/Session NameDatePlease choose the response that is most like you.

Not like me at all

A little like me

Mostly like me

A lot like me

I can express what I’m thinking through [art medium]. ○ ○ ○ ○When I see someone’s [art project], it tells me a lot about them.

○ ○ ○ ○When I’m doing [art project], I try to make it look like something else I’ve seen before.

○ ○ ○ ○

Scoring—Circle the number and add (student should not see this)I can express what I’m thinking through [art medium].

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

16

When I see someone’s [art project], it tells me a lot about them.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

When I’m doing [art project], I try to make it look like something else I’ve seen before.

Not like me at all4

A little like me3

Mostly like me2

A lot like me1

Total: _________



If you were going to do a [art medium] project, what would you do and why?

Scoring—Circle the rating that best fits the response.(student should not see this)Cannot identify a project. 1Can identify a project, but no reasons are given. 2Can identify a project, and one reason. 3Can identify a project, and at least one reason involves self-expression.

4

17

Observations

This observation will require inquiry.

Ask the student to explain the choices he/she made when completing [art project].

Cannot identify any choices. 1Can identify at least one choice, but cannot provide one reason.

2

Can identify at least one choice and provide one reason. 3Can identify a more than one choice and reasons. 4

18

Increased understanding of differing cultures and how they connect to one’s own

Survey Items


Not like me at all

A little like me

Mostly like me

A lot like me

I feel comfortable being around people who sound or look different than me.

○ ○ ○ ○Art or dance from other cultures really interests me. ○ ○ ○ ○I think other cultures are just as important as mine. ○ ○ ○ ○I think of myself as being part of a culture. ○ ○ ○ ○Scoring—Circle the number and add (student should not see this)I feel comfortable being around people who sound or look different than me.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

Art or dance from other cultures really interests me.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I think other cultures are just as important as mine.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I think of myself as being part of a culture.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

Total: _________

19



Describe a difference between your culture and another culture. Is one way better than the other? Why or why not?

Scoring—Circle the rating that best fits the response.(student should not see this)Cannot name a difference. 1Can name a difference, but cannot provide a reason for their opinion.

2

Can name a difference, can provide one or more reasons for their opinion, but shows bias.

3

Can name a difference and provides one or more reasons that show respect.

4

Observations

Circle One Pre-test Post-testStudent NameProgram/Session NameDateCheck one box per row for each behavior observed on the date above.

Never/ Seldom

Occasionally

Often Almost always

1. Comfortable working with students who look or sound different.

□ □ □ □2. Bullies students who look or sound different. □ □ □ □Scoring—Circle the number and add1. Comfortable working with students who look or sound different.

Never/ Seldom1

Occasionally

Often3

Almost always4

20

22. Bullies students who look or sound different.

Never/ Seldom4

Occasionally3

Often2

Almost always1

Total: _________

Increase in self-confidence to perform a task.



Not like me at all

A little like me

Mostly like me

A lot like me

I know how to prepare for an [art medium] demonstration or performance.

○ ○ ○ ○I would be comfortable helping younger or less experienced people with their [art medium] project.

○ ○ ○ ○I am comfortable [“performing” or “showing my artwork”] in front of others.

○ ○ ○ ○I know how to [insert task] from beginning to end. ○ ○ ○ ○I know I can finish anything I start. ○ ○ ○ ○Scoring—Circle the number and add (student should not see this)I know how to prepare for an [art medium] demonstration

Not like me at all

A little like me

Mostly like me

A lot like me

21

or performance. 1 2 3 4I would be comfortable helping younger or less experienced people with their [art medium] project.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I am comfortable [“performing” or “showing my artwork”] in front of others.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I know how to [insert task] from beginning to end.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I know I can finish anything I start.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

Total: _________



If you were going to [perform a task or project], how would you start? What would you do next?

Scoring—Circle the rating that best fits the response.(student should not see this)Is not clear on how to begin. 1Can provide a step in the project or task, but not how it begins.

2

Can describe an appropriate beginning. 3Can describe how to start and at least one more step in the process.

4

Student ID: ______________Program/Session Name: _________________

22

Date: ______________

Are you comfortable [performing task]? Why or why not?

Scoring—Circle the rating that best fits the response.(student should not see this)Not comfortable. 1At least some comfort, but no reasons provided. 2At least some comfort, with one reason provided. 3At least some comfort, with more than one reason provided.

4

Observations

Circle One Pre-test Post-testStudent NameProgram/Session NameDate

Observe the student performing the task and choose the category that best fits.

Student does not seem able to perform task on his/her own OR performs the task, but in a consistently incorrect manner.

1

For the most part, student seems unsure of how to perform the task.

2

23

Student seems confident in performing the task with minimal direction.

3

Student is very confident performing the task and even acts as a leader for others.

4


Observe the student performing the task or the product of a project and choose the category that best fits.

Student is willing to take risks in work.

Never/ Seldom1

Occasionally2

Often3

Almost always4

24

Increased ability to reflect upon one’s own abilities and performance



Not like me at all

A little like me

Mostly like me

A lot like me

I know where I could improve my skills in [arts or task]. ○ ○ ○ ○When I complete a task or assignment, I think about how I could do it differently next time.

○ ○ ○ ○I am comfortable thinking about how well I can [insert skill or task].

○ ○ ○ ○If my [insert performance or project] is not going well, I can do something about it.

○ ○ ○ ○I know which parts of [insert performance or task] are easy for me.

○ ○ ○ ○Scoring—Circle the number and add (student should not see this)I know where I could improve my skills in [arts or task].

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

When I complete a task or assignment, I think about how I could do it differently next time.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I am comfortable thinking about how well I can [insert skill or task].

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

If my [insert performance or project] is not going well, I

Not like me at all

A little like me

Mostly like me

A lot like me

25

can do something about it. 1 2 3 4I know which parts of [insert performance or task] are easy for me.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

Total: _________Short-Answer Questions (written or oral)


Think about a [task or performance] you did. Name one thing you did well and one thing you can improve on next time.

Scoring—Circle the rating that best fits the response.(student should not see this)Cannot name a positive or negative aspect of performance.

1

Can name either a positive or negative aspect, but not both.

2

Can name both a positive and negative aspect of performance, but not much detail is provided.

3

Can name both a positive and negative aspect of performance and provides detail.

4


How would you know if your [performance, art piece, project] is successful or not?

Scoring—Circle the rating that best fits the response.(student should not see this)Cannot provide criteria. 1

26

Criteria are provided but are subjective in nature (e.g., “I like it.”)

2

At least one objective criterion is provided. 3More than one objective criteria are provided. 4

Observations

Use with caution, as self-reflection is an internal process.


Observe the student performing the task and choose the category that best fits.

Student does not demonstrate awareness of his/her own skills or performance.

1

Student is minimally aware of his/her own skills and performance.

2

Student is somewhat aware of his/her own skills and performance.

3

Student is very aware of his/her own skills and performance.

4

27

Increased ability to assess situations from multiple perspectives and consider multiple solutions to one problem

Survey Items


Not like me at all

A little like me

Mostly like me

A lot like me

When I make a decision, I try to think about how others will be affected.

○ ○ ○ ○There are many ways to solve the same problem. ○ ○ ○ ○When I’m working on a project, I’m willing to try something new. ○ ○ ○ ○I can learn a lot from people whose experiences are different from mine.

○ ○ ○ ○Scoring—Circle the number and add (student should not see this)When I make a decision, I try to think about how others will be affected.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

There are many ways to solve the same problem.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

When I’m working on a project, I’m willing to try something new.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I can learn a lot from people whose experiences are different from mine.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

Total: _________

29



Tell me about a time when you were stuck on a project and what you did about it.

Scoring—Circle the rating that best fits the response.(student should not see this)Cannot name a challenge encountered. 1Can name a challenge, but not how it was addressed. 2Can name a challenge and one solution. 3Can name a challenge and describes considering multiple solutions.

4


Is there one right way to [insert art medium or project]? Why or why not?

Scoring—Circle the rating that best fits the response.(student should not see this)Believes there is one right way. 1Believes there is more than one right way, but cannot provide reasoning.

2

Believes there is more than one right way, and can provide one reason or other perspective.

3

30

Believes there is more than one right way, and can provide multiple reasons or perspectives.

4

Observations


Never/ Seldom

Occasionally

Often Almost always

1. Generates ideas easily. □ □ □ □2. Asks questions about other people’s work or ideas—shows curiosity about other perspectives.

□ □ □ □3. Listens to others’ suggestions about how to improve performance or alternatives.

□ □ □ □

Scoring—Circle the number and add Never/ Seldom1

Occasionally2

Often3

Almost always4

1. Generates ideas easily. □ □ □ □2. Asks questions about other people’s work or ideas □ □ □ □

31

—shows curiosity about other perspectives.3. Listens to others’ suggestions about how to improve performance or alternatives.

□ □ □ □

Total: _________

32

Increased cooperative and collaborative behaviors to successfully complete a shared task

Survey Items


Not like me at all

A little like me

Mostly like me

A lot like me

I like to do my part when working in a group. ○ ○ ○ ○Others in my group count on me. ○ ○ ○ ○I can learn more if I work in a group than if I work on my own. ○ ○ ○ ○When you are working with others, you can’t always expect to get your way.

○ ○ ○ ○What I do affects others in the group. ○ ○ ○ ○Scoring—Circle the number and add (student should not see this)I like to do my part when working in a group.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

Others in my group count on me.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

I can learn more if I work in a group than if I work on my own.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

When you are working with others, you can’t always expect to get your way.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

What I do affects others in the group.

Not like me at all1

A little like me2

Mostly like me3

A lot like me4

Total: _________

33



What are the advantages of doing a project with others rather than on your own?

Scoring—Circle the rating that best fits the response.(student should not see this)Cannot name a difference between individual and group work.

1

Can name a difference, but only reflecting negative attitudes toward group work.

2

Can name one advantage of group work compared to individual work.

3

Can name more than one advantage of group work compared to individual work.

4

Observations


Never/ Seldom

Occasionally

Often Almost always

1. Compromises. □ □ □ □2. Participates with effort in group activities. □ □ □ □3. Demonstrates affirming behavior by openly showing support for others’ contributions and abilities.

□ □ □ □

34

Scoring—Circle the number and add1. Compromises. Never/

Seldom1

Occasionally2

Often3

Almost always4

2. Participates with effort in group activities.

Never/ Seldom1

Occasionally2

Often3

Almost always4

3. Demonstrates affirming behavior by openly showing support for others’ contributions and abilities.

Never/ Seldom1

Occasionally2

Often3

Almost always4

Total: _________

Frequently Asked QuestionsWording of surveys

Q: The surveys are not age-appropriate or the surveys are not easily understood by students. What should we do?

A: In the short-term, you can modify the survey to make it more relevant to your program (e.g., replace task with an actual activity) and substitute words that have the same meaning (e.g., “neighborhood” for “community”). Please note: wording should not change the meaning of the item. When in doubt, please contact the UW-Milwaukee Center for Urban Initiatives and Research (CUIR).

In the long-term, we aim to have a group revise surveys to be more appropriate and relevant for Milwaukee youth.

Scale items

Q: The provided scale items of “Not like me” to “A lot like me” do not work for our program. What should we do?

A: See page 6 for alternatives.

35

Youth having a “bad day”

Q: Sometimes our youth have really bad days and it affects their assessment. Help!

A: If a youth is clearly not acting in typical ways due to a “bad day,” consider assessing them another time. But do not wait for the “perfect” mood before assessing! Just use this option if someone is obviously behaving in ways that are out of character; otherwise, go ahead and assess.

We have existing outcome measures

Q: Is it acceptable to use our existing outcome measurement tools?

A: If those tools relate directly to the outcome you selected, it is generally acceptable to use another tool. However, keep in mind that these should be summative (outcome-focused), as opposed to formative (process-related), tools. When in doubt, please contact CUIR.

I saw change in my students last year, but surveys didn’t pick it up

Q: Can I just report my observations of whether students achieved the outcome?

A: You may use observations rather than surveys, but use the provided instrument. We need to be precise about what kind of change happened and how much. Also, it’s human nature to rate a student high when we feel positive about what they’ve accomplished, regardless of the criteria tied to a particular outcome. The observation instrument has clear scoring guidelines to help data be reliable and less subject to (understandable) bias.

Multiple sessions and/or programs Q: Our program happens over multiple sessions and/or we have multiple programs. How can this process work for us?

A: This issue is very much case by case, but in general, if your session/programs are attended by the same students, it is advised that you assess once in the beginning (pre) and once at the very end (post). If attendees are different, for instance you have a spring program and a summer program, it is advised that you complete the pre/post cycle twice—once for each separate group of students.

36

Matching students on surveys

Q: We have trouble matching pre/post surveys. Is there something we can do?

A: Because attendance varies in out-of-school time programming, there will naturally be unmatched surveys. However, to help match the maximum number of surveys, we suggest going through each matching variable together as a group each time the survey is administered. For instance, you might say, “Everyone write their first initial; now you should write a ‘B’ for boy or ‘G’ for girl…” This will help the matching information be accurate and complete.

Outcome achievement

Q: What counts as outcome achievement?

A: Remember that all outcomes are stated in terms of an increase or improvement. If a student raised their score on the assessment tool on the post-test compared with the matched pre-test, this student achieved the outcome. If the student maintained their score, this does not count as an improvement or increase.

Ceiling effect

Q: We think our students are exhibiting a “ceiling effect,” where they enter our program at a high level of achievement. What can we do?

A: First, in order for us to determine whether there is a true ceiling effect, we need more data. At this point, it’s unclear how widespread this is, whether it’s limited to certain tools, and/or whether this is related to age. We hope to have enough data during this year to conduct validity and reliability analyses.

However, if your experience is that students consistently rate at the top for your chosen outcome at the beginning of programming, consider that a different outcome or level of outcome (intermediate vs. short-term) might be more appropriate.

37

general guidelines - milwaukeerecreation.netmilwaukeerecreation.net/mps-recreation/resources/... ·...

Documents