Making the Most of Your Data: Strategies for Evaluating Your Program

Greta Colombi, NDTAC; and John McLaughlin, ED

2

How Do You Use Your Title I, Part D, Data?

Do you: Look at your Consolidated State Performance Reports

(CSPRs) each year?

Incorporate what you know into applications for funding?

Make statements about how these tables do not really reflect what goes on in the classroom?

Attempt to explain them to your stakeholders?

Put results away in your files?

Hope that the CSPR will go away before next year?

Why using data is important

How data can be used

Strategies for using data

Activities

3

Agenda

Consider ways you can better use your data

Consider what technical assistance (TA) and support you could provide to your subgrantees to encourage data use

4

Outcomes

5

Data Tells You and Your Programs . . .

Where you’ve been

Where you are

Where you’re going

How to get there

6

Barriers to Using Data

Your program’s data are handled separately from your program.

Your program’s culture does not focus on data.

Gathering data is perceived to be a waste of time.

Staff lack adequate orientation and training in the value of data collection.

Staff have had negative experiences with data collection.

Staff are not aware of other programs’ successes in using data.

Staff think that data are collected “just for the State or the Feds.”

7

Working With What You Have

The same data you collect and report…

Demographics of students (race/ethnicity, age, and gender) Academic performance in reading and mathematics

Academic and vocational outcomes Student and facility counts

Program spending*

can be used for… Accountability Program promotion/marketing Program management and improvement

* States do not report program spending within the CSPR, but should have this information at hand.

8

Functions of Data

Help us identify whether goals are being met (accountability)

Tell our departments, delegates, and communities about the value of our programs and the return on their investments (marketing)

Help us replace hunches and hypotheses with facts concerning the changes that are needed (program management and improvement)

Help us identify root causes of problems (program management and improvement)

9

Program Components by Data Function

Program Accountability

Program Marketing/ Promotion

Program Improvement

Student demographics

Are the appropriate students being

served?

How are you addressing the

needs of diverse learners?

Which students need to be better

served?

Student achievement

Are students learning?

What are students learning? What gains have they

made?

How can we help improve student achievement?

Student academic outcomes

Are students continuing their

education?

What are students doing to continue their education?

How can we help improve student

academic outcomes?

10

Strategies for Improving Data Use

Accountability

– Monitor data based on national benchmarks

– Set State benchmarks and monitor program performance

Program Improvement

– Evaluate program (formative and/or summative)

Marketing

– Develop and distribute State/program report cards

11

Data Use Improvement Activities

Meet with SEA staff (data, programmatic) to analyze the data you have

Request disaggregated data from subgrantees/ programs to improve data use

Communicate findings with subgrantees

Support subgrantee/program evaluations

– Communicate allowability of funding

– Include evaluation requirements in program applications/formal agreements

– Provide TA at conferences/meetings or during monitoring on benefits of using data and how to do so

12

Data Use Model

Two Components of the Model:

Data Analysis

and

Program Improvement

Model developed by the National Reporting System for Adult Education support project at the American Institutes for Research

13

14

Focusing the Question

Break the question into inputs and outcomes: Inputs (what your program contributes):

Teacher education, experience, full-time/part-time

Instructional curriculum

Hours of instruction per week

Outcomes (indicators of results): Improved posttest scores

Completed high school

Earned GED credentials

15

Focusing/Refining the Question (1)

Poor Question:

Does my program have good teachers?

Good Question:

Does student learning differ by teacher?

Better Question:

Do students in classes taught by instructors who have more teaching experience have higher test scores than those taught by new teachers?

16

Focusing/Refining the Question (2)

Poor Question:

Is my program helping the most needy students?

Good Question:

Are students who are below grade level learning less in my program than other students?

Better Question:

Are students who are below grade level advancing levels at the same rate as students at grade level?

17

Developing a Data Analysis Plan

What data do you already have that will answer your question?

What additional data, if any, will you need to answer your question?

If so, where will you get the additional data?

What’s your plan for obtaining the data you need—and what’s your timeline?

18

Analyzing and Interpreting Your Data

Keep your original question in mind.

Look for patterns and differences.

Use appropriate data and statistics.

Disaggregate the data.

Consider data quality.

Draw appropriate conclusion(s).

Remember serendipity: Be open to the unexpected.

19

Presenting Your Data (1)

Frequency Tables

Show numbers and percentages by category, e.g., ethnicity, gender, age.

Provide crosstabulations, e.g., ethnicity by age.

20

Presenting Your Data (2)

Graphs and Charts

Bar Chart: Categories are displayed as bars, e.g., students

by age.

Pie Chart: A slice of the pie shows proportion of the

whole, e.g., various ethnicities of total students.

Line Chart: Data form a continuous measure/trend (not

categories), e.g., posttest scores.

21

Presenting Your Data (3)

Communication Strategies

Article by education reporter in local newspaper

Public meeting or news conference presented by superintendent or dean

Newsletters Special events, e.g., open house Web sites Annual report

22

Conclusion

Using your data can help you (1) ensure accountability, (2) make program improvements, and (3) market your program.

It is key to look at your data, involve others, and consider how you can use the data.

You can integrate data use activities in regularly scheduled activities.

23

Activities

Activity 1

– Discuss possible reasons for the scenarios included in the handout

– Consider:

(1) What additional data would you need to better understand the root cause of the problem? and

(2) What could be done about it?

24

Activities

Activity 2

– Review the scenario

– Consider how you could address the issue(s) in regularly scheduled activities