how to effectively measure and monitor progress on five...

How to Effectively Measure and Monitor Progress on Five-Year Grants

Presented by: Aaron Lieberman (Chairman of the Board);

Andrew Davis (Chief Partnership Officer)

Region V Head Start Association | October, 2015

Agenda

• Introductions & Learning Objectives

• Understanding the Requirements of 5-Year Grant Submission

• Setting Long-Term Goals

• Ongoing Monitoring and Use of Data

• Making it Real in Your Program

• Founded in 2002

• $50 million annual revenue

• 1048 employees

• Serving 5000 children in four states

Developing powerful new approaches to data, curriculum and family services approach

• Founded in 2009• $3.2 million annual revenue• 24 employees

• Intensive partnerships with over 30 Head Start programs serving more than 36,000 children

Leveraging Acelero Learning’s Head Start innovations to impact

larger early childhood community

Key Drivers of Success

• Extensive use of data– Proprietary reporting system includes weekly MBI report with 100

different summary indicators

– Monthly MBI + report with 54 different indicators

– Approach to data served as basis for development of Shine Insight data system

• Early childhood education focus on extensive coaching, intentional curriculum, and new approach to assessment

• Family services focus on leveling with families about the achievement gap and then setting outcome-based family life practice goals with every family

Introductions: About You!

• Name / Title

• Program / Children Served

• Years in Head Start

• Have you submitted your 5-year grant application already?

• What is one thing your program is focused on tracking better this school year?

• Share Out!

Today’s Goals

Desired Learning OutcomesBy the end of the session participants will be able to:• Articulate a practical understanding of how the use

of data will play a part in both setting and tracking long-term goals

• Build quantifiable measures related to each program goal, and set benchmarks / targets over the course of five years

• Articulate how quarterly check-ins on goal-related metrics and program data can enhance planning and programmatic self-assessment practices

BUILDING AND USING DATA DASHBOARDS

What’s new in Head Start

What is a data dashboard?

How can data dashboards be helpful in your program?

• Ongoing / regular tracking of compliance measures

• Structured, focused look at outcomes data for children and families

• Clear way to highlight key measures that will help you track progress toward your program goals

Manage by… Monitoring System

Manage by Information

Compliance & Process

Indicators

ShineStat

Quarterly Self-Assessment

Manage by Outcomes

Outcomes Indicators

What Makes Good Compliance Monitoring?

• Programs need a reporting tool which measures Head Start compliance indicators. Key aspects should:

� Include information by unit that is managed to and held accountable for results (i.e. classroom, or center, or zone, etc.)

� Show trends from prior month (or time period)

� Evolve over time as you use

� Be used as both a tool to identify areas that need more support and to hold people accountable

• To be effective, information must be up to date, real-time and run every other week (or monthly, at a minimum)

Types of Data to Enter, Capture, and Monitor

• ERSEA

– Current enrollment, openings, waitlist

– Eligibility, by category

• Disabilities / Mental Health

– Active IEPs / IFSPs

– Completed or incomplete or past-due screenings (Dev and SE)

• Family Services

– Completed Strength Assessments and Family Partnership Agreements

• Health

– Completed, Past-Due, Expired Health Screenings and Immunizations

– Medical / Dental Homes

• Early Learning

– Completed Home Visits and Parent/Teacher Conferences

– Completed Assessments, by checkpoint

Our Answer: Manage By Information Report

Rows represent centers and the dark purple row shows the program average or total.

Columns are numbered sequentially for reference

Historical trend data is included at the bottom.

Conditional formatting with colors help visually highlight areas.

4

3 2

1Frontline Staff / Coordinators input real-time data into

data system

Administrator(s) runs 20+ summary reports from data system and keys data into excel

spreadsheet (MBI)

MBI is distributed to wide audience

for review

Frontline Staff / Coordinators

respond to MBI issues and follow-up

Regular MBI Process (if not built into data system)

MBI Relies on Accurate, Current Data in Your Systems

If it is not in _________ (Shine Insight, Child Plus,

PROMIS, COPA, etc.)

it did not happen!

Managing by Information Not just reporting, but acting…

• Distribution of MBI report among key leaders and governance groups increases accountability for all players

• Ideally allows leadership to focus on data –not personalities – in problem solving and tracking progress

• Ultimately it is about identifying problem areas, and then making changes to improve, when key numbers indicate concerns

Basic Outcomes Tracking: The MBO

The MBI:The Limitations

• Almost exclusively focused on process indicators (in comparison to outcomes)

• Are short-term by nature – things that change every week

• Can lead you to “miss the forest for the trees” if there are no other measurement systems in place

The MBI:The Limitations

For example…

• What if we conduct all of your educational assessments, but children are leaving our program not prepared for kindergarten?

• What if we complete our family partnership agreements on time, but our families never meet those goals?

• What if we finish all our screenings on time, but never successfully help families follow-up on health issues?

What Does Good Outcomes Monitoring Look Like

Why it is needed…

• The new five-year grants structure, aligned monitoring system and new proposed performance standards place a greater emphasis on outcomes than ever before.

• For that reason, to be a GREAT Head Start program, it is NOT ENOUGH to just meet minimum standards for compliance

• The current HS Performance Standards are a floor, not a ceiling!

Our Answer: THE MBO Report

• The MBO Report allows us to:

– Set program-wide goals for key outcomes we wantto accomplish

– Manage and measure to see if we accomplish them, and how we can improve

– Review these outcome reports quarterly throughout the year, and adjust our work accordingly

Managing by OutcomesProcess vs. Outcome

Process� Completing

assessments on time

� Completing family strengths assessments

� Completing required screenings on time

Outcome� Children mastering skills

� Families achieving goals

� Children receiving treatment to help them become healthier

Table Discussion:

What will be your biggest challenge in terms of tracking outcomes in the four areas (child, family, health, special populations)?

Why?

Goals, Measures and Targets

Ctr

1:

PF

T

Ctr

2:

MLK

Ctr

3:

Ach

iev

e

How can this translate to tracking progress on five-year goals?

5 Year Grants Preview: Why does this matter?

Goals• Broad, inspirational statements that

describe what you seek to accomplish; targets to be reached.

Objectives

• Subparts of goals that are Specific, Measureable, Attainable, Realistic and Timely (= SMART).

Impacts

• The influence or effect on a specific Head Start population (e.g. staff, children, families, communities)

Creating a dashboard for your five-year goals

• What are the impacts you are looking to measure?

• What information must you collect in order to measure them?– Who is involved?

– When is the information collected?

• What units and levels are the most meaningful? (Child, classroom, center, program, etc.)

Group Exercise

Use one of the long-term goals for your program that you brought in. Discuss with your neighbor the answers to these questions:

• What impacts are you tracking?

• What types of information and systems are involved?

• When is it meaningful to collect and report / discuss this information during the year?

• What levels are the most meaningful to display? (Classroom level, Center level, region level, etc.)

• What would your ideal “dashboard” for this goal look like?

– It’s OK to include more than one area to track!

UNDERSTANDING THE REQUIREMENTS FOR NEW 5-YEAR GRANT SUBMISSIONS

What’s new in Head Start

Five Year Funding ApplicationsThe Big Picture: Baseline Applications

Section I. Program Design and Approach to Service Delivery

– Long Range Goals, Objectives and Program Impacts (4 criteria)

– Service Delivery (14 criteria)

– Approach to School Readiness (2 criteria)

– Parent, Family and Community Engagement (PFCE) (4 criteria)

– Governance, Organizational and Management Structure, and Ongoing Oversight (10 criteria)

Section II. Budget and Budget Justification (12 criteria)

Baseline Application: Long Range Goals, Objectives and Program Impacts

Specify the expected, measurable program impacts for each year of the five year period in the following areas:

– Outcomes for children and families including school readiness, health services, support for special populations, and effective parent and family engagement

Baseline Application: What has changed?

• Original application required applicants to specify measurable program impacts aligned with the long-range goals and objectives to be achieved for each year of the grant in the following areas:

– Outcomes for children and families, including school readiness, health services, support for special populations, and effective parent and family engagement

– Program and fiscal management systems; oversight and accountability

– Enhanced community involvement and resources

– Unique community and organizational goals

• Latest application (April 2015) only suggests these areas of focus – however, they kept the frame for alignment with program goals!

DEVELOPING 5-YEAR GOALS FOR CHILD OUTCOMES

Goal Development Example

5-Year Application (and Programmatic) Priorities

Year 1

• Describeprogram approach

• Defineanticipated outcomes

• Measuresof program progress

Year 2

• Focus on progresstowards goals, outcomes, and intended impacts

Year 3


Year 4


Year 5

• Provide evidence of the program’s cumulative impact on the children, families, communityGather evidence to submit in Year 5.

Example: Our ECE Approach

Structured Curriculum– Content-Rich – Intentional Teaching– Provide Scaffolds

Criterion-Based Professional Development– Self-Assessment– Coaching – Professional Learning Communities

Focused Assessment– Lean and Mean Approach– Ongoing outcomes monitoring

Next step: Define Anticipated Outcomes

• Given the focus of your approach, what outcomes do you expect? Why?

• Do you have a current “baseline”?

• What measures are in place to find out if you are on track to achieving those outcomes?


Process� Completing

assessments on time

� Home visits and parent-teacher conferences held twice per year

Outcome� Children achieving school readiness goals

� Parents understand school readiness goals and are prepared to support them at home

Four Key Outcome AreasChild Outcomes

How prepared will our children be for kindergarten as a result of

our program?

School Readiness Goals

Family OutcomesHow effective will our families be at preparing their children for

success in school as a result of our program?

PFCE Outcomes

Health OutcomesHow will our children be healthier as a result of

our program?

Special PopulationsHow will special populations be positively impacted as a result of

our program?

Sample Measures and Targets for Child Outcomes

Objective: Measure:

Ensure that

transitioning

children leave

our program

prepared for

kindergarten

Assessment on School Readiness Goals

Year 1 Target:

Demonstrate

that 75% of

children

enrolled in HS

for two years

meet school

ready targets.

Year 2 Target:

Demonstrate

that 80% of

children

enrolled in HS

for two years

meet school

ready targets.

Year 3 Target:

Demonstrate

that 85% of

Children

enrolled in HS

for two years

meet school

ready targets.

Year 4 Target:

Demonstrate

that 90% of

children

enrolled in HS

for two years

meet school

ready targets.

Year 5

Evidence:

External

assessment

validates Year

4 progress

(PPVT, etc)

FCE Program Approach

Year 1

• Describe program approach

• Define anticipated outcomes

• Measures of program progress

Year 2

• Focus on progress towards goals, outcomes, and intended impacts

Year 3


Year 4


Year 5

• Provide evidence of the program’s cumulative impact on the children, families, community

Our FCE Approach

Contracting with Families to Close theAchievement Gap

– Family Engagement Contract – Achievement Gap Dialogue– Family Life Practices– Family Engagement “Report Cards”

Differentiated Caseloads and Targeted FamilySupport

– Tiered Family Self-Assessment Instrument– Family Success Rubric– Outcomes-based Goal Setting

Our FCE Approach (cont.)

Competency-based Professional Development

‒ Success Rubrics for all staff

‒ Formal monthly observations and reflective coaching for Advocates and Coordinators

‒ Monthly data driven individual and group supervision

‒ Professional Learning Communities

Table Exercise: What is YOUR approach?

Family engagement?

Family self-assessment?

Professional development?

Anticipated Outcomes

Year 1

• Describe program approach

• Define anticipated outcomes

• Measures of program progress

Year 2


Year 3


Year 4


Year 5

• Provide evidence of the program’s cumulative impact on the children, families, community


Process

� Completing family strengths assessments

Outcome

� Increased family engagement (reading activities at home, PEER cards, progress on the Family services “Roadmap”)

Four Key Outcome AreasChild Outcomes

How prepared will our children be for kindergarten as a result of

our program?

School Readiness Goals

Family OutcomesHow effective will our families be at preparing their children for

success in school as a result of our program?

PFCE Outcomes

Health OutcomesHow will our children be healthier as a result of

our program?

Special PopulationsHow will special populations be positively impacted as a result of

our program?

Our Measures and Targets for Family Outcomes

Objective: Measure:

Ensure that all

families

accomplish

their family life

practice goal

to help their

child be

prepared for

success in

kindergarten.

Family outcome goal tracking system (Shine Insight)

Year 1

Target:

70% of

families will

accomplish

their family

life practice

goal.

Year 2 Target:

80% of

families will

accomplish

their family

life practice

goal.

Year 3 Target:

90% of

families will

accomplish

their family

life practice

goal.

Year 4 Target:

100% of

families will

accomplish

their family

life practice

goal.

Year 5

Evidence:

Outside

assessment of

impact on

families

Group Exercise

� What are your expected outcomes for children ORyour expected outcomes for families based on your ECE or FCE approach? (choose one area to focus on for this exercise)

� How would you measure them?

� What would a good target be for your first year of implementation?

� Given all of these inputs – how would you shape this into language for a long-term goal?

Example Measures and Targets

Objective: Measure:

??

Year 1 Target:

?

Year 2 Target:

?

Year 3 Target:

?

Year 4 Target:

?

Year 5

Evidence:

?

Other Goal Areas

• Other outcomes goals:– Goals related to health outcomes

• Obesity• Asthma• Local / specific health concerns

– Goals related to special populations served by your program

• Other program goals: – Program and fiscal management systems; oversight and

accountability– Enhanced community involvement and resources– Unique community and organizational goals

SHORT-TERM OBJECTIVES AND QUARTERLY TRACKING SYSTEMS

“Aim at Something High”

Goals and Measures

Goal # 1 Short-term Objectives

Ensure that transitioning

children leave our program

prepared for kindergarten

• Time Spent Coaching

• Coaching Sessions / Classroom

• CLASS scores

• Days Wastage (Attendance,

enrollment gaps / churn)

�Most important measure is in the goal itself

� Rest of measures track items we feel are key

contributors to achieving that goal

Goals and Measures

Goal # 2 Short-term Objectives

Ensure that all families

accomplish their family life

practice goal to help their

child be prepared for

success in kindergarten.

• ADA, retention, absenteeism

data

• Parent report card data

(participation in parent

meetings, etc.)

�Actual progress on families achieving family goals is

key outcome

� Parent participation in required meetings/events is

key contributor

How will program data allow us to consistently track these measures?

• Coaching

• Child Outcomes

• CLASS scores

• Days wastage

Education

• Family goals and progress

• Parent participationFamily

Services

Quarterly

Dashboard for

Program Results

Key questions

Where are we now?

According to our five-

year goals, where

should we be?

What are our next

steps?

Using Five Year Goals to Drive On-Going Self-Assessment

• If you align work with your goals, you should be able to measure progress quarterly

• In addition to other compliance measures, this quarterly data is ideal to use as the basis of on-going self-assessment and planning quarterly meetings

• You can use this to END the once a year self-assessment madness

Our Answer: Shine Stat

• Three times a year presentation of key measures

• Audience: Key staff, Board and PC members

• Focuses on actual data from previously established measures

• Cuts through clutter – allows you to see forest through the trees

• Increases accountability and engagement

Shine Stat Sample Measures

Early Childhood Education

• CLASS scores

• average # of coaching cycles per classroom

• average # of coaching-of-coaches cycles per coach

Family & Community Engagement

• # of parent meetings attended

• Progress on Family Life Practice goals

58

1.21.1 1.1 1.08

1.17

0.97

0.8

1.2

1.0

1.201.15

0.851.0

0.7

0.90.79

1.03

0.74

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

September October November December January March

Average # of Coaching Cycles Completed Within 24 Hours, per

classroom

Clark

CPHS

MMHS

Sample Shine Stat

Target : >1 cycle/month

Goal 1: Increase ECE impact. Are teachers/CDs receiving regular coaching?

1.3

2.5

1.4

1.732.00

1.55

1.0

2.3 2.3

1.861.57

0.86

0.4

1.31.5

1.25 1.33

2.00

0.0

0.5

1.0

1.5

2.0

2.5

3.0

September October November December January March

Average # 1:1 Coaching Sessions, per Center Director

Clark

CPHS

MMHS

Analysis� The general expectation

for teachers and CDs is monthly coaching, differentiated by need (regardless of Center size)

� The FMR – post CLASS notification – made February and March abnormal months

� Year to Date (through March)

• Coaching has consistently met our target

59

Sample Shine Stat

Target: 50%Goal 2: Increase FCE impactAre families making progress on FLP goals?

6% 7%12%

18%

25%

37%43%

55%

19%

0%

10%

20%

30%

40%

50%

60%

Family Life Practices

Family Goals

Self-Sufficiency

Family Goals

Support for Children

Family Goals

Support for Family

Members

Family Goals

Home-Based

Family Goals

% of goals with major progress,

Winter 11-12 (left), Fall 12-13 (middle), Winter 12-13 (right)

25%

9%

35%

0%

10%

20%

30%

40%

% of FLP goals with major progress,

Winter 2012-13

CC

CP

MM

Cla

rk

Analysis� 25% of Family Life Practice

Goals have seen major progress, compared to 6% at the same time in winter 2011-2012

� Family Life Practice Goals have seen less progress than other types of goals

• 37% of Self-Sufficiency goals have resulted in major progress

• 43% of Support for Children and Family goals have resulted in major progress

• 55% of Support for Family Member goals have resulted in major progress

60

Sample Shine StatTarget: 75%

Goal 2: Increase FCE ImpactAre parents attending Achievement Gap workshops and

Center Committee Meetings?

0%10%20%30%40%50%60%70%80%90%

100%

% of families attending center events YTD

PCC

FLP

48%

20%

42% 40%

72%

32%

61% 59%

41%

4%

19%26%

60%

27%32%

44%

0%

10%

20%

30%

40%

50%

60%

70%

80%

CC CP MM ACELERO

PCC through Dec 12

PCC through Apr 13

FLP through Dec 12

FLP through Apr 13

Analysis� Our target is for 75% of

enrolled families to attend Family Life Practice workshops

� Since January, the percentage of parents who have participated in at least one workshop has increased from 41% to 60%

� Participation in Center Committee meetings has been higher, with 72% of parents having participated in at least one Center Committee meeting

� Participation in Herb Kaufman, Gates and Sunflower is >80% for FLPs and >80% for PCC

Shine Stat: How do we run these meetings?

• Poster-sized boards with data in each area posted around a room

• Groups of leadership team members physically walk around to examine information, discuss

• Structured review of each service area –

– What stood out?

– What trends came forward?

– What should our takeaways be at this point?

Documenting Planning and Follow-Up

• After each Shine Stat self-assessment and planning meeting, we will document a clear action plan

• Notes include: Next Required Actions, written in who, what, by when format

• Summarize for annual planning meeting as our “self-assessment action plan” (or “Improvement Plan”)

• Annual summary forms a basis of resubmission to RO for tracking progress on five year goals!

Management by Monitoring System: Review

Manage by Information

Compliance & Process

Indicators

ShineStat

Quarterly Self-Assessment

Manage by Outcomes

Outcomes Indicators

Questions?

Thanks for your participation today!

Please stay in touch!

• Visit us at www.shineearly.com

• Sign up for our mailing list

• Follow us on social media (Twitter @ShineEarly, connect on LinkedIn)

• Get our business cards!

how to effectively measure and monitor progress on five...

Documents