new evaluating infant/toddler activities: what, who, how, and when · 2018. 5. 22. · overview and...
TRANSCRIPT
The Infant/Toddler & School-Age Child Care Institute
Kelley Perkins, Infant Toddler Specialist Network
Holly Wilcher, Infant Toddler Specialist Network
Joyce Bridges, Tennessee Child Care Resource and Referral Network
Briana Jiminez, Tennessee Child Care Resource and Referral Network
Karen Wright, Tennessee Child Care Resource and Referral Network
Arlene Kasper, Virginia Department of Social Services
Evaluating Infant/Toddler
Activities: What, Who, How, and
When
1
Welcome
Kelley Perkins, State Capacity Building Center Infant/Toddler Technical Assistance Specialist,
Region III
Holly Wilcher, State Capacity Building Center Infant/Toddler Technical Assistance Specialist,
Region IV
2
Overview and Agenda
◆ Welcome
◆ Why evaluation?
◆ What is evaluation?
◆ What are some best practices with evaluation?
◆ What are some state examples of evaluation
related to infant/toddler care?
◆ What are your questions about evaluation?
◆ Closing
3
Professional Learning Goals
Define evaluation.
Identify the purpose of evaluation.
Explore strategies states use to evaluate infant/toddler quality initiatives.
4
What Is Your Experience with Evaluation?
◆ I’m familiar with the term.
◆ I’ve participated in evaluation efforts.
◆ I’ve read evaluation reports.
◆ I’ve helped develop an evaluation plan.
◆ I’ve conducted evaluations.
◆ All of the above
◆ Other
5
The Infant/Toddler & School-Age Child Care Institute
Why Evaluation?
6
Child Care and Development Fund (CCDF)
Plan Draft Preprint Guidance on Evaluation
7.5.2. Describe the measureable indicators of progress relevant to this use of funds that the
state/territory will use to evaluate its progress in improving the quality of child care programs and
services for infants and toddlers within the state/territory and the data on the extent to which
the state or territory has met these measures.
7
Source: Office of Child Care, Administration for Children and Families, U.S. Department of Health and
Human Services. (2017). CCDF plan preprint 2019-2021 draft. Retrieved from
https://www.acf.hhs.gov/occ/resource/ccdf-plan-preprint-2019-2021-draft
Infant/Toddler Specialist Network (ITSN)
Community of Practice Evaluation Questions
How do other states evaluate their ITSNs?
How do you evaluate without additional
funding?
How do we evaluate whether we are
impacting quality?
How can we prove or evaluate whether
relationship-based care is working?
How do we measureeffectiveness in
working with providers?
What tools are available for tracking?
What are the tools used for assessment?
What are different states doing?
How do we demonstrate
accountability for measuring
relationship-based practices?
How do we measure and document the effectiveness of
time spent building relationships
with providers?
8
Source: Metz, A., Naoom, S .F., Halle, T., & Bartley, L. (2015). An integrated stage-based framework for implementation
of early childhood programs and systems (OPRE Research Brief OPRE 2015-48). Washington, DC: Office of Planning,
Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
9
The Infant/Toddler & School-Age Child Care Institute
“The making of a judgement about the amount,
number, or value of something; assessment”
What Is Evaluation?
10
Source: Oxford Dictionaries, https://en.oxforddictionaries.com/definition/evaluation
Types of Evaluation
Formative evaluation
Process and implementation evaluation
Outcome and effectiveness evaluation
Impact evaluation
Source: U.S. Centers for Disease Control and Prevention. (n.d.). Types of evaluation. Retrieved from
https://www.cdc.gov/std/Program/pupestd/Types%20of%20Evaluation.pdf
11
Evaluation Model Example: Kirkpatrick
Source: Kirkpatrick Partners. (n.d.). The Kirkpatrick model. Retrieved April 17, 2018,
from https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model
Results
Behavior
Learning
Reaction
12
The Infant/Toddler & School-Age Child Care Institute
What Are Some Best Evaluation
Practices?
13
What Do You Want to Evaluate?
◆ What is the “what”?
◆ How does a logic model support this effort?
◆ How can you use logic model builders?
▪ Child Welfare Information Gateway:
https://www.childwelfare.gov/topics/management/effe
ctiveness/logic-model/
▪ University of Minnesota: https://cyfar.org/build-your-
own-logic-model
14
The Infant/Toddler & School-Age Child Care Institute
Virginia Infant/Toddler Specialist Network
Sample Logic Model
15
Source: Division of Child Care and Early Childhood Development, Commonwealth of Virginia Department of Social Services. (2011).
RFP # CECD-12-042. Richmond, Virginia. Retrieved from https://www.dss.virginia.gov/files/forms/grants/cecd-12-
042/attachment_e.pdf
16
RFP # CECD-12-042. LOGIC MODEL – INFANT & TODDLER SPECIALIST NETWORK
17
RFP # CECD-12-042. LOGIC MODEL – INFANT & TODDLER SPECIALIST NETWORK
Source: Division of Child Care and Early Childhood Development, Commonwealth of Virginia Department of Social Services. (2011).
RFP # CECD-12-042. Richmond, Virginia. Retrieved from https://www.dss.virginia.gov/files/forms/grants/cecd-12-
042/attachment_e.pdf
Considerations for Data Collection
If you have data, you can evaluate!
If you have data, you can answer questions!
Data are valuable in words (qualitative) and numbers (quantitative)!
18
The Infant/Toddler & School-Age Child Care Institute
What Are Some State Examples of
Evaluation Related to Infant/Toddler
Care?
19
State Examples of Infant/Toddler
Quality Initiative Evaluation
◆ Wisconsin infant/toddler credential
▪ https://childcareta.acf.hhs.gov/sites/default/files/public/2013062
8pdwcenterwievaloverviewfinal.pdf
◆ Colorado Expanding Quality in Infant/Toddler Care
Initiative
▪ https://www.tandfonline.com/doi/abs/10.1080/10409289.2014.9
41260?src=recsys&journalCode=heed20
20
Outcome and Impact Evaluation:
Wisconsin
Wisconsin wanted to know if participating in an infant/toddler credential led participants to advance their education and/or job promotion.
Source: National Center on Child Care Professional Development Systems and
Workforce Initiatives. (2013). Wisconsin credential evaluation overview. Retrieved from
https://childcareta.acf.hhs.gov/sites/default/files/public/20130628pdwcenterwievaloverv
iewfinal.pdf
21
Impact Evaluation: Colorado
Colorado evaluated the effectiveness of two types of infant/toddler caregiver professional preparation efforts that worked to improve participants’ knowledge, self-efficacy, and interactions with children relative to a community comparison group that experienced neither intervention.
22
The Infant/Toddler & School-Age Child Care Institute
Virginia’s Infant/Toddler Specialist
Network Evaluation
23
VA Infant & Toddler Specialist Network“Achieving Excellence in Early Care”
▪ Inputs
▪Outputs
▪Outcomes
Network Logic Model
25
Overview of Activities/Outcomes
▪ High Level Outcome
▪ Activities
▪ Staff Responsible
▪ Output
▪ Outcomes
▪ Evaluation
26
Purpose of Network Evaluation
27
▪ Extent
implementing
planned activities
▪ Extent services are
useful in meeting
providers’ needs
▪ Results or benefits
of the services
Knowledge Gained from the Evaluation
28
▪Quality of child care
▪ Information for continual
improvement
▪ Progress on meeting outcomes
▪ Information for changes in
program
▪ Effective practices/services
Changes to the Evaluation
▪ Sustainability of
Improvements
▪ Qualitative
documentation
▪ Frequency of
questionnaire use
29
Interesting Findings and Challenges
▪ Consistency of results
▪ Variations between
regional offices
▪ High levels of
satisfaction
▪ Feasibility of the
frequency of evaluations
▪ Scheduling of
Environment Rating
Scale assessments30
Next Steps
▪New infant and toddler
mental health consultation
▪ ITERS-3
▪ Pre- and post-knowledge
measures
▪ Consideration of outside
evaluation
▪ Continual improvements
31
32
For more information, contact us at:
www.va-itsnetwork.org
33
The Virginia Infant & Toddler Specialist Network is a project of Child Development Resources, and is supported by the Virginia Department of Social Services Grant #93.575,
with funds made available to Virginia from the U.S. Department of Health and Human Services.
Arlene Kasper
Program Development Consultant
Virginia Department of Social Services
804-726-7647
Amy Stutt
Statewide Director
Virginia Infant & Toddler Specialist Network
757-378-3173
The Infant/Toddler & School-Age Child Care Institute
Tennessee's Infant/Toddler
Credential Pilot Evaluation
34
PLANNING FOR THE INFANT & TODDLER CREDENTIAL PILOTTENNESSEE CHILD CARE RESOURCE & REFERRAL NETWORK
SIGNAL CENTERS, INC.
TN DEPARTMENT OF HUMAN SERVICES
What we had
A funder with the desire to pilot a statewide
Infant & Toddler Credential
An existing network
Pilot Objectives:To increase caregiver knowledge of
▪ Child development (theory & practice)
▪ Relationship-based models of care and education
▪ The importance of interactions to infant mental health
▪ Individual curriculum design
▪ The role of family & community in infant and toddler care
To qualify for participation
❑ Employed for 6 months (By DHS licensed facility);❑ Provide 3 letters of recommendation;❑ Sign a statement of personal commitment.
5 Key Concepts
1. A safe and healthy classroom environment
2. A developmentally appropriate classroom environment and curriculum
3. An increased understanding of child development
4. A respect for the role that family and community plays in the life of a child
5. An increased understanding of the social and emotional needs of infants and toddlers.
Tier 4Tier 1• 45 Clock hours of IT
training OR TECTA IT orientation & 15 additional clock hours
• At least 6 months experience in direct care of infants & toddlers in DHS licensed childcare
• 40 clock hours of onsite coaching
• Basic portfolio development
15 Participants
In East TN
Tier 2• CDA with IT
Endorsement OR 45 clock hours of IT training/3 credit hours of IT academic coursework OR Tier 1 credential plus 12 additional clock training hours
• At least 1 year experience in direct care of infants & toddlers in DHS licensed childcare
• 40 clock hours of onsite coaching
• Intermediate level portfolio
15 Participants
in Middle TN
Tier 3• Associates Degree
with 9 IT specific credit hours OR CDA w/IT Endorsement plus 9 additional IT specific credit hours OR Tier 2 credential plus 9 additional credit hours
• At least 18 months experience in direct care of infants & toddlers in DHS licensed childcare
• 40 clock hours of onsite coaching
• Advanced level portfolio
15 Participants
In West TN
• Bachelors or Advanced degree with 15 credit hours of IT specific academic coursework
• At least 18 months experience in direct care of infants & toddlers in DHS licensed child care
• 40 hours of onsite Mentoring OR 15 hours of onsite coaching plus 20 hours serving as a mentor to an IT caregiver
• Advanced level portfolio
5 Participants
In Middle TN
Why evaluate?
credibility perspectiveonly chance
What did we want to know?
Demonstratable differences?
❑ Beliefs
❑ Practice
❑ Quality
Instruments Used in Evaluation
▪Provider Background Questionnaire
▪Organizational Climate Questionnaire
▪Teacher Efficacy Scale
▪Parent Caregiver Relationship Scale
▪Environment Rating Scales®
▪Caregiver Interaction Scale (Arnett)
▪Caregiver Feedback Survey
▪Director Feedback Survey
▪Coach Background Questionnaire
▪Knowledge & Skill Inventory for Coach
▪Coach Professional Comfort Scales
ERS® and Environment Rating Scale® are registered trademarks of Teachers College, Columbia University.
Outcomes
Teacher self-efficacy
Parent/caregiver relationships
What Now?Moving forward with lessons learned…
Questions, Thoughts, and Reflections
about Evaluation
47
Additional Resources to Support
Evaluation of Quality Initiatives
◆ 2016 Quality Investments: How to Use Administrative Data and Other
Sources to Evaluate What Is Working
This webinar series examines evaluation options and implementation strategies
to help states and territories identify the resources, data sources, and costs
associated with evaluating quality initiatives, with a particular focus on evaluation
related to Child Care and Development Fund quality activities.
https://childcareta.acf.hhs.gov/resource/webinar-series-quality-investments-how-
use-administrative-data-and-other-sources-evaluate
◆ Elements to Build Capacity for Evaluation and Accountability: Discussion
Guide (2014)
This guide identifies six elements for building capacity for evaluation and
accountability. Questions to prompt discussion and reflection among planning
and stakeholder groups are followed by suggested strategies to address each
element.
https://childcareta.acf.hhs.gov/sites/default/files/public/sssn_assessingcapacityfor
evaluation_0.pdf
48
Additional Resources to Support
Evaluation of Quality Initiatives
◆ FRIENDS Child Welfare Gateway Evaluation Toolkit
This online compendium of information and resources supports
developing an individualized outcome evaluation plan from the ground up.
http://www.friendsnrc.org/evaluation-toolkit
◆ Research and Evaluation Capacity: Self-Assessment Tool and
Discussion Guide for CCDF Lead Agencies (2017)
This tool guides users through a process of reflecting on their research
and evaluation capacity.
https://www.acf.hhs.gov/sites/default/files/opre/research_and_evaluation_
capacity_self_assessment_tool_finalcleanv3.pdf
49
Additional Resources to Support
Evaluation of Quality Initiatives
◆ Developing a Staffed Family Child Care Network: A Technical
Assistance Manual (2017)
This document outlines the four stages of successful network
implementation—exploration, installation, initial implementation, and
full implementation.
https://childcareta.acf.hhs.gov/resource/developing-staffed-family-
child-care-network-technical-assistance-manual
◆ Strengthening State and Territory Infant/Toddler Child Care
System Policies and Practices: A Tool for Advancing
Infant/Toddler Child Care Quality (2017)
This tool is designed to support state child care leaders and
infant/toddler stakeholders as they assess, prioritize, plan, implement,
and evaluate state policies to strengthen infant/toddler child care
quality.
https://childcareta.acf.hhs.gov/resource/strengthening-state-and-
territory-infanttoddler-child-care-system-policies-and-practices
50
Kelley Perkins, Infant/Toddler Specialist
Child Care State Capacity Building Center
Holly Wilcher, Infant/Toddler Specialist
Child Care State Capacity Building Center
Thank you for your participation!
Contact Information
51