Taking Stock: Taking Stock: Assessing & Improving Assessing & Improving

Early Childhood Learning Early Childhood Learning and Program Qualityand Program Quality

Board of the MassachusettsBoard of the MassachusettsDepartment of Early Education and CareDepartment of Early Education and Care

May 13, 2008May 13, 2008

Dr. Sharon Lynn KaganDr. Sharon Lynn Kagan Columbia University & Yale UniversityColumbia University & Yale University

OverviewOverview Impetus for the Work

Challenges

Proposed System Design

Differing Viewpoints on Using Child Assessment Data for Local Agency Accountability

Action Steps

Implications

Part I Part I

Impetus for the WorkImpetus for the Work

Growing Importance of ECEGrowing Importance of ECE Throughout the United States and

countries around the world, Early Childhood Education is ascending to prominence.

States are investing more dollars in the field, with growth in virtually every segment of ECE, including:– Direct Services– Professional Development– Quality Rating Systems

Growing InvestmentsGrowing Investments These growing investments are

matched by calls to assess that the investments are paying off.

Are programs delivering the services they say they are delivering?

Are children achieving the outcomes desired?

Are the investments yielding appropriate returns?

Growing Emphasis on ECEGrowing Emphasis on ECE Practitioners also are interested in

results, recognizing that:– The early years come only once for a

child – The early years are critical to long term

development– Teachers are the keys to making

programs high quality– Teachers and policy makers can learn

from data about children.

Interest in AccountabilityInterest in Accountability Interest in Accountability is not

limited to ECE alone. All human service agencies are being

asked to take on a results orientation. Business and industry has provided a

lead in this area and fueled efforts toward greater efficiency and quality.

There is a prevailing ideology that accountability provides the information to make progress.

Many Factors Provide ImpetusMany Factors Provide Impetus

So there is a collision of interest that propels a focus on outcomes and accountability:– New data that attests to the importance

of the early years– New investments and the need to see if

they are making a difference.– New movement toward accountability

as an elixir for quality improvement.

Part II Part II

Challenges Challenges

Four ChallengesFour Challenges

1. Structural Challenges

2. Conceptual Challenges

3. Technical Challenges

4. Resource Challenges

Challenges: StructuralChallenges: Structural Fragmented non-system of programs

for preschool-aged children

Disjointed early childhood and public education policies

MultipleMultiple Standards and AssessmentsStandards and Assessments

Child Care Head Start State Pre-K Special Education

Program Quality

Standards

State Licensing Standards(50 states)

Quality Rating Systems (QRS) (13 states + 29 pilots)

Program Performance

Standards

State Program Standards(39 states)

IDEA regulationsState program

standards

Assessing Program

Quality

Licensing VisitsQRS Assessments

(13 + 29)

PRISM Reviews Program Monitoring(30 states)

State Program Monitoring

Standards for

Children’s Learning

Early Learning Guidelines(49 states)

Head Start Child Outcomes

Framework

Early Learning Guidelines(49 states)

3 functional goals

Child Assessments

No current requirements National Reporting

System

Pre-K Assessments(13 states)

Kg. Assessments(17 states)

States report % of children in 5

categories on 3 goals

Research/Evaluations Yes Yes Yes Yes

Kindergarten to Grade 3 Standards, Assessments, Data

Challenges: StructuralChallenges: Structural Costs, burdens, confusion of multiple

standards, assessments, & reports.

Multiple new initiatives all at once.

Pre-K–K-3 disconnects:

– Pre-K assessments aren’t transferred to schools.

– Standards, assessments, curricula aren’t aligned.

Challenges: ConceptualChallenges: Conceptual Early childhood assessment training

and practice vs. standards-based assessment and curriculum.

Early childhood reliance on program standards/data vs. expanded interest in children’s learning.

Challenges: TechnicalChallenges: Technical Need appropriate assessment tools

and methods to report on:

– Progress/status of young children in all domains of learning and development

– Young ELLs and children with disabilities

– Program quality in diverse local agencies

Challenges: ResourcesChallenges: Resources Limitations and inequities in

funding for:

– Programs

– Infrastructure

Risk that accountability efforts ignore and exacerbate inequities in resources

Task Force ChargeTask Force Charge Recommendations for a state

accountability system for early education programs for pre-kindergarten children and linkage to standards-based assessments in kindergarten-grade 3.

OriginsOrigins Conceived by The Pew Charitable Trusts

as part of their Advancing Quality Pre-K For All initiative.

Additional funding from the Foundation for Child Development and the Joyce Foundation.

Task Force convened in fall, 2005, report release fall, 2007.

Presentation reflects progress-to-date.

Dr. Sharon Lynn Kagan, Chair Dr. Eugene Garcia, Vice-Chair

– Dr. W. Steven Barnett– Ms. Barbara Bowman– Dr. Mary Beth Bruder– Dr. Lindy Buch– Dr. Maryann Santos

de Barona– Ms. Harriet Dichter

– Mr. Mark Friedman– Dr. Jacqueline Jones– Dr. Joan Lombardi– Dr. Samuel Meisels– Ms. Marsha Moore– Dr. Robert Pianta– Dr. Donald Rock

Task Force MembersTask Force Members

Part IIIPart III

Proposed System DesignProposed System Design

Framing BeliefsFraming Beliefs Accountability is here to stay.

Programs should be held to performance standards that are documented and verified.

Assessments should inform policy decisions and be tied to program enhancement efforts.

Current approaches to accountability and assessment must be reformed.

State Accountability & Improvement State Accountability & Improvement System DesignSystem Design

Pre-K-Grade 3 Alignment and Linkages

Assessment/Program Improvement Approaches

APPROACHES I CHILD POPULATION

IIPROGRAM POPULATION

III STATE PROGRAM

EVALUATION

IV LOCAL AGENCY

QUALITY

COREQUESTION

How well are all children progressing in

learning and development?

What is the quality of all early childhood

programs?

What is the quality and how well are children progressing in specific

state programs?

What is the quality in local agencies?

HOW DATA IS USED

- Oversight of state investments/initiatives - Planning new investments/initiatives - Baseline information for K-12 education planning

- Oversight of state investments/initiatives - Planning new investments/initiatives - Baseline information for K-12 education planning

- Program-wide improvement efforts - Refining standards/policies- Appropriations decisions

- Technical assistance to individual agencies.- Awarding incentives and recognition to local agencies for program improvements - Decisions on funding local agencies

Infrastructure

Early Learning &Program Quality

Standards

Program Rating & Improvement

Professional Development Data Management & Reporting

InfrastructureInfrastructure

System Infrastructure

Early Learning

&Program Quality

Standards

Program Rating &

Improvement

Professional Developmen

t

Data Manageme

nt & Reporting

InfrastructureInfrastructure Early Learning and Program

Quality Standards– Alignment between:

• Standards, assessment systems and curricula

• Standards between ages and grades

• State and federal program structures and funding streams

• Child and program standards

InfrastructureInfrastructure Program Rating & Improvement

– Assesses and reports on the quality of all forms of early education programs

– Provides technical assistance and professional development to improve quality

– May provide public recognition/incentives to reward higher levels of quality

InfrastructureInfrastructure Professional Development System

– Links informal training with formal education, provides career pathways, links education and compensation.

– Supports training on assessment administration, analysis and use.

InfrastructureInfrastructure Data Management and Reporting

– All-in-one place data on:• Children

• Programs

• Workforce

– Unified system of child identification numbers.

– Provides for quality assurance of data and assessments.

Assessment ApproachesAssessment Approaches States vary in:

– What they want to know– How they plan to use data– Available resources

States may implement one or any combination of options.

Report includes cautions/safeguards for each option.

Assessment/Improvement ApproachesAssessment/Improvement Approaches

APPROACHES Child Populatio

n

ProgramPopulatio

n

State Program

Evaluation

Local Agency Quality

COREQUESTION

How well are all young

children progressin

g in learning

and developme

nt?

What is the quality of all early

education programs?

What is the quality and

how are children

progressing in specific

state programs?

What is the quality in

local agencies?

AllAll Four Approaches Discuss:Four Approaches Discuss:

Questions that can be Answered

Data to be Collected

Designs for Data Collection

Uses of the Data

Challenges and Cautions

I. Child Population ApproachI. Child Population Approach How well are all young children

progressing in learning and development?

– Data:• Demographic data collected on representative

sample of all young children

• Comprehensive data on health, well-being, pre-school enrollment

– Designs: • At Kindergarten Entry

• At Ages 1, 3, 4

• Longitudinal ECLS State Strategy

I. Child Population ApproachI. Child Population Approach How well are all young children

progressing in learning and development?

– Uses: • Planning interagency investments/initiatives

• Legislative oversight

• Baseline information for public education

– Challenges/Cautions: • Gaining access to unrolled preschoolers

• Can’t use data to maker inferences about pre-school programs’ impacts/quality

• No causal attributions

• Expensive and not routinely done

II. Program Population II. Program Population ApproachApproach

What is the quality of services in all early childhood programs?– Data:

• Program quality• Workforce• Public investments

– Design:• Data collected from states

Program Improvement (PRS/QRS) System

II. Program Population II. Program Population ApproachApproach

What is the quality of services in all early childhood programs?– Uses:

• Planning interagency investments/initiatives

• Legislative oversight

• Baseline information for public education

– Challenges/Cautions:• Need to consider quality of programs in light of

investments in them

• Need to be sensitive to changes in program quality

• Unable to attribute causality to program quality rankings

III. State Program Evaluation III. State Program Evaluation ApproachApproach

What is the quality and how well are children progressing in specific programs?– Data:

• Sample of centers that represent the universe of programs to be studied

• Comprehensive data on children, program, and teachers

• Data aligned with program mandates and standards

– Design: • Standard Evaluation Design-program and

instrument identification, data collection and analysis

• Program Rating System Design-uses data for program information from the PRS/QRS

III. State Program Evaluation III. State Program Evaluation ApproachApproach

What is the quality and how well are children progressing in specific programs?– Uses:

• Program-wide improvement efforts• Refining standards/policies• Appropriations decisions

– Challenges/Cautions:• Attribution of single program effects due to

many children’s participation in multiple programs (e.g., lack of pure control group)

• Pre-mature evaluation inappropriate• Implementation fidelity to program design

IV. Local Agency Quality IV. Local Agency Quality ApproachApproach

What is the quality in local agencies?– Data:

• Program quality data in relation to state standards

• Observations of teaching/learning opportunities

– Design:• Use PRS/QRS data• On rotational basis

– 1/3 annually– Low performing programs

IV. Local Agency Quality IV. Local Agency Quality ApproachApproach

What is the quality in local agencies?– Uses:

• Technical assistance to individual local agencies

• Awarding incentives and public recognition

– Challenges/Cautions:• Heavy burden on state agencies

• Heavy costs to do classroom observations

• States need to develop technical assistance to address program needs/weaknesses

Pre-K—Grade 3 IntegrationPre-K—Grade 3 Integration Align standards, assessments and

reporting on:– Children’s progress– Quality of teaching/learning

opportunities

“Vertical” teams of teachers/managers to:– Review assessment information

– Enrich learning experiences and teaching strategies

Joint professional development

Part IV Part IV

Differing Viewpoints on Differing Viewpoints on Using Child Assessment Data Using Child Assessment Data

for Local Agency for Local Agency AccountabilityAccountability

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies Task Force reached an easy

consensus on: – Infrastructure components

– Four assessment/improvement approaches

– Linking of pre-kindergarten through grade 3

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies Some members said this wasn’t

enough. Needed more information:

– How well are children in local agencies learning?

– How does what they are learning compare with state standards?

– Are some local agencies unusually effective in fostering learning progress?

To do this, some members of the Task Force advocated collecting and reporting data on all children's performance and to distill it so it could be used by local agencies for planning and improvement.

They recognized that we can have children in good programs who don’t do well, and they contend that program quality is NOT a proxy for child outcomes.

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies These Task Force members

recognized the huge challenges associated with this:

1. When we disaggregate data, it will be held to a higher standard.

2. The field needs better tools3. The field needs well-trained reliable

data collectors4. The field needs systematic ways of

collecting, cleaning, and analyzing data.

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies Most importantly, these Task Force

members noted, we need to be very careful not to misuse data:– Not to teach to the test.– Not to use data to label or place

children.– Not to defend programs.

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies The Task Force members who

support the use of agency-level child assessment data acknowledged the challenges, but contended, over time, states would benefit by using both child and program quality data.

They recommended nine crucial safeguards.

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies1. Collect enriched data on the programs,

including teacher-child interactions, nature of curriculum interventions, and nature of teacher training.

2. Collect detailed data on the children, including prior ECE experience and primary language.

3. Impel the state to align its assessments to all domains in standards.

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies4. Select approaches that include direct

observation.

5. Base reporting on progress.

6. Collect data at more than one point in time.

7. Stagger strategies for data collection:– One third in each of 3 years

– Priority accorded to low-quality programs

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies8. Do not, under any circumstances,

report any data on individual children.

9. Use the data collected to make program improvements.

SAFEGUARDS #8 AND #9 ARE, BY FAR, THE MOST IMPORTANT

ONES RECOMMENDED.

Differing Viewpoints on Using Differing Viewpoints on Using Child Assessment Data on Child Assessment Data on

Local AgenciesLocal Agencies

While there was consensus on the safeguards recommended, the Task Force did not reach consensus on offering this a

separate approach.

Part V Part V

Action StepsAction Steps

Action Steps: LegislaturesAction Steps: Legislatures

Provide adequate funding for programs and infrastructure for ongoing assessments and program improvements.

Action Steps: State AgenciesAction Steps: State Agencies

Develop a strategic plan for early childhood accountability and program improvement system.

Create a robust, positive, and rigorous culture for early childhood accountability efforts.

Enable local Pre-K–3 partnerships.

Action Steps: Federal Action Steps: Federal GovernmentGovernment

“Harmonize” information systems.

Fund research and development for better assessment tools.

Support ongoing longitudinal research on children and programs.

Action Steps: Local AgenciesAction Steps: Local Agencies

Create opportunities for teachers and managers to review assessments and enhance children’s learning opportunities.

Initiate dialogue with local school districts.

The BenefitsThe Benefits For Children: Enhanced learning

opportunities and improved outcomes

For Legislators: Better data to guide state policies and investments

For Teachers/Directors: Targeted and well-resourced professional development and program improvement efforts

For the Early Childhood Profession: Enhanced public awareness and credibility

Action Steps: Follow-UpAction Steps: Follow-Up The Pew Charitable Trusts approved

a grant to the Council of Chief State School Officers (CCSSO) to disseminate the report and work with selected states in implementing recommendations.

Project will build on CCSSO efforts to improve state education standards, assessments, and data systems for all children—preschool through high school and beyond.

Part VI Part VI

ImplicationsImplications

Implications Implications

Acknowledge the intensity of sentiment regarding potential misuse of child assessment data.

Attend to the infrastructure and the resources to do the collection.

Think systemically.

Implications Implications Need more work on tools that are

aligned with standards and curriculum.

Need more work on hard-to-capture domains in assessment construction.

Need far more work on the considerations regarding English Language Learners and children with disabilities.

Implications Implications Need states to focus on effective

accountability systems. Such accountability systems should be

designed to meet the state’s needs. Such accountability systems should take

into consideration state resources (human, technical, and fiscal).

Every state should develop and begin implementing an ECE accountability system that provides data and respects families, teachers, children, and child development principles.