PPT Presentation Template:PPT Presentation Template:

This PPT includes all slides to present This PPT includes all slides to present a district or building level overview of a district or building level overview of PVAAS. This was used with a district-PVAAS. This was used with a district-

wide K-12 faculty group in a small wide K-12 faculty group in a small school district, where one grade level school district, where one grade level

represented one building versus represented one building versus multiple buildings on a district level multiple buildings on a district level

report.report.

PVAAS:PVAAS:Evaluating Growth,Evaluating Growth,

Projecting PerformanceProjecting Performance

Today’s SessionToday’s Session

Concept of PVAASConcept of PVAAS Overview of ReportingOverview of Reporting

• Math and Reading, Grades 4-8Math and Reading, Grades 4-8• Math and Reading, Grade 11Math and Reading, Grade 11• Science, Grades 4, 8 and 11Science, Grades 4, 8 and 11• Writing, Grades 5, 8 and 11Writing, Grades 5, 8 and 11

School District DataSchool District Data Questions and AnswersQuestions and Answers

What Is What Is PVAAS?PVAAS?

PVAAS IS:

• Analysis of existing assessment data– Longitudinal PSSA scores in available

content areas• Math, Reading, Science, Writing

– Provides reliable estimates

• A measure of a school’s effectiveness

• Critical Relationship: Growth Data vs. Achievement Data

PVAAS:

• Is not another test

• Is not a comparison of single test scores

• Critical concept:Does not provide a measure of

growth for an individual student

High Achievement

Average Achievement

Low Achievement

Current Data Collection Current Data Collection ApproachesApproaches

Achievement is based Achievement is based on each student’s ending point.on each student’s ending point.

PVAAS Adds Another Variable To Achievement…Progress

Below Growth

Standard

At or AboveGrowth

Standard

Progress is based Progress is based on each student’s starting pointon each student’s starting point..

Viewing Achievement and Viewing Achievement and Progress TogetherProgress Together

Scatter Plots:Scatter Plots:

Grades 4-8Grades 4-8

Reading and MathReading and Math

The Scatter Plot The Scatter Plot Coordinate SystemCoordinate System

Achievement and Growth - 4th Math

0

56

-20 0 20

PVAAS Growth Measure

% P

rofi

cie

nt

or

Ad

van

ced

Vertical AxisPSSA Percent Proficient or Advanced

Horizontal AxisPVAAS Growth

Measure

AYP Percent Proficient Target

Growth StandardBoundary

One Year

Growth

Scatter Plot Quadrants Scatter Plot Quadrants and Their Meaningsand Their Meanings

Achievement and Growth - 4th Math

0

56

-20 0 20

PVAAS Growth Measure

% P

rofi

cie

nt

or

Ad

van

ced

Quad 1Adequate Achievement

Positive Growth

Quad 4Below Adequate

Achievement Positive Growth

Quad 2Adequate Achievement

Negative Growth

Quad 3Below Adequate

AchievementNegative Growth

SD Scatter PlotsSD Scatter PlotsGrade 4Grade 4

Reading Math

Insert District Scatter Plots Here

SD Scatter PlotsSD Scatter PlotsGrade 5Grade 5

Reading Math

Insert District Scatter Plots Here

SD Scatter PlotsSD Scatter PlotsGrade 6Grade 6

Reading Math

Insert District Scatter Plots Here

SD Scatter PlotsSD Scatter PlotsGrade 7Grade 7

Reading Math

Insert District Scatter Plots Here

SD Scatter PlotsSD Scatter PlotsGrade 8Grade 8

Reading Math

Insert District Scatter Plots Here

Viewing Achievement and Viewing Achievement and Progress TogetherProgress Together

Scatter Plots:Scatter Plots:

Grade 11Grade 11

Reading and MathReading and Math

Grade 11 Scatter PlotsGrade 11 Scatter PlotsThe PSSA-PVAAS Scatter Plots visually display the following The PSSA-PVAAS Scatter Plots visually display the following data from the statewide distribution of PSSA results for the data from the statewide distribution of PSSA results for the most current school year.most current school year. AchievementAchievement

• The mean The mean predictedpredicted performance of the cohort. performance of the cohort. • The actual mean performance of all students in the state is displayed The actual mean performance of all students in the state is displayed

for referencefor referenceGrowthGrowth

• School Effect School Effect ≈ Observed Mean – Predicted Mean≈ Observed Mean – Predicted Mean• Standardized School Effect = School Effect relative to its error – Standardized School Effect = School Effect relative to its error –

displays degree of confidencedisplays degree of confidence

The following two slides explain the meaning of each of the The following two slides explain the meaning of each of the regions of an 11regions of an 11thth grade scatter plot. grade scatter plot.

The Scatter Plot The Scatter Plot Coordinate System for Grade 11Coordinate System for Grade 11

Horizontal AxisGrowth Measure in

Standard Errors

Vertical AxisMean

Predicted Scaled Score Growth Descriptor

Boundaries (±2SEs)

2010 Mean Scaled Score

Scatter Plot Regions Scatter Plot Regions and Their Meanings for Grade 11and Their Meanings for Grade 11

SD Scatter PlotsSD Scatter PlotsGrade 11Grade 11

Reading Math

Insert District Scatter Plots Here

The TWO Types ofThe TWO Types ofPVAAS InformationPVAAS Information

Today

Looking Back/Evaluation…

Value-added Growth Reports For Cohorts of Students

Looking Forward/Planning…

PVAAS Projection Reports For Individual Students

and Cohorts of Students

NO individual measure of student growth.

PVAAS PVAAS ReportingReporting

Student ProjectionsStudent Projections

Wouldn’t it be great to know the likelihood that a student will be proficient

on a future PSSA?

Student Projection MethodologyStudent Projection Methodology

A separate longitudinal modeling process A separate longitudinal modeling process focused on the performance scores of focused on the performance scores of individual students.individual students.

The model uses The model uses allall of the available data (both of the available data (both math and reading) to project future PSSA math and reading) to project future PSSA performances in Math, Reading, Science and performances in Math, Reading, Science and Writing.Writing.

Projections can be calculated for an individual Projections can be calculated for an individual student or groups of students.student or groups of students.

How do I know that the projectionHow do I know that the projectionis accurate?is accurate?

The projection is precise, and is created using a model that has been reviewed and approved by four different peer review panels and the GAO (US Government Accountability Office).

The most recent 2008 growth model proposal to USDOE includes information regarding the statistical model and projection reliability study.(http://www.ed.gov/admins/lead/account/growthmodel/pa/index.html)

Recent studies have confirmed that the PVAAS projections (even as far as 3 years into the future) are more reliable at looking at the future performance of a student than the most recent PSSA score.

PVAAS Student Projection ReportPVAAS Student Projection Report

PVAAS Projections for Groups of Students

Student Search

Identified Students Meeting Search Requirements

PVAAS Reporting PVAAS Reporting Projections to Basic, Proficient, and Advanced Projections to Basic, Proficient, and Advanced

Grade Last Tested

Math and Reading: 3 4 or 5 4 5 or 6 5 6 or 7 6 7 or 8 7 8 8 11

Writing: 3 5 4 5 5 8 6 8 7 8 8 11

Science:

3 4 4 NA 5 8 6 8 7 8 8 11

Grade Last Tested Grade Last Tested

Projection Summary ReportsProjection Summary Reports What is it?What is it?

• Report that summarizes the numbers and Report that summarizes the numbers and percents of students in likelihood ranges of percents of students in likelihood ranges of performing at the proficient level on a future performing at the proficient level on a future PSSA exam.PSSA exam.

How might a school use this report?How might a school use this report?• Intervention PlanningIntervention Planning• Resource AllocationResource Allocation• Strategic PlanningStrategic Planning• School Improvement PlanningSchool Improvement Planning

CautionsCautions• This report provides another indicator about This report provides another indicator about

likelihood of future performance. likelihood of future performance.

School (Single Grade)School (Single Grade)Projection Summary ReportProjection Summary Report

Wouldn’t it be great to know if your students made one year’s growth for the year of schooling

they experienced last year?

In Math and Reading (4-8 and 11)?In Science (4, 8 and 11)?In Writing (5, 8 and 11)?

Value-AddedValue-Added

PVAAS Growth MethodsPVAAS Growth Methods

Testing Subjects – Grades Methodology

PSSA in consecutive

years Math & Reading – 4-8

Growth Standard

PSSA not in consecutive years

Writing – 5, 8, 11Science – 4, 8, 11

Math & Reading – 11Predictive

Value-added Measure of GrowthValue-added Measure of GrowthMath and Reading (Grades 4-8)Math and Reading (Grades 4-8)

Comparison to a State Growth StandardComparison to a State Growth Standard

The Growth Standard specifies the minimal The Growth Standard specifies the minimal acceptable academic gain from grade to acceptable academic gain from grade to grade for a cohort of students. grade for a cohort of students.

The use of a Growth Standard creates the The use of a Growth Standard creates the possibility that possibility that ALLALL schools can demonstrate schools can demonstrate appropriate growth.appropriate growth.

PVAAS Value-Added Growth Descriptors – Grade 4 – 8

Math and Reading

The PA Growth Standard specifies the minimal acceptable academic gain from grade to grade for a cohort of students.  The Growth Standard is met when the students in a school or district on average maintain their relative position when compared to the 2006 statewide distribution of student scores (2006 is the base year).

Estimated mean NCE gain equal or greater than zero.  There is evidence that the school has met or exceeded the Growth Standard.  All schools can achieve this rating. Estimated mean NCE gain less than zero by one standard error or less.  There is slight evidence that the school has not met the Growth Standard. Estimated mean NCE gain less than zero by between one and two standard errors.  There is moderate evidence that the school has not met the Growth Standard. Estimated mean NCE gain more than two standard errors less than zero.  There is significant evidence that the school has not met the Growth Standard.

PVAAS School ReportPVAAS School ReportGrades 4 through 8 - Math and ReadingGrades 4 through 8 - Math and Reading

What Value-added Gain Means!What Value-added Gain Means!

0 10 20 30 40 50 60 70 80 90 100

NCE Scale

State Mean = 50

Cohort’s Achievement

6th grade60.7

Cohort’s Achievement

7th grade66.6

PVAAS Gain ≈ Change in position from 6th to 7th grade

≈ 66.6 – 60.7 ≈ 5.9 NCE points

PVAAS Growth MethodsPVAAS Growth Methods

Testing Subjects – Grades Methodology

PSSA in consecutive years

Math & Reading – 4-8Growth

Standard

PSSA not in consecutive

years

Writing – 5, 8, 11Science – 4, 8, 11

Math & Reading – 11Predictive

Value-added Measure of Cohort GrowthValue-added Measure of Cohort Growth Math and Reading (Grade 11)Math and Reading (Grade 11)Science (Grades 4, 8, and 11)Science (Grades 4, 8, and 11)Writing (Grades 5, 8, and 11)Writing (Grades 5, 8, and 11)

Based on a Based on a predictive methodology…predictive methodology…

A predicted score is calculated for each A predicted score is calculated for each student;student;

Mean of predicted scores is calculated for Mean of predicted scores is calculated for students in a cohort;students in a cohort;

Mean of observed scores of students in the Mean of observed scores of students in the cohort;cohort;

School Effect is the difference between the School Effect is the difference between the Observed and the Predicted Scores.Observed and the Predicted Scores.

PVAAS Predictive Analysis PVAAS Predictive Analysis

Year 1 Year 2 Year 3 PSSA PSSA PSSA (11th)

7th 8th Predicted Actual

1. We collect all of the individual student data that is available for a minimum of two years.

2. All of these data are used to develop individual student profiles.

3. PVAAS then calculates a Predicted Performance Value based on the student profile for each student.

Reading 1280 1300

Math 1330 1305

1290 ??

1320 ??

PVAAS Ratings of School EffectPVAAS Ratings of School EffectRemember:

School Effect Mean of Student Scores – Mean of Predicted Scores

So, if the students perform as expected, School Effect 0.

The ratings are in terms of the expected value of 0.

-2 SEs 0 +2 SEs

Not Detectably

DifferentAboveBelow

Standard Error (SE) is a measure of variation in the prediction.

Writing, Science & Grade 11 Reading/Math Growth Descriptors

Progress is defined by the district or school effect, which is a measure of the difference between observed performance and predicted performance.

Progress is significantly above the progress of the average district in the state.

Progress is not detectably different from the progress of the average district in the state.

Progress is significantly below the progress of the average district in the state.

PVAAS School ReportPVAAS School ReportGrades 11- Math and ReadingGrades 11- Math and Reading

Grades 4, 8 & 11 – ScienceGrades 4, 8 & 11 – ScienceGrades 5, 8 & 11 – WritingGrades 5, 8 & 11 – Writing

Summary ofSummary ofPVAAS Growth MethodsPVAAS Growth Methods

Testing Subjects – Grades Methodology

PSSA in consecutive years

Math & Reading – 4-8Growth

Standard

PSSA not in consecutive years

Writing – 5, 8, 11Science – 4, 8, 11

Math & Reading – 11Predictive

Summary:Summary:One Year’s GrowthOne Year’s Growth

Growth Standard Method

Math and Reading – Grades 4 through 8

Cohort maintains its achievement position.

Predictive Method

Math & Reading – Grade 11Writing – Grades 5, 8, 11 Science – Grades 4, 8, 11

Cohort’s actual performance is as expected.

PVAAS Performance Diagnostic ReportPVAAS Performance Diagnostic Report

The Key is the RED Whisker!The Key is the RED Whisker!

Growth Standard MethodologyGrowth Standard MethodologyWhat the whiskers tell us…What the whiskers tell us…

Growth Standard

(One Year’s Growth)

Met the Growth Standard;Made One Year’s Growth

Yellow

Below Growth Standard;Less than One Year’s Growth

Rose

Exceeded Growth Standard;More than One Year’s Growth

Green

Predictive MethodologyPredictive MethodologyWhat the whiskers tell us…What the whiskers tell us…

Observed – Predicted

(One Year’s Growth)

NDD – No Detectable Difference Between Observed Score and

Predicted Score

BELOW Expectations -- Observed Score Was Less

Than Predicted Score

ABOVE Expectations -- Observed Score Was Greater

Than Predicted Score

Performance Diagnostic ReportPerformance Diagnostic Report

Greater Detail – based on % of studentsGreater Detail – based on % of students

Click on Hot Keys for lists of students.

Student Lists from Performance Student Lists from Performance Diagnostic ReportsDiagnostic Reports

Your District Your District and School and School

DataData

PA’s Intent for PVAASPA’s Intent for PVAAS

Another Tool for Continuous Another Tool for Continuous ImprovementImprovement

Measure of Growth to Add to Measure of Growth to Add to Achievement/Status MeasuresAchievement/Status Measures

Not to Be Used in IsolationNot to Be Used in Isolation

Questions and FeedbackQuestions and Feedback

Questions:Questions:

PVAAS Materials or PVAAS Materials or Statewide ImplementationStatewide Implementation

pdepvaas@iu13.orgpdepvaas@iu13.org

717-606-1911717-606-1911

PVAAS Report Web SitePVAAS Report Web Site

https://pvaas.sas.comhttps://pvaas.sas.com

www.pde.state.pa.us

333 Market Street Harrisburg, PA 17126