MCAS Standards Validation:MCAS Standards Validation:High School Introductory High School Introductory

Physics Physics

Sheraton HotelBraintree, MA

September 17-18, 2007

Overview of August Overview of August Standards SettingStandards SettingIndependent standard-setting for:

BiologyChemistryIntroductory PhysicsTechnology/Engineering

Cut scores recommended for:Failing/Needs ImprovementNeeds Improvement/ProficientProficient/Advanced

Each panel composed of:• high school science teachers and administrators• university representatives• community representatives

Post Standard-Setting Post Standard-Setting AnalysisAnalysis

• Analyzed each of the recommended cut scores (n=12) across all four content areas

• Validated 10 of 12 recommended cut scores

• Issues related to 2 of the recommended cuts:– Raw score ranges associated with two cuts for

F/NI and NI/P for Introductory Physics were extreme outliers

– Some Introductory Physics panelists expressed concern about standard-setting process

Plenary SessionPlenary Session Welcome Overview of August Standard Setting

Meeting Post Standard Setting Analysis Standard Setting versus Standards Validation STE tests: efforts to establish “equivalence”

and uses of test results Recap of Body of Work (BOW) method and

role of performance level descriptors Questions and Answers

Additional Analyses Additional Analyses Conducted Conducted

Relationship of projected cut scores to psychometric properties of test (TCCs and TIFs)

Relationship of previous student performance in science with that of IP test based on recommended cut scores

Relationship of student work classified as Needs Improvement and Proficient to:Introductory Physics performance level descriptorsstudent work classified as Needs Improvement and

Proficient for Biology, Chemistry, and T/E

Conclusions Post Standard-Conclusions Post Standard-SettingSetting

• Interpretation of Needs Improvement and Proficient diverged significantly from interpretation made in the other content areas

• Considered statistical adjustment as remedy but determined Standards Validation preferable given high stakes use of results

• Delay in posting of scaled score/raw score conversion tables for all four content areas until standards validation completed

Remember, we knew this Remember, we knew this was going to be was going to be challenging…challenging…

Standard setting for multiple tests–-where roughly “comparable” standards across all tests is the goal—is technically challenging

Other approaches to standard setting considered:

Overlapping content panelsOne panel composed of all representatives for

all four content areasIndependent panels

General Phases of Standard General Phases of Standard SettingSetting

Data-collection phase

Policy-making/decision-making phase

Standard Setting vs.Standard Setting vs.Standards ValidationStandards Validation

Standard setting– Process of establishing original cut

scores– Panelists are not provided initial cut

points or focused cut point range

Standards validation – Process of validating cut scores– Panelists are provided initial cut points

or a narrowed cut point range

Introductory Physics Introductory Physics Standards ValidationStandards Validation

Warning

Needs Improvement

Proficient

Cut score

needed

Cut score

established

Cut score

needed

Advanced

Standards Validation Standards Validation Bodies of work to be classified represent a narrowed

range of student work based upon:

Standard setting panelists’ recommendations (August)Psychometric and statistical analysesAnalysis of classification of student work for each of the 12

recommended cut scores (conducted by DOE and Measured Progress content experts)

Pinpointing cut score forFailing/Needs ImprovementNeeds Improvement/Proficient

Massachusetts Massachusetts High High School School Competency Competency Determination Determination (CD)(CD)Requirements: Requirements: Class of 2010Class of 2010

Meet local graduation requirements Attain scaled score of 220 or higher

on high school MCAS tests in ELA, mathematics, and STE (or equivalent on MCAS Alternate Assessment)

To earn diploma, a student must:

Efforts to Establish Efforts to Establish “Equivalence” “Equivalence”

of High School STE Testsof High School STE TestsThe High School STE tests:

are based on parallel test design and development processes

are based on performance level descriptors that are comparable in scope and rigor

have been designed to have similar psychometric properties

External Validation of Efforts External Validation of Efforts to Establish “Equivalence” of to Establish “Equivalence” of

STE TestsSTE TestsPerformance level descriptors used in

standard setting have been externally validated by Massachusetts teachers

Technical/psychometric properties analyzed by Dr. Ronald Hambleton, University of Massachusetts, Amherst

Test design and plans for standard setting endorsed by the MCAS National Technical Advisory Committee (TAC)

What is the Body of Work What is the Body of Work Procedure?Procedure?

Standard Setting method where panelists:

examine student work (actual responses to test questions) and performance level descriptors

make a judgment regarding the performance level to which the student work most closely corresponds.

Body of Work Body of Work FundamentalsFundamentals

Examine the student’s responses to multiple-choice questions

Examine the student’s responses to open-response questions

Judge the student’s knowledge and skills demonstrated relative to the PLDs

Panelists do not need to reach consensus on the classifications

Materials Used During Materials Used During Standards ValidationStandards Validation

Performance Level Descriptors• General• Content specific

Bodies of Student Work• Responses to constructed-response

questions• Multiple-choice summary sheet

Rating Forms

General MCAS General MCAS Performance Level Performance Level

DescriptorsDescriptorsNeeds Improvement Students at this level demonstrate partial understanding of subject matter and solve simple problems.

Proficient Students at this level demonstrate a solid understanding of challenging subject matter and solve a wide variety of problems.

AdvancedStudents at this level demonstrate a comprehensive and in-depth understanding of rigorous subject matter, and provide sophisticated solutions to complex problems.

Needs ImprovementOn MCAS, a student at this level

ProficientOn MCAS, a student at this level

AdvancedOn MCAS, a student at this level

Conceptual Understanding and Factual Knowledge

Demonstrates a partial understanding of some facts, concepts, principles, and theories

Uses basic scientific terms

Demonstrates a solid understanding of many facts, concepts, principles, and theories

Uses appropriate scientific terms

Demonstrates a comprehensive, in-depth understanding of many facts, concepts, principles, and theories

Applies scientific terms in an appropriate context

Scientific Process and Skills

Interprets simple data and creates generalized questions without necessarily using the scientific method

Identifies a problem to be solved

Uses most of the steps of the scientific method to design and interpret experiments

Finds solutions to a variety of problems

Designs and evaluates scientific experiments and generates full interpretations of data

Finds solutions to complex problems

Application/Synthesis

Makes simple predictions about a specific topic

Makes predictions based on information given

Justifies predictions in a general sense

Makes sophisticated predictions

Synthesizes a wide array of information from multiple sources

Applies knowledge to abstract or novel situations

GeneralGeneral STE Performance Level STE Performance Level DescriptorsDescriptors

What Next?What Next? Take the Introductory Physics test Discuss the Performance Level

Descriptors Complete the Item Map Complete training round Complete an evaluation form Complete individual ratings Receive feedback from first round of

ratings Discuss feedback and provide final ratings Complete final evaluation form

Ground RulesGround Rules Role of facilitator is to “facilitate” and keep

process on track Process solely focused on recommending

performance standards (cut scores) for MCAS MCAS performance level descriptors are

integral to process but are not up for debate Panelists’ recommendations are vital; however,

final cut scores determined by the MDOE Each panelist must be in attendance for the

duration of the process for his/her judgments to be considered

Each panelist must complete evaluation form at the end of the event

Cell phones off, please!

Agenda Agenda Monday, September 17Breakfast 8:00 am – 9:00 amPlenary 9:00 am – 10:30 am

Break 10:30 am – 10:45 amWork session 10:45 am –12:00 pmLunch 12:00 pm – 1:00 pmWork session 1:00 pm – 5:00 pm

Tuesday, September 18Breakfast 8:00 am – 9:00 amWork session 9:00 am – 12:00 pmLunch 12:00 pm – 12:45 pmWork session 12:45 pm – until completion

Questions?Questions?