Validity Evaluation

NCSA PresentationNAAC/NCIEA GSEG CONSTORIA

Validation is “a lengthy, even endless process” (Cronbach, 1989, p.151)

NAAC/Center for Assessment. 4/30/08

3

Kane’s argument-based framework

• “…assumes that the proposed interpretations and uses will be explicitly stated as an argument, or network of inferences and supporting assumptions, leading from observations to the conclusions and decisions. Validation involves an appraisal of the coherence of this argument and of the plausibility of its inferences and assumptions (Kane, 2006, p. 17).”

Two Types of Arguments

• An interpretative argument specifies the proposed interpretations and uses of test results by laying out the network of inferences and assumptions leading from the observed performances to the conclusions and decisions based on the performances

• The validity argument provides an evaluation of the interpretative argument (Kane, 2006)

NAAC/Center for Assessment. 4/30/08

The NAAC-GSEG

• Builds off of previous projects (NHEAI and first phase of NAAC) that focused on technical documentation

• Collecting and evaluating validity evidence was a significant challenge

• This project focuses on having states (5) create and evaluate validity arguments

• Our focus is NOT on single studies or even sets of studies, but on how the studies support or refute a validity argument

NAAC/Center for Assessment. 4/30/08

NAAC/Center for Assessment. 4/30/08

Process

• Theory of Action

• Interpretive Argument

• Validity Evaluation

Theory of Action

Types of Studies• Surveys

– Teachers and Principals • Skills, resources, supports

– Scorer background– Consequential validity– Parent Surveys– Learner Characteristics Inventories

• Classroom Visit Protocols• Scoring Observation Protocols• Performance-level judgment assignment• Focus group

Prioritizing the Studies

• Based on the Theory of Action… – Which studies will give us the best information to

develop an interpretive argument?– How will we synthesize the study results into our

interpretive argument?

Evaluating the Validity Argument

• Haertel (1999) reminded us that the individual pieces of evidence (typically presented in separate chapters of technical documents) do not make the assessment system valid or not, it is only by synthesizing this evidence in order to evaluate the interpretative argument can we judge the validity of the assessment program.

An approach for organizing synthesis for validity evaluation (Ryan & Marion)Example 1 Example 2

Dimension of Validity Evidence

Response processes Generalization (reliability)

Proposition/ Claim

The rubric captures appropriately the knowledge and skills valued and assessed

Scorers apply the rubric accurately and consistently when scoring the AA‐AAS

Validity Study Questions

What features and components of the GAA portfolio entries and scoring rubric do scorers attend to during the scoring process?

Given training and support materials similar to those used by the state’s AA-MAS vendor, will scorers achieve an acceptable level of consistency and accuracy in their ratings?

Criteria What is acceptable or good enough evidentiary support?

Typically r=.8?

Evidence (Data) Collected

Think-aloud protocols and interviews with a group of scorers (n = 14).

Scores examined for consistency & accuracy for a set of trained scorers (w/out SWD expertise)

Results Summary of results Mixed results

Claim and support?

Is the claim for the proposition supported by the evidence? How strong is the support?

Weak correlational results for inter-scorer agreement

Potential alternative explanations?

Do the results reveal other potential likely explanations?

Scorers of AA-AAS must have content expertise to be effective scorers

Next Steps

• States are completing their studies and comparing their results with their interpretive arguments.

• They will then evaluate the study results interpretive argument comparisons to develop the validity evaluation.

Contact Information

• Jacqui Kearns, Ed.D.NAAC

University of Kentucky www.naacpartners.org• Scott Marion NCIEA www.nciea.org