University of Maryland Slide 1May 2, 2001

ECD as KR*

Robert J. Mislevy, University of MarylandRoy Levy, University of Maryland

Eric G. Hansen, Educational Testing Service

(builds on work with Linda Steinberg and Russell Almond)

March 6, 2003

* Evidence-centered design as knowledge representation

University of Maryland Slide 2May 2, 2001

Knowledge Representations

A knowledge representation (KR) is a structure for expressing, communicating, and thinking about important entities and relationships in some domain. • Maps, wiring diagrams, physics equations, nested lists.

• Object models, for business systems and computer systems.

• Evidence-centered design models & structures

KRs are surrogates for something else--a real world situation, or a class of situations, or a representation in other KRs.• They capture some entities and relationships, but ignore others.

• The included entities, relationships, and processes are the ontology of the KR -- what kinds of things you think about, and how.

University of Maryland Slide 3May 2, 2001

KRs are useful when they highlight important relationships and make them easier to work with. KRs facilitate analogies across problems and domains.

• In what ways are AP Studio Art, the SAT, Hydrive, and a language proficiency oral interview alike?

KRs make it easier to acquire and structure information.• E.g., ECD design process in ETS Teaching & Learning programs

KRs can facilitate working together.• ECD object model for sharing, re-using, repurposing the elements and processes in

assessments. KRs are significant in planning.

• What will a solution have to look like? What elements in assessments can vary substantially, but what relationships must hold?

Overlapping KRs coordinate work in complex systems.• Multiple ECD KRs, with bridges among them, for different, interrelated parts of assessment

(substance to argument to specs & models to operation to reporting).

University of Maryland Slide 4May 2, 2001

*** Warning -- cognitive overload ***

University of Maryland Slide 5May 2, 2001

Where you usually start:What are all the kinds of things that are

important to know and do, when and how? What does good work look like?Not generally organized according to

assessment arguments. (In ECD, “domain analysis”)

KRs: Idiosyncratic from domains, as evolved to suit domain purposes.

University of Maryland Slide 6May 2, 2001

Where you usually want to go:Operational assessment system: Pieces

and processes that gather, evaluate,and report, to achieve assessment purpose. (In ECD, “assessment delivery system.”)KRs: Object model for delivery system.

University of Maryland Slide 7May 2, 2001

How do you get from here to there? That is, from knowledge about the domain, to objects and

processes that meet the purposes you had in mind?

University of Maryland Slide 8May 2, 2001

What’s in between (1):Assessment argument: What knowledge,

skill, accomplishments, etc., of students do you want do draw inferences about? What do you need to see them say, do, or make?

What circumstances can evoke this evidence? (Messick, 1984)

KRs: Toulmin & Wigmore diagrams%% EH: How about:

What’s in between (part 1)

.. Just a thought..

University of Maryland Slide 9May 2, 2001

What’s in between (2):Organizing argument in KRs that presage the

structure of assessment elements and processes. Still substantively meaningful.

(In ECD, “Domain Modeling”)KRs: ETS “paradigms”; T&L forms; PADI Design Patterns; Bayes nets for arguments;

BEAR construct map structure.

University of Maryland Slide 10May 2, 2001

What’s in between (2):Organizing argument in KRs that presage the structure of assessment elements and

processes. Still substantively meaningful.(In ECD, “Domain Modeling”)

KRs: ETS “paradigms”; T&L forms; PADI Design Patterns, Bayes nets for

arguments.

What’s in between (2, continued):Also has implications for what information you need before

administration of the assessment, and how you can

interpret the results.%% EH: Very good point for

disability access..

University of Maryland Slide 11May 2, 2001

What’s in between (2):Organizing argument in KRs that presage the structure of assessment elements and

processes. Still substantively meaningful.(In ECD, “Domain Modeling”)

KRs: ETS “paradigms”; T&L forms; PADI Design Patterns, Bayes nets for

arguments.

Need to establish correspondence between the

common assessment KRs and the domain-specific KRs that

address key entities and relationships from that domain,

as they need to be organized into the assessment argument, thence

assessment structures. E.g., BioKIDS’ structure/demand

matrices and FOSS’s filled-in construct maps

University of Maryland Slide 12May 2, 2001

What’s in between (3):Models and specifications for operational

elements and processes.(ECD “Conceptual Assessment Framework”)

KRs: Student, Evidence, & Task models; Bayes nets; Measurement-model equations; Task Templates;

Generalized rubrics, scoring algorithms

University of Maryland Slide 13May 2, 2001

The upshot: Work through KRs, get machinery that embodies the substantive

assessment argument, to meet the purposes you had in mind.