1/∞CRESST/UCLA

Towards Individualized Instruction Using Technology

Gregory K. W. K. Chung

Annual CRESST ConferenceLos Angeles, CA – January 22, 2007

UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing

2/∞CRESST/UCLA

Structure of Talk

• Problem statement

• Research questions

• Technical approach to diagnoses and prescription

• Study design

3/∞CRESST/UCLA

Problem Statement

4/∞CRESST/UCLA

Released NAEP Items

NAEP 1990.69 correct

Current StudyLo: .73 correctHi: .87 correct

NAEP 2003.48 correct

Current StudyLo: .48 correctHi: .69 correct

NAEP 2003.52 correct

Current StudyLo: .63 correctHi: .87 correct

.37 incorrect

.13 incorrect

.52 incorrect

.31 incorrect

.27 incorrect

.13 incorrect

.48 incorrect

.52 incorrect

.31 incorrect

5/∞CRESST/UCLA

Even Simpler Problems

Lo: .36 correctHi: .68 correct

Lo: .73 correctHi: .88 correct

Lo: .68 correctHi: .90 correct

Lo: .60 correctHi: .60 correct

Lo: .56 correctHi: .90 correct

6/∞CRESST/UCLA

Study Context

• Why pre-algebra?

• Pre-algebra provides students with the fundamental skills and knowledge that underlie algebra

• In fall 2004, 57% of CSU first-time freshmen needed remediation in mathematics (22,000 students)

• Remediation includes basic math, pre-algebra, geometry, and algebra

• By fall 2005, 3500 students did not complete remediation and were disenrolled

7/∞CRESST/UCLA

Study Context

• Tasked with developing instructional and assessment supports for pre-algebra

• Develop an approach to support rapid diagnosis and remediation of pre-algebra knowledge and skill gaps

• Develop assessment and instructional tools to support classroom instruction

• Test approach with middle school students

• Do it with technology, do it now

8/∞CRESST/UCLA

Research Questions

9/∞CRESST/UCLA

Research Questions

• What is the architecture for a diagnosis and remediation system?

• What are necessary components of such a system, how is diagnosis performed, how is remediation performed, how is the system validated, how is effectiveness measured?

10/∞CRESST/UCLA

Research Questions

• To what extent can a Bayesian network codify and capture the structure and properties of pre-algebra?

• This question addresses the use of BN for representing domain knowledge (i.e., pre-algebra) and automated reasoning.

11/∞CRESST/UCLA

Research Questions

• What are methods for developing “instructional parcels”?

• What is the structure of the student interaction, what is the format of the media delivery, what are techniques that can engage students, what are the scalability issues, what are the development bottlenecks?

12/∞CRESST/UCLA

Basic Approach

Automated Reasoning

(Bayes net of pre-algebra)

Pre-algebra pretest

adding fractions distributive

property

multiplyingfractions

Individualized instruction

Individualized posttest

13/∞CRESST/UCLA

Domain Analysis

• Specify domain structure as a Bayes net

• Specify concepts and influences among concepts

• Attach math items (“observables”) to concepts to form scales

• Develop assessment items using a step-by-step derivation of a few complex problems

• Internally coherent

• Enables precise diagnoses

14/∞CRESST/UCLA

Pre-algebra Bayes Net

definitions

operations

transformations

common errors

15/∞CRESST/UCLA

Domain Sampling

16/∞CRESST/UCLA

Sample Items

DP (OPER)

CP-ADD (CE)

AP-ADD (CE)

DP (CE)

DP (CE)

DP (CE)

DP (OPER)

DP = distributive propertyCP-ADD = commutative property of additionAP-ADD = associative property of addition

CE = common errorOPER = operation

17/∞CRESST/UCLA

Instructional Design Issues

18/∞CRESST/UCLA

Study

19/∞CRESST/UCLA

Study Design and ProcedurePretest

(1st occasion)

Assign Instructi

onal Parcels

to S

Score Tests

(at UCLA)

Generate Predictio

ns

Assign S to

Conditions

Occasion 1

Occasion 2

Computer Instruction

Posttest

Experimental

Condition

Posttest

ControlCondition

Computer Instruction

2 days

20/∞CRESST/UCLA

Expected Relations

Pretest Posttest

Pretest Posttest

Pretest Posttest

Pretest Posttest

Low

Pri

or

Kn

ow

led

ge

Hig

h P

rior

Kn

ow

led

ge

No Instruction Instruction

Score Score

Score Score

21/∞CRESST/UCLA

Results (L/M Prior Know.)

aWilcoxan signed rank test (non-parametric paired test).

Scale

Instruction No Instruction

neg. pos. tie pa neg. pos. tie pa

Adding fractions 2 19 5 <.01 1 4 0 ns

Multiplying fractions 8 20 5 .06 3 4 3 ns

Reducing fractions 2 2 2 ns 1 0 3 ns

Transformations 5 23 6 <.01 1 8 3 .02

Distributive property 8 17 5 .02 6 3 1 ns

22/∞CRESST/UCLA

Results (L/M Prior Know.)

aWilcoxan signed rank test (non-parametric paired test).

Scale

Instruction No Instruction

neg. pos. tie pa neg. pos. tie pa

Commutative property of addition

1 1 5 ns 1 1 1 ns

Commutative property of multiplication

0 5 2 .04 3 0 2 .08

Associative property of addition

3 3 3 ns 1 1 5 ns

Multiplicative inverse 5 9 5 ns 1 4 0 ns

Multiplicative identity 6 19 6 <.01 4 2 4 ns

23/∞CRESST/UCLA

Discussion

• Preliminary results promising

• Individualization of instruction and assessment tractable

• Instructional parcels appear effective but more work needed

• Next steps

• Refine items and domain sampling

• End-to-end automation

• Continue gathering validity evidence

24/∞CRESST/UCLA

Questions

25/∞CRESST/UCLA