©2012 carnegie learning, inc. in-vivo experimentation steve ritter founder and chief scientist...

©2012 Carnegie Learning, Inc.

In-vivo Experimentation

Steve RitterFounder and Chief Scientist

Carnegie Learning

©2012 Carnegie Learning, Inc.

An attempt to find meaning in three acts

• Design: Geometry Contiguity (Vincent Aleven, Kirsten Butcher)

• Modeling: Adjusting learning curve parameters (Cen, Koedinger, Junker)

• Personalization: Word problem content (Candace Walkington)

©2012 Carnegie Learning, Inc.

DESIGN

©2012 Carnegie Learning, Inc.

Geometry angles

©2012 Carnegie Learning, Inc.

Contiguity

Early Version Commercial Version(Carnegie Learning)

Research Version

(Carnegie Mellon)

Butcher, K., & Aleven, V. (2008). Diagram interaction during intelligent tutoring in geometry: Support for knowledge retention and deep transfer. In C. Schunn (Ed.) Proceedings of the Annual Meeting of the Cognitive Science Society, CogSci 2008. New York, NY: Lawrence Earlbaum.

Hausmann, R.G.M. & Vuong, A. (2012) Testing the Split Attention Effect on Learning in a Natural Educational Setting Using an Intelligent Tutoring System for Geometry. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society. (pp. 438-443). Austin, TX: Cognitive Science Society.

©2012 Carnegie Learning, Inc.

Early Tutor

©2012 Carnegie Learning, Inc.

Revised (commercial) tutor

©2012 Carnegie Learning, Inc.

Geometry Contiguity

• Design and field experimentation– Butcher and Aleven (2008)• Diagram interaction led to better transfer

and retention

• Analysis of impact– Hausmann and Vuong (2012)• Unit-level effects mixed• Advantage for harder skills

©2012 Carnegie Learning, Inc.

Geometry Angles

©2012 Carnegie Learning, Inc.

Lessons

• Change is constant• Transition from research to

production always requires adaptation

©2012 Carnegie Learning, Inc.

MODELING

©2012 Carnegie Learning, Inc.

Skillometer

©2012 Carnegie Learning, Inc.

Expression Writing

©2012 Carnegie Learning, Inc.

What gets learned?

©2012 Carnegie Learning, Inc.

Bayesian Knowledge Tracing

Cognitive tutor traces these skills differently

©2012 Carnegie Learning, Inc.16

Learning Curve Parameter Fitting• Field study looking at learning area of geometric figures

– One group used adjusted learning parameters based on previous year’s data

• Optimized group took 12% less time to reach same performance

• Significant learning gain in both groups• No difference in learning gain between groups (p = 0.772 )

0

20

40

60

80

100

120

Square

Parallelogram

Triangle

Trapezoid

Polygon

Circle

Optimized

Control

©2012 Carnegie Learning, Inc.

Lessons

• Learning efficiency is a great outcome

• Small, systemic changes can have big impact

• Optimizing skills requires appropriate skill model– Koedinger, McLaughlin and Stamper

(2012) - LFA

©2012 Carnegie Learning, Inc.

PERSONALIZATION

©2012 Carnegie Learning, Inc.

Word problem customization

©2012 Carnegie Learning, Inc.

Personalization field study

• Students who got problems related to their interests made fewer errors

• Also affected subsequent unit• Interaction with readability

©2012 Carnegie Learning, Inc.

Lessons

• Content matters– Challenge for knowledge component

modeling

• Are we personalizing preferences, reading level or both?

©2012 Carnegie Learning, Inc.

Summary

• It’s not about whether A is better than B– It’s about why A is better than B