lak12 e2coach presentation

04/10/2023 LAK 12: Vancouver

What to do with actionable intelligence: E2Coach as an intervention engine

Tim McKay, University of Michigan Departments of Physics and Astronomy, LSA Honors Program

How learning analytics and computer tailored

communications enable us to provide individualized

feedback, encouragement, and advice to thousands of

students

Mysteries of teaching and learningWhat do students know when they come? What

do they do while in class? What do they learn and remember?

Can we predict outcomes? When? For whom?Can we change outcomes? What should we do?

When? How? For whom?


Learning Analytics and Knowledge• Education systems generate

a rich stream of increasingly accessible data which can inform teaching and learning

• I’m from big data cosmology, how 100s of millions of galaxies get to be the way they are. We’re exploring what data tells us about how students get to be the way they are

• Learning Analytics: the Better-Than-Expected project for introductory physics– Generates actionable

intelligence• An intervention: adaptation

of PH computer tailored communication in E2Coach– Acts on the intelligence

provided by LA

Our E2Coach application – grown from analytics, aware of general information like identity and goals, reaching into real time, content

specific student performance data


Better-than-expected in Physics

• LA investigation of two year-long intro physics sequences

• 48,579 students over 14 years

• Institutional data => construct predictions of student outcomes

• Identify those who do better (and worse) than expected: Find out why

• Admissions information– High school GPA– SAT and ACT– State and Country of origin– First generation and SES– Gender

• Internal UM information– Cumulative GPA– Number of credits: UM and

transfer – Exam scores– Homework grades– Final grade in this courseMuch more data is available…


Essential findings from BTE

• Student grades can be predicted with half letter grade accuracy– Incoming UM GPA the

most powerful predictor– Weak additional

information in SAT/ACT Math

• There is real dispersion: students do better (and worse) than expected

• Significant performance disparities are apparent– Gender: especially strong

in courses where female students are seriously underrepresented

– First generation college students

– Students from low socio-economic status households


Mean and error on the mean for each bin

One sigma dispersion around the mean for each bin

One-to-one line…


Exploring BTE/WTE

Gendered performance disparity seen in intro physics courses nationwide

Male students

Female students

Non first-gen students

First-gen students

Performance disparity seen for first-generation college students, also for low SES students…

Learning analytics make all of these explorations possible, even

the qualitative ones. They provide actionable intelligence.


What to do with actionable intelligence?

• We now have John Campbell’s ‘obligation of knowing’: how we expect them to perform, and what leads to success

• How can we tailor our approaches and interactions to optimize the success of all?

• Options for response:– Tell someone and let

them act: student, instructor, advisors: scale remains a challenge

– Develop tools which act directly in response to student state:• Intelligent tutors etc.• Computer tailored

communication systems


Tailoring is well established and tested in public health, and has seen major commercial application. An extensive body of peer-reviewed research reports on tests of efficacy in design across interventions ranging from smoking cessation and

diabetes control to cancer treatment decision making and depression. This research provides a strong base for the design of new computer tailored interventions


MTS was built by the University of Michigan’s Center for Health Communications Research, an established leader in

computer tailored public health interventions. MTS is a mature, fully open-source software system for computer

tailored communication.


E2Coach: tailored support for

physics students• Used LA and MTS to

construct “E2Coach”: an Electronic Expert coaching system for intro physics courses

• You can find a basic information about the project online:

http://sitemaker.umich.edu/ecoach

• Three groups of players:– Department of Physics– CHCR leadership and

staff– Consultants from across

the campus• Project goals:

– Improved performance and affect for all students

– Reduced disparitiesThe E2Coach team:

Tim McKay, Kate Miller, Jared Tritz, Gus Evrard, Dave Gerdes in Physics

Vic Strecher, Ed Saunders, Holly Derry, Mike Nowak at CHCR


How does E2Coach work?

MTS

Detailed information

about thousands of students and their current status

Expertise of hundreds of students, dozens of instructors and behavior change experts Individually

personalized messages:

what we all agree we

would say to each

student, if only we could…

The Michigan Tailoring System: a mature open-source software system for

creating content designed specifically for an individual based on data about that

individual

Where the real effort lies


Expertise and Information

• Structured interviews with faculty

• Survey of 70+ student study group leaders

• Better-than-expected interviews

• Input from students with different backgrounds has extreme relevance!

• Knowledge of each course and its structure

• Real-time input from the course gradebook

• Input from the student– Background, goals and

interests, planned effort, desired and expected grades, self-efficacy, confidence in physics

• Opt-In: 54% (953 total)

04/10/2023

What we provide• Tailored advice on all aspects of the course,

including testimonials from relevant peers


Performance feedback


First measures of impact

• First term ended four days ago: final scores for enrolled students 2.3% (4) higher

• Currently examining effects vs. usage

• Disparities on gender, SES, first-gen status

• This is a complex intervention, with many parts: which are key?

• Testing in fall using fractional factorial design

Score differences observed in 104 random samples

Actual measured score difference


E2Coach in the LA landscape…

SOLAR: Open Learning

Analytics: an integrated & modularized

platform

Siemans, G., et al. July 2011http://solaresearch.org

BTE project and other analytics

Computer tailored

communication


Learning Analytics at Michigan

• Redesigning for Physics in the fall– Full enrollment– Much tighter approach

• Expanding to situations w/diverse student bodies– Intro Stats: 1800 – Epidemiology: Masters– Freshman orientation

• UM Provost has empowered Learning Analytics Task Force

• Charge:– Improve information

environment for LA– Support LA projects with

three funding cycles– Revise institutional

metrics used for teaching and learning

lak12 e2coach presentation

Economy & Finance