20 years: what the data show and what they don't (287820054)

20 Years: What the Data Show and What They Don’t

Chuck Dziuban & Patsy Moskal

Center for Distributed LearningResearch Initiative for Teaching Effectiveness (RITE)

University of Central Florida

About UCF• Orlando, FL• Over 63,000 students; 2nd largest university in U.S.• Carnegie classification: RU/VH Research University• 216 degree programs; 11 colleges; 11 campuses

• Academic Year 2014-15• 38% of total university student credit hours• 78% of all students took at least one online course

• 80% of all undergraduates (47,116)• 61% of all graduate students (6,469)

The Evaluation PlanStudents

Faculty Institution

Where’s the data??

LMS

ApplicationsSPI

SIS

Surveys

The end goal…

Turning data…

…into INFORMATION

Some Lessons Learned Through the Years

Success

Success

Summer 12 Fall 12 Spring 13 Summer 13 Fall 13 Spring 140

102030405060708090

10090 87 88 91 87 88

94 90 92 94 91 9089 87 90 90 89 89

F2F (n=647,390) Blended (n=73,629)Fully Online (n=189,208)

Perc

ent

1. Does success equal learning?• Answer: No

2. Are the differences significant?• Answer: Yes

Success

Question that was never asked and should have been…

• Is there selection bias in your data?• Answer: Yes

Significance may not be significant

The question for all ages

“How large does my sample have to be?”

Statistical (classical) hypothesis tests are a function of 3 things:

1) Significance Level .05? .01? …or something else?

2) Sample Size

Tiny? Small? Medium? Large? Huge?3) Some Effect Size

A difference that means something to me∆1 –Doesn’t matter

∆2 –Really important to me

How much is enough?

I don’t care about this

∆1

How much is enough?

∆2I care

about this

Statistical significance testing (SD = 15)Sample Size

27502500225020001750150012501000750500

x1=100 x2=101 ES=.06.01.02.03.04.05.07.10.14.20.29

Significant

NotSignificant

So the strategy is…

1) Pick ∆2 first This is important to me

2) Then pick a significance level .05, .01, or something else

3) Pick a sample size that will catch ∆2 but not ∆1

When treatment is not treatment

Blended learning as a boundary object

Blended Learning

Evaluators Journalists

Students

Faculty

DeansLibrarians

Provosts

The way we blendOnline tutorials

E-mails

WEB MODULES

CoMpUtEr LaBs

Optional websites

Tele-web software

Computer cluster room

Internet tutorials

Electronic field trips

Supplemental web

activities

Tech equipped classrooms

Evidence-based practice

Discussion

Virtual experiments

The way we measure

MuLtIpLe ChOiCe

Class ranks

student ratings

Class work

Placement tests

Motivation questionnaires

Achievement tests

Cooperativeness scales

Mathematics anxiety scales

Self-confidence scales

Self-report surveys

APPREHENSION TESTSReading

power

Conceptual testsElectric circuit tests

Can you predict success and

bandwidth analytics?

Add one logistic regression analysis for predicting non-success DFW (n=258,212)

  R2

Modality .003Course Level .022Class Size .024Gender .029Ethnicity .035Age .035SAT .034College .047High School GPA .074Cumulative GPA .405

D* 1

Classification and regression tree for predicting non success in online courses

Overall Non Success

D* 3-10

D* 2

13.3%n = 70948

34.3%n = 39646

7.5%n = 31302

24.4%n = 14070

44% n = 25576

* Deciles Prediction Accuracy 94%

D* 1-2

•Can you take action on GPA?•Answer: Yes and No

•Why does GPA predict?

Scarcity and the cognitive bandwidth tax

Could it be scarcity?

Scarcity and the cognitive bandwidth taxC

ogni

tive

Ban

d W

idth

Scarcity Factors

Tuition Housing Books Transportation Child Care

Work Add. Cost

Safety

Scarcity

Scarcity and the cognitive bandwidth taxC

ogni

tive

Ban

d W

idth

Scarcity Factors

Tuition Housing Books Transportation Child Care

Work Add.Cost

Safety

Cognitive Bandwidth

Is there an answer to scarcity?

Harris Rosen

www.tangeloparkprogram.com

Program Components

Early Childhood Education

ParentLeadership

ScholarshipsVocational

Alumni

Tangelo Park Crime Rates 1994-2013Standardized by 1993 Figures

94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 10 11 12 13

Crim

e R

ates

0

20

-20

-40

-60

-80

+3% +4%

-38%-30%

-4%

-33%

-38%-45%

-49%

-67%-61%

-35%-47%

-53%

-45%

-52%

-48%-53%-66%

-66%

Completion rates for those who entered

Community college Vocational school College/ university Graduate school0

102030405060708090

100

32

8377

83

Perc

ent

94 95 96 97 98 99 00 01 02 03 04 05 06 07 08 09 10 11 12 130

102030405060708090

100

0 0

2836

5867 70 70 68 72 87 76

8288 87 86

76 73 74 74

100

7264

4233 30 30 32 28

1324

1812 13 14

24 27 26 26

Perc

ent

2-yr-old program

Scholarship

TPP Budget: 2-yr-old program and scholarship (percent of total)

Return on Investment to Society

• Total 20 year investment: $10,000,000Half for Early Childhood programHalf for scholarships

• Lance Lochner, University of Western Ontario

• Return on investment: $7 for every $1 spent

Come together blended research

Louis Guttman meets Brené Brown meets the Center for Fiction

Literary Criticism

Image Analysis

Survey

Grounded Theory

Research Initiative for Teaching Effectiveness

For more information contact:Dr. Chuck Dziuban

(407) 823-5478 Charles.Dziuban@ucf.edu

Dr. Patsy Moskal(407) 823-0283

Patsy.Moskal@ucf.edu

http://rite.ucf.eduhttp://www.if.ucf.edu/