Engaging Students: Developing and Measuring

Guiding Principles for Learning

Chad ComptonPaul FreebairnRonald Miller

Sela Unga

Brigham Young UniversityApril 29, 2013

About BYUH

• Small campus• Diverse student population• Student focused

Challenge

• Improve Quality• Serve More Students• Decrease Costs

Half Day Faculty Retreat

• Small group discussion of principles and best practices of teaching and learning.

• Flip chart paper posted on wall (80 comments)

• 10 dots for all faculty to vote

• Top 20 best practices

Steering Committee

• Purpose: Develop framework of teaching and learning (guiding principles)

• 12 faculty, 4 students• Weekly meetings (read and discuss)

1+ years• Student/faculty interviews and focus

groups

First Things

• Teaching as a craft, art, and science

• Focused on principles of learning

Result: 6 Principles of Learning Framework

1. Motivated by desire to serve2. Active Not Passive3. Self directed and responsible4. Meaningful self reflection and

assessment5. Open to change6. Skilled in learning process

Principles to Practices

• Prepare

• Engage

• Improve

Implementation• General Education • Coordinated with student life• Increased classroom visits (lunch)• Emailed inspirational quotes regularly• Annual faculty and department reports • Presented framework to faculty and students• Implemented a 1-year new faculty orientation• Organized the Center for Teaching and Learning• Held on and off-campus seminars with guest speakers• Created brochures, website resources, and video vignettes• Conducted a student time study• Benchmarked student learning activities, opinions, and

time usage

Institutional Research TeamThe Committee

Administrators, Faculty, IR Staff, Students

IR InternSupervisor

Student Worker

Voluntee

r

Voluntee

r

Student Worker

Voluntee

r

Voluntee

r

Student Worker

Voluntee

r

Voluntee

r

Student Worker (Focus

Groups)

Voluntee

r

Voluntee

r

Framework Survey Development

Institutional Research Team (IRT) given task to

benchmark student learning activities, opinions, and time

usage

Fall 2011: IRT developed potential survey questions to provide insight about student

engagement in the learning process

Framework Survey

• About Yourself – course load, credit load, # of classes requiring preparation beforehand,

hours spent per week on various activities (school & work, exercise & entertainment, and church, service, & other)

• About the Framework– 2 open-ended questions: understanding and application of the Framework

• About Your Learning– Prepare, Engage, and Improve– 5-point scale: never (1), hardly ever (2), sometimes (3), often (4), very

often (5)

• Demographics – Gender, Age, Regional Home Area, Class, GPA, etc.

Framework Survey: About Your Learning

Prepare• Read and study materials before class• Prepare thoughtful questions before class• Understand assigned materials well enough to teach or explain it

Engage• Set personal goals • Actively participate in class • Work with other students outside of class

Improve• Take initiative to learn on your own• Increase the effectiveness of your study skills• Engage in self-assessment and meaningful reflection

Encouraging Students

Teachers giving more busy work

Training us as future leaders

Applying what we learn

Time management skills

Enter to Learn Go Forth to Serve

Spiritual Learning

Tests, Quizes, Classes, Graduating

I don't know or not familiar with it

Improve

Engage

Did not respond appropriately to the question

Prepare

Studying, Secular Learning, Education

1%

2%

2%

3%

4%

6%

6%

12%

16%

18%

21%

23%

23%

30%

In your own words, describe the BYU-Hawaii Framework for Student Learning

The majority of students

described the Framework as

“preparing”, “engaging”, or “improving”

their learning

39% of the respondents did not

know, did not respond to the

question appropriately, or were not familiar

with the Framework

Text analysis taken from Framework Survey

Assign more homework

They are merciful & patient

Review material from previous class period

Teachers are open to different views & perspectives

Require you to teach (i.e. Videos)

Expect concepts to be learned outside of class

Ask for feedback (i.e. Class flexible to student's understanding)

Office hours and tutors (i.e. Availability to students)

Group work & in-class activities

Ask students questions

Allow students to ask questions & help them understand

Encouragement to participate

Teach real life application & lead by example

Syllabi, instruction, assignments, & grades based on the framework

Spiritual activities & application (i.e. Prayer in class)

Class Discussions

Required pre-class preparation

1%

1%

1%

2%

2%

2%

3%

3%

4%

4%

5%

10%

11%

12%

14%

18%

19%

22% of the re-spondents gave

examples of how instructors help them to engage in the learning

process

Please give a few specific examples of how you see the BYU-Hawaii Framework for Student Learning being applied by

instructors in your class

37% of the respondents gave

examples of pre-class preparation or class

discussion when describing how the

Framework was being applied.

Text analysis taken from Framework Survey

Prepare: As a student, how often do you…?

1

2

3

4

5

2.84

3.68

2.713.31

3.713.25

Overall Mean

very often -

often -

sometimes -

hardly ever -

never -

Additional findingssingle married p

prepare thoughtful questions to ask in class 2.78* 3.10* 0.004

thoroughly read and study materials in advance 3.62* 3.95* 0.004

Female Male p

prepare thoughtful questions to ask in class2.74* 2.96* 0.019

Significant effects: p<.05* 1= never; 2= hardly ever; 3= sometimes; 4= often; 5= very often

Engage: As a student, how often do you…?

1

2

3

4

5

3.82

3.363.10

2.71

3.613.32Overall Mean

very often -

often -

sometimes -

hardly ever -

never -

Additional findingsMainland

USA Asia Hawaii PacificOther

International p ŋp2

discuss ideas from your readings or classes with faculty members outside of class 2.54* 2.99** 2.42* 2.98 2.85 0.000 0.046

Freshman Sophomore Junior Senior p ŋp2

discuss ideas from your readings or classes with faculty members outside of class 2.55* 2.49* 2.53* 3.07*** 0.000 0.062

Female Male p

discuss ideas from your readings or classes with faculty members outside of class 2.59* 2.88* 0.004

single married p

discuss ideas from your readings or classes with faculty members outside of class

2.62* 3.07* 0.000

actively participate in class by asking questions, expressing your point of view, and responding to others’ views

3.55* 3.85* 0.011

Significant effects: p<.05* 1= never; 2= hardly ever; 3= sometimes; 4= often; 5= very often

Improve: As a student, how often do you…?

1

2

3

4

5

3.75 3.673.95 3.83 3.75

3.79

Overall Mean

very often -

often -

sometimes -

hardly ever -

never -

Additional findingsMainland

USA Asia Hawaii PacificOther

International p ŋp2

increase the effectiveness of your study skills and habits 3.51* 3.87* 3.67 3.70 3.94* 0.001 0.039

change the way you think or act because of what you’ve learned in class 3.74* 3.84 3.91 3.89 4.21* 0.043 0.022

Freshman Sophomore Junior Senior p ŋp2

seek further opportunities to learn 3.79 3.56* 3.63 3.89* 0.011 0.024

single married p

seek further opportunities to learn 3.70* 3.96* 0.009

increase the effectiveness of your study skills and habits 3.61* 3.90* 0.004

take initiative on your own to learn things that you don’t understand 3.89* 4.16* 0.004

change the way you think or act because of what you’ve learned in class

3.74* 3.84* 0.043

Significant effects: p<.05* 1= never; 2= hardly ever; 3= sometimes; 4= often; 5= very often

About Your Learning: Prepare, Engage, and Improve

discuss ideas from readings with faculty outside of class

prepare thoughtful questions

study assigned materials with others before class

work with other students outside of class

understand materials well enough to teach

test ideas with others in class

actively participate in class

ask for guidance from the spirit

increase the effectiveness of study skills

thoroughly read and study materials before class

seek further opportunities to learn

engage in self-assessment

set personal goals and standards

change the way you think or act

take initiative on your own

0% 10% 20% 30% 40% 50% 60% 70% 80%

21%

22%

23%

35%

42%

46%

53%

56%

58%

60%

62%

63%

65%

67%

75%

Percentage of students responding with often or very often

Prepare Engage

Improve

More students...

• take initiative on their own to learn• thoroughly read and studied

materials before class• set personal goals • engage in self-assessment

About Your Learning: Prepare, Engage, and Improve

take initiative on your own

seek further opportunities to learn

engage in self-assessment

thoroughly read and study materials before class

ask for guidance from the spirit

test ideas with others in class

prepare thoughtful questions

discuss ideas from readings with faculty outside of class

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

2%

2%

6%

6%

7%

9%

11%

12%

13%

16%

18%

28%

34%

42%

44%

Percentage of students responding with hardly ever or never

Prepare Engage

Improve

Fewer students...

• discussed ideas with faculty outside of class

• studied before class with others• prepared thoughtful questions

before class• worked with others outside of class

Exploratory Factor AnalysisQuestions Factor 1 Factor 2 Factor 3

Prepare

prepare thoughtful questions to ask in class 0.085788 0.462264 0.612411

thoroughly read and study materials in advance 0.238854 0.016551 0.707059

study with others to make sure you understand assigned materials 0.154039 0.141104 0.734530

understand assigned materials well enough to teach or explain it 0.079345 0.735859 0.224951

ask for guidance from the Spirit 0.326257 0.106098 0.367008

Engage

set personal goals and standards 0.558461 0.193091 0.230917

test ideas with others in class 0.308393 0.562841 0.335045

work with other students outside of class 0.150854 0.865002 -0.061181

discuss ideas from readings and classes with faculty members 0.213787 0.620976 0.279427

actively participate in class 0.182878 0.203032 0.550451

Improve

seek further opportunities to learn 0.731545 0.173406 0.218000

increase the effectiveness of your study skills and habits 0.733764 0.208398 0.125593

take initiative to learn on your own 0.765110 -0.010432 0.170907

change the way you think or act 0.741580 0.144300 0.142496

engage in self-assessment and meaningful reflection 0.740721 0.145936 0.153575

New Structure

Questions

Prepare

prepare thoughtful questions to ask in class

thoroughly read and study materials in advance

study with others to make sure you understand assigned materials

Engage

understand assigned materials well enough to teach or explain it

work with other students outside of class

discuss ideas from readings and classes with faculty members

Improve

seek further opportunities to learn

increase the effectiveness of your study skills and habits

take initiative to learn on your own

change the way you think or act

engage in self-assessment and meaningful reflection

Validity & Reliability

New Structure

Chi-Square Statistic/p-value 395.637/ p < 0.000

Steiger-Lind RMSEA 0.135

Population Gamma Index 0.873

Joreskog GFI 0.860

Akaike Information Criterion 0.995

Cronbach’s alpha 0.836

Original Structure

Chi-Square Statistic/p-value 750.462/ p < 0.000

Steiger-Lind RMSEA 0.127

Population Gamma Index 0.838

Joreskog GFI 0.820

Akaike Information Criterion 1.838

Cronbach’s alpha 0.866

Time Usage Study

Winter 2012: The weekly schedule and

daily time usage survey were developed

Selected random groups of students to participate in the

Time Study during winter, summer, and fall 2012

Students first completed the Learning Framework

Survey before tracking their time.

1 Week at a GlanceLearning Framework Survey

(predicted from random samples of the entire student population- n=382)

Class Prepare StudyCampus

ActivitiesWork

Exercise and Entertainment

Church and

Service

Eat, Clean, Sleep

Total Hours

Mean 13.50 hrs.

6.99 hrs.

12.18 hrs.

3.08 hrs.

14.19hrs.

29.07 hrs.

7.40 hrs.

44.37 hrs.

130.78hrs.

Median 14 5 10 2 19 24 6 50

Mode 15 10 10 2 19 24 6 68

1 Week at a GlanceLearning Framework Survey

(predicted from time study participants- n=105)

Class Prepare StudyCampus Activitie

sWork

Exercise and Entertainment

Church and

Service

Eat, Clean, Sleep

Total

Mean14.04 hrs.

6.83 hrs.

11.56 hrs.

3.19 hrs.

14.05 hrs.

31.29 hrs.

8.20 hrs.

53.62 hrs.

142.78 hrs.

Median 14 4 8.5 2 19 23 7 58

Mode 14 4 15 2 19 21 6 62

Time Study(actual time recorded in student daily time logs- n=105)

Mean12.25 hrs.

5.64 hrs.

16.09 hrs.

3.16 hrs.

13.36 hrs.

42.60 hrs.

7.94 hrs.

60.47 hrs.

161.51 hrs.

Median 12 4 15.5 0.25 16 38 6 64.5

Mode 12 0 17 0 0 47 5 67

Time Study

Campus Activities

Prepare

Church and Service

Class

Work

Study

Exercise and Entertainment

Eat, Clean, Sleep

0 10 20 30 40 50 60 70

3.2

5.7

7.9

9.8

13.4

16.1

42.6

60.5

3.2

6.8

8.2

14.0

14.1

11.6

33.8

53.6

PredictedActual

Hours spent per week

Time Study Key Findings

• underestimated time spent studying outside of class

• underestimated time spent on exercise and entertainment-related activities

We're in it for the long-term

• Creating new culture of learning takes time, resources, leadership, and buy-in

• No "quick fix" available

• Patience, persistence, and vision needed

• What's next?

Please Contact Us

• Chad Compton, Associate VP for Instruction chad.compton@byuh.edu

• Paul Freebairn, Dir. Assessment & Testing freebaip@byuh.edu

• Ronald Miller, Associate Professor of Psychology ronald.miller@byuh.edu

• Sela Unga is the IR Planning Analyst Intern unga2@go.byuh.edu