“creating a more educated georgia” using what you have: observational data and the scholarship...
TRANSCRIPT
“Creating A More Educated Georgia”
Using What You Have: Observational Data and the
Scholarship of Teaching
Catherine FinneganBoard of Regents of University System of Georgia
Agenda
• Introductions and Definitions
• Sources of Data in CMS
• Study Examples– Engagement – Retention – Instruction
University System of Georgia
35 public colleges and universities– 4 Research Universities,– 15 Regional/State Universities – 4 State Colleges – 12 Associate Colleges– 253,552 students– 9,553 full-time faculty
Office of Information and Instructional Technologies
• Supports and coordinates the delivery of innovative technology resources, services, and solutions.
• Establishes a communications conduit among executive management for the university system about information and instructional technology.
Advanced Learning Technologies
• Provides academic enterprise systems and services for USG institutions.
• Fosters the development and implementation of collaborative online degree programs and training materials.
• Conducts research and evaluations to influence policy making, instructional practice and technology development.
Technology Use in Courses
Adapted from Campus Computing Study,2002-2004.
0%
10%
20%
30%
40%
50%
60%
70%
80%
E-mail InternetResources
Webpages forCourse
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Public Univ. Private Univ. Public 4-Yr.College
Private 4-Yr.College
CommunityColleges
Rising Use of IT in Instruction
Percentage of courses using course management tools, by sector, 2000-2004
Adapted from Campus Computing Study,2002-2004.
USG Faculty Use of CMS 2005
• Nearly half (46.3%) of all USG faculty currently use a CMS in their instruction.
• Almost two-thirds of users have increased their usage over time.
• Over two-thirds of users believe that a CMS has provided important advantages in improving student engagement in learning.
• Over two-fifths of non-users would use a CMS if their issues were addressed.
What CMS was Used For
• 90.6% enhanced their face-to-face instruction
• 43.8% deliver fully on-line instruction
• 43.8% deliver hybrid courses
* Based on 46.3% of respondents who were currently using a CMS.
CMS and Student Engagement
• Increased amount of contact with their students (55.6%)
• Increased student engagement with the course materials (63.5%)
• Allowed for inclusion of more interactive activities in their class (54.2%)
• Allowed them to accommodate more diverse learning styles (67.6%)
* Based on 46.3% of respondents who were currently using a CMS.
Evaluation
• Measures the effectiveness of an ongoing program in achieving its objectives
• Aims at program improvement through a modification of current operations
• Two types of evaluations:– Project– Program
Assessment
• Systematic collection, review, and use of information about educational programs undertaken for the purpose of improving learning and development
• Two types of audience:– Accreditation– Accountability
Scholarship of Teaching
• Sustained inquiry into teaching practices and students’ learning in ways that allow other educators to build on one’s findings
• Directed toward other instructors in one’s field and beyond
Now Tell Me
• What you are interested in learning about your teaching practices and your students’ learning?
• What projects are you now conducting?
• What data are you using to investigate?
Surveys
e-Portfolio
Content
SIS
CMS In Scholarship of Teaching
E-learningSystem
Student Online ActivityLOGON
RE-READLECTURE
NOTES
REPLY toMESSAGE
READMESSAGE
LOGOFF
CREATE NEWMESSAGE
Emergence of a New Data Set
= Large Data Set
How is this data different from other inputs to pedagogical research?
• It’s what the students actually did– Compared to self-reporting
• It captures the steps of the process– Rather than the outcome alone
• It’s quantitative
• It’s easy to collect this data across a large number of students.
How can CMS data be used?
• See patterns and trends
• Tell a story that explains the results
• Identify areas of improvement and targeted change
• Evaluate impact of changes
Patterns of Movement in Courses
Comparison of Withdrawing, Non-Successful and Successful Student Access Logs
0
20
40
60
80
100
120
140
160
Access Order
Page Accessed
Withdrawer Non Successful Successful
New evidence for…?
• Course level inquiry
• Cross course and programmatic research
• College-wide policy review
Typical Sources of Data
• Student course evaluations and surveys
• Content analysis
• Grade distributions
• Interviews
• Portfolio review
CMS Data Sources
• Individual, course, group and institutional activity reports
• Assessment reports
• Survey reports
• Discussions
• Assignments
• Content analysis
Advantages of CMS Data
• Data captured automatically as students interact with software
• Reports available at each level (course, group, institution)
• Time parameters of reports allow more timely and granular review
• Consistency of data across time and course
• Instructor control of tools
Disadvantages of CMS Data
• Only reports actions – doesn’t explain them
• Access to data based on role
• “Canned” report data limited
• Data collection dependent on proper formatting of content and assessment
Activity Data Reports Available to Instructors
• Summary of Activity
• Tool Usage
• Components Usage
• Content File Usage
• Entry and Exit Pages
• Student Tracking
Entry Into Reports and Tracking
Available from
TEACH only
List of Available Reports
Date and time parameters can be set.
Summary of Activity ReportsProvides a general overview of student and auditor activity
Information contained• Total number of sessions• Average session length• Average sessions/day
– by weekday
– by weekend
• Most active day• Least active day• Most active hour of day• Least active hour of day
Example Summary of Activity Report
Tool Usage ReportsProvides an overview of how often tools are used
Tools available• Assessments• Assignments• Bookmarks• Calendar• Chat/Whiteboard• Content File• Discussions• Mail• Media Library• Notes• PowerLinks Proxy Tool• SCORM Module• Organizer Page• URL
Information contained• Total number of
sessions for each tool• Average time per
session• Total time for all tool
sessions• Percent time for each
tool compared with total time
Example Tool Usage Report
Component Usage ReportsProvides an overview of how often students use each component of a course
• Component – which component student has accessed
• Visits – total number of times student has visited a component
• Average time/visit – average time students spend per visit
• Total time – total amount of time students spent for all components
• Percent of total visits – relates time spent in a given component compared to
total time spent for all components
ExampleComponent Usage Report
Entry and Exit Page ReportsProvides an overview of pages used most frequently for course entry and exit
• Page Name– which page student entered or exited
• Tool Used– which tool was used to enter or exit
• Page Usage– total number of times student entered or exited from
the page• Percent of Total Usage
– relates the number of times a page is used to enter or exit to total number of entries or exits
Example Entry and Exit Page Reports
Content Usage ReportsProvides an overview of the content files viewed by students
• Content file – the content file that students have
accessed
• Sessions – the total number of content file
sessions
• Percent of Total Sessions – relates the number of content file
sessions to the total number of sessions for all content files
Content File Usage Report
Content File Usage Graph
Student Tracking ReportsProvides an overview of student activities in the course, displaying both general and detailed statistics
First Access Last Access Sessions Total Time• Mail
– Read Messages – Sent Messages
• Discussion– Read Messages – Sent Messages
• Calendar• Chat and Whiteboard• Assessments• Assignments• URL• Media Library• Content Files
Aggregate Student Tracking
Individual Student Tracking
Data from Quizzes and Surveys
• Performance– Displays student scores for quiz submissions
• Item Statistics– Displays performance statistics for individual
questions. Compares the performance of selected students with the entire class
• Summary Statistics– Compares all students’ results in one table
• Class Statistics– Displays class performance for individual
questions
PerformanceDisplays student scores for quiz and survey submissions
Item StatisticsDisplays performance statistics for individual questions.
Item StatisticsDisplays performance statistics for individual questions. Compares the performance of selected students with the entire class
Summary StatisticsCompares all students’ results in one table
Class StatisticsDisplays class performance for individual questions
Additional Data Sources
• Discussions and Mail
• Assignments
• Course Evaluations and Surveys
• Student Information Systems
Now Tell Me
• Considering the projects that you outlined earlier, – What data found in a CMS might
be used to investigate your theories?
– How would you collect this data?– Would you triangulate this data
with other sources?
Typical Statistical Methods
• Frequency Distributions and Trends
• Measures of Central Tendency
• ANOVA
• Regression
Want to play with some data?
• Go to http://www.statcrunch.com
• Create an account
• Upload data file:
ExampleData.xls
• Run Summary Statistics
“Creating A More Educated Georgia”
Studies on Student Persistence and Achievement
Research Setting: eCore®
• Fully online, collaboratively developed, core curriculum courses offered jointly by institutions in the University System of Georgia. Supported by University System.
• Courses include the humanities, social sciences, mathematics, and sciences.
• Over 25 courses and 2000 enrollments in Spring semester
• http://www.gactr.uga.edu/ecore/
Underlyling Problem: Student RetentionOverall Course Retention: Fall 2000-Spring 2003
Withdraw and Complete
43145
128
220 266245
294631
1569
84
82 12185
144204
0%
20%
40%
60%
80%
100%
Fall 2000 Spring 2001 Summer2001
Fall 2001 Spring 2002 Summer2002
Fall 2002 Spring 2003
Term
Percent
Findings from Four studies
• Predicting Student Retention & Withdrawal
• Tracking Student Behavior & Achievement Online
• Examining Student Persistence and Satisfaction
• Perspectives and Activities of Faculty Teaching Online
Study 1: Predicting Student Retention & Withdrawal
• Purpose: to investigate student withdrawal and retention in eCore courses.
• How well can a student’s group membership (completion & withdrawal) be predicted?
• A two group Predictive Discriminant Analysis (PDA) is used to predict students’ withdrawals and completions in online courses.
• Authors: Morris, Wu, Finnegan (2005).
Variables
• Two grouping variables - student completers - student withdrawers
• Nine predictor variables - gender, age, verbal ability,
math ability, current credit hours, high school GPA, institutional GPA, locus of control and financial aid.
Model A: Two-group PDA Predictive Model, Spring 2002
Grouping Variable
Age
Inst Cum GPAHS GPA
SAT-Verbal
SAT-Math
Withdraw
Complete
Inst Cum Cr HR
Gender
Model A : Findings
• The most important predictors in Model A are
- high school GPA
- mathematic ability (SAT-math)
• Model A, prediction with 62.8% accuracy
Model B: Two-group PDA Predictive Model, Fall 2002
Grouping Variable
FA
Locus
Withdraw
Complete
Model B : Findings
• Financial aid showed significant differences between the responses of withdrawers and completers (x2=4.84, df=1, p<.05). Completers were more likely to receive financial aid that withdrawers.
• Locus of control has significant differences between the responses of withdrawer and completer(X2= 4.205, df= 1, p<.05). Completers were more likely to have internal motivation than withdrawers.
• Model B predicted with 74.5% accuracy
Study 1: Summary
• Students withdraw for a variety of reasons.
• Primary instructional reasons for withdrawing included too much work in the online course, preferred the classroom environment, and disliked online instruction.
• High school grade point average and mathematics SAT were related to retention in the online courses.
• Students who completed courses were more likely to have received financial aid.
• Students who completed courses were more likely to have a higher internal locus of control.
Study 2: Tracking Student Behavior & Achievement Online• Purpose: to examine student behavior by
tracking what students do online and how long they spend on each activity.
• Data: analyzed student access tracking logs.
• Coded over 300,000 student activities.
• Frequency: number of times student did a behavior
• Duration: time spent on the behavior
• Authors: Morris, Finnegan, Wu (2005)
Research Questions
• What are the differences and similarities between completers and withdrawers in various measures of student behavior online?
• How accurately can achievement be predicted from student participation measures in online learning courses?
Variables (n=8)
Frequency and Duration of– viewing course content– viewing discussions– creating new discussion posts– responding to discussion posts
• Over 400 students and 13 sections of 3 courses
Frequency of Learning Activities
0
100
200
300
400
500
Average Over Term
Withdrawer Non-Successful
Successful
English Geology History
Content Pages Viewed
0200400600800
1000120014001600
Average Over Term
Withdrawer Non-Successful
Successful
English Geology History
Discussion Posts Viewed
Frequency of Learning Activities
0
10
20
30
40
50
Average Over Term
Withdrawer Non-Successful
Successful
English Geology History
Original Posts Created
0
20
40
60
80
100
Average Over Term
Withdrawer Non-Successful
Successful
English Geology History
Follow-up Posts Created
Duration of Learning Activities
N=423
Total Time Spent During Term
Viewing Content
Viewing Discussions
Creating Original Posts
Creating
Follow-up Posts
Average Overall Time Per Week
Withdrawers
n=13710 hours 2.6
hours3 hours <1 hour <1 hour <1 hour
Non-Successful Completers
n=72
18 hours 9 hours 6 hours <1 hour <1 hour 1.2 hours
Successful Completers
n=214
54 hours 19 hours 19 hours 1 hour 1.5 hours 3.75 hours
Findings: Completers & Withdrawers
• Completers had more frequent activity and spent more time on task on all 4 measures than unsuccessful completers and withdrawers.
• Withdrawers spent significantly less time and had less frequent activity than completers on all 4 measures (p>.001).
Expected.
• Significant differences in participation also existed between successful and unsuccessful completers.
Multiple Regression Model for Impact of Participation on
Achievement
Successful and Non-Successful Completersn = 286
Findings: Successful and Unsuccessful Completers• The participation model explained 31% of
the variability in achievement.
• 3 of 8 variables were significant at the p.<.05 level and good predictors of successful completion (achievement/grades).– # of content pages viewed – # of discussion posts viewed – Seconds viewing discussions
Summary: Study 2
• Time-on-task matters; withdrawers did engage significantly in number or duration of activities at the online site.
• Successful completers engaged significantly with the online course:– Going repeatedly to content pages
(frequency)
– Going repeatedly to discussion posts (frequency)
– Spending significant time reading discussion posts (duration)
Study 3: Understanding Student Persistence and Satisfaction
• Purpose: To investigate issues that affect course completion, course withdrawals and satisfaction with online courses.
• Survey (n=505, response 22%)
• Indepth Interviews– 8 withdrawers
– 8 completers
• Authors: Boop, Morris, Finnegan (2005)
Successful completers
• Felt “membership” in the course.
• Understood course layout, expectations, assignments.
• Faculty feedback was important.
• Clarity about course was important.
• Used words indicating “drive” and “persistence” to succeed. Could overcome course-related” problems.
Withdrawers/Unsuccessful Students
• Spoke of being “lost” & “confused” in the course.
• Needed more direction & help from faculty to understand the course goals, expectations, assignments & design.
• Needed more explicit help with discussions and understanding involvement.
• Needed more managerial and navigational help.
Study 4: Perspectives and Activities of Faculty Teaching Online
• Purpose: To explore the activities and perspectives of faculty teaching online
• Interviews (n=13)
• Analysis of archived courses (10)
• Authors: Morris, Xu, Finnegan (2005)
Classification of Faculty Roles
Classification of Faculty Roles (N=10)
0
100
200
300
400
500
N N N N E E E E) E E
Number of Postings
Pedagogical Social Managerial
Summary: Study 4• Novice instructors are far less engaged
with students online.• Experienced faculty posted with a ratio of
1:6 --faculty to student posts• Experienced faculty interchanged
pedagogical, managerial, and social roles online
• Students in courses with experienced faculty engaged more often in discussions
• Faculty visibility is important to student participation.
• Novice faculty need extensive assistance to understand online instruction.
Best Practices:Students
• Students should be advised that for online courses– Time on task matters for successful
achievement;– Online courses may be activity and time
intensive; – requires pro-active, engaged students;– Will not be easier for academically marginal
students;– Students should directly (and as needed) seek
instructor help to understand course structure and course-related objects and objectives
Best Practices: Faculty 1
• Faculty should – Understand Low participation early in the
term as an indicator for withdrawal or unsuccessful completion.
– Should monitor/track all students early in the course term to see lags in participation
– Understand the role of student expectations & attitudes in persistence
– Should understand the role of Locus of Control in Withdrawing and Unsuccessful completion
Best Practices: Faculty 2
– Should engage managerial functions to explain course layout, assignments expectations (may be more important than pedagogical function at times)
– Understand that course layout and instructions are not necessarily intuitive to the students
– Should seek to understand previous academic preparation of students and make adjustments accordingly
Comparing Student Performance to Programmatic Learning Outcomes
• Link graded activities within courses to eCore® common student learning outcomes
• Determine achievement of learning outcomes based on trends in grades
• Identify additional means of documenting student achievement of learning outcomes
Benefits of CMS Data
• New quantitative evidence– Complements survey, grades, and portfolio
data– Very detailed information about
engagement and learning process
• Reduce burden on faculty and staff– Automatically collects evidence – Leverages tools already in use
Opportunities for Studies• Increase awareness of data
sources available to study pedagogy and outcomes
• Encourage systematic analysis of existing data for pedagogical improvement
• Identify additional data elements within CMS and other data sources
Challenges for Studies
• Use of CMS not widespread nor extensive
• Essential tools not used (i.e., gradebook)
• Siloed data sources (Green’s ERP Turtle)
Conclusions
• Data collected in CMS and other systems can be used to inform the scholarship of teaching– Systematic and ongoing
• New sources of data offer opportunities to study perennial questions from different perspectives.
Thank You!
Catherine Finnegan
Presentations and Citations Available at: http://alt.usg.edu