the indicators project - ascilite version

The Indicators ProjectIdentifying effective learning: adoption, activity, grades and external factors

Colin BeerDavid JonesKen Clark

http://tr.im/FvRQ

Twitter: #eair

http://www.flickr.com/photos/anders-vindegg/3408838186/

Surprising patterns in CQU LMS data

Apparent benefit in LMS usage analysis that is

Different from established patterns

Due to limitations in current work?

Cross-LMS

Cross-institutional

Longitudinal

Want to play?

Overview

http://www.flickr.com/photos/atbaker/1577665185/

Context

Level of feature adoption

Link between participation & grades

Reflection and Future work

Overview

Context

http://www.flickr.com/photos/dnorman/251646154/

The data that we, educators gatherand utilize is all but garbage.(Wiley,

http://www.flickr.com/photos/mikebaird/2087879492/

(Dawson, 2009; Dawson et al, 2009)

(Bakharia et al, 2009)

(Malikowski, 2008; Malikowski, et al, 2007;Malikowski et al, 2006)

(Campbell et al, 2007)

(Griffiths, 2007)

(Dawson, 2006)

(Black et al, 2008)

(Dawson et al, 2008)

(Heathcoate et al, 2005)

(Hornik et al, 2008)

(Romero et al, 2007)

http://www.flickr.com/photos/mwichary/2188958154/

LMS reporting toolsOnly what is in the

Limited views

Some surveysHuman

recollectionLMS reporting toolsOnly what is in the

Limited views

Some surveysHuman

recollection

Manual checksTime-

consuming

Error prone

Limited views

Some surveysHuman

recollection

Limited longitudinal data

Manual checksTime-

consuming

Error prone

Limited views

Some surveysHuman

recollection

Limited cross-LMS comparisons

Manual checksTime-

consuming

Error prone

Limited views

Some surveysHuman

recollection

Limited cross-LMS comparisons

Limited cross-institutional comparisons

Manual checksTime-

consuming

Error prone

Limited views

“LMS” situation

Students

Webfuse 1997-2009

Pre-2010 “LMS”

WebCT 1999-2004

Blackboard 2004-2009

Webfuse 1997-2009

Post-2010 “LMS”

Blackboard 2004-2009

Moodle….

Students - 2009

CQ on-campus

Students - 2009

CQ on-campus

Distance education

Students - 2009

CQ on-campus

Distance education

AIC on-campus

8444 796

http://www.flickr.com/photos/dnorman/251646154/

The data that we, educators gatherand utilize is all but garbage.(Wiley,

http://indicatorsproject.wordpress.com

Enabling comparisons of LMS usageacross institutions, platforms and

http://www.flickr.com/photos/david_jones/36379215/

Overview

Context

http://www.flickr.com/photos/chokingsun/3495110670/

There are more similarities than differences among LMS (Black et al,

TransmittingContent

CreatingClass

Interactions

ComputerBased

Instruction

Move to features in unexplored categories, until each category is considered for different learning needs

Level 1: Most used Level 2: Moderately used Level 3: Rarely used

• Content/files• Announcements• Gradebook

More than 50% of courses More than 20% of courses (much) less than 20% of courses

EvaluatingStudents• Quizzes• Drop box

Evaluatingcourse and instructors

• Surveys• Chat• Discussion forum• Email

• Some quizzes• Adaptive release

(Malikowski et al, 2007)

What we did

Wf BbLMS logs

LMS Independent

What we did

Wf BbLMS logs

LMS Independent

What we did

% of courses adopting “Malikowski features”

Year Blackboard Webfuse2005 777 3042006 810 2962007 874 2522008 945 2252009 868 174

# course sites per “LMS”

Malikowskitop & bottomrange

Blackboard

Webfuse

http://www.flickr.com/photos/wadem/2730257498/

The particular trajectory of emergence is not wholly

determined either by the intentions of the human actors or by the material properties of technology, but rather by the interplay of the two(Jones,

http://www.flickr.com/photos/nedrichards/3234490934/

Future Work

Will Moodle be different?

What about other institutions?

Are Malikowski ranges out of date?

When is a feature adopted?

…….

Overview

Context

http://www.flickr.com/photos/wolfgangstaudt/2279651479/

The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course (Dawson et al,

Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance. (Dawson, 2009; Morris et al,

http://www.flickr.com/photos/jamuraa/813966437/

What about

Different student types?

Different levels of staff interaction?Different staff academic background?

(Fresen, 2007)

CQU context

Wf BbLMS logs

LMS Independent

What we did

Feature usage

Wf BbLMS logs

LMS Independent

What we did

Feature usage

Student records

Wf BbLMS logs

LMS Independent

What we did

Feature usage

Participation, grades,external factors

Student records

Participation

Discussion forum

Entire course siteJust course forum

# replies# posts

http://www.flickr.com/photos/adactio/1259245482/

N = 510,158

Student/courses

Mixed design ANOVA

All reported effects are significant

Significant interaction effectbetween grades and participation

Hits on course site

Hits on course forum

Distance education students

CQ on-campus students

AIC students

Distance education students

# of replies

# of posts

CQ on-campus students

AIC students

What about

(Fresen, 2007)

CQU context

http://www.flickr.com/photos/orinrobertjohn/3321067724/

Course groups based on staff hits

Group Courses Student-courses

Students/course

Definition

High 678(17%)

156,770 231 x>3000 hits

Medium 980(25%)

143,314 146 3000>x>1000

Low 1474(37%)

102,214 69 1000>x>100

Super low 849(21%)

68,354 80 100>x>0

High staff participation courses

Replies & posts

High staff participation courses

Super low staff participation courses

Replies & posts

Super low staff participation courses

Posts comparisons

Superlow

http://www.flickr.com/photos/nedrichards/3234490934/

Future Work

- Course size

Examination of super-low

Networks of interaction (Bakharia et al, 2009; Dawson,

Forum content

What about

(Fresen, 2007)

CQU context

Course offerings with

Staff with grad certs in L&TSignificant input from instructional designerStaff who have received teaching awards

Staff teaching education coursesNone of the above

n=3288

None of the above

Grad cert in L&T

Teaching awards

Education discipline

Instructional design

Overview

Context

Refleection and Future work

http://www.flickr.com/photos/mikebaird/2985066755/

Some interesting patterns

Useful idea

Needs more work

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Useful Data/Informat

T&L Practice Research

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Enabling

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Enabling

Avg course hits

DE under grad students

By age

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Enabling

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Enabling

WhyHow

http://www.flickr.com/photos/dmatos/540688952/

http://www.flickr.com/photos/sylvar/1138341328/

http://www.flickr.com/photos/fncll/145149313/

http://tr.im/FvRQ

http://indicatorsproject.wordpress.com

Bakharia, A., E. Heathcote, et al. (2009). Social networks adapting pedagogical practice: SNAPP. Same places, different spaces. Proceedings ascilite Auckland 2009. Auckland: 49-51.

Black, E. W., K. Dawson, et al. (2008). "Data for free: Using LMS activity logs to measure community in online courses." Internet and Higher Education 11(11): 65-70.

Black, E., D. Beck, et al. (2007). "The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments." Tech Trends 51(2): 35-39.

Campbell, J., P. DeBlois, et al. (2007). "Academic analytics: A new tool for a new era." EDCAUSE Review 42(4): 40-42.

Coates, H., R. James, et al. (2005). "A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning." Tertiary Education and Management 11(1): 19-36.

Dawson, S. (2009). "'Seeing' the learning community: An exploration of the development of a resource for monitoring online student networking." British Journal of Educational Technology: in press.

Dawson, S., L. Macfadyen, et al. (2009). Learning or performance: Predicting drivers of student motivation. Same places, different spaces. Proceedings ascilite Auckland 2009, Auckland.

Dawson, S., E. McWilliam, et al. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008. Melbourne: 221-230

Dawson, S. (2006). "Online forum discussion interactions as an indicator of student community." Australian Journal of Educational Technology 22(4): 495-510.

Fresen, J. (2007). "A taxonomy of factors to promote quality web-supported learning." International Journal on E-Learning 6(3): 351-362.

Griffiths, M. E. (2007). Patterns of user acitivity in the Blackboard course management system across all courses in the 2004-2005 academic year at Brigham Young University. Department of Instructional Psychology and Technology., Brigham Young. Master of Science: 117.

Heathcoate, L. and S. Dawson (2005). "Data Mining for Evaluation, Benchmarking and Reflective Practice in a LMS." E-Learn 2005: World conference on E-Learning in corporate, government, healthcare and higher education.

Hornik, S., C. S. Saunders, et al. (2008). "The Impact of Paradigm Development and Course Level on Performance in Technology-Mediated Learning Environments." Informing Science. The international journal of an emerging transdiscipline 11(11).

Jones, M. (1999). Information systems and the double mangle: Steering a course between the scylla of embedded structure and the charybdis of strong symmetry. Information Systems: Current Issues and Future Chalenges. T. Larsen, L. Levine and J. DeGross. Laxenburg, Austria, IFIP: 287-302.

Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research 36(2): 149-173.

Malikowski, S., M. Thompson, et al. (2006). "External factors associated with adopting a CMS in resident college courses." Internet and Higher Education 9(3): 163-174.

Morris, L., C. Finnega, et al. (2005). "Tracking student behavior, persistence, and achievement in online courses." Internet and Higher Education 8(3): 221-231.

Romero, C. b., S. n. Ventura, et al. (2007). "Data mining in course management systems: Moodle case study and tutorial." ScienceDirect Computers and Education (51 (2008)): 368-384.

Salmon, G. (2005). "Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions." ALT-J, Research in Learning Technology 13(3): 201-218.

Wiley, D. (2009). "The LHC and education." Retrieved 29 November, 2009, from http://opencontent.org/blog/archives/1098.

the indicators project - ascilite version

Education

patient safety indicators: software documentation, version...

ascilite presentation

conole ascilite paper

ascilite presentation december 2013

water quality indicators (wqi) tool version 2

bprofcomm ascilite 2009

reviewing learning designs with heart - ascilite 2009

key performance indicators of rail freight corridors version...

indicators of economic development - full version

ascilite workshop web 2 assessment slideshare

ascilite 2018 conference program...

pediatric quality indicators software instruction,...

ascilite 2014 conference handbook

ascilite presentation final 19 nov

ahrq quality indicators software instructions, … ·...

guidelines for full-papers to be published in the ascilite

conole keynote ascilite 2009 conference presentation

ascilite live! webinar a multidisciplinary approach to...

evaluation of technology integration in teaching statistics...

template - ascilite 2016 | ascilite 2016 conference... ·...