The Advantages of Monitoring Sakai Usage

By

Jose L. Silva-Lugo Carl J. Perry

John R. Boekenoogen e-Learning Support Services

Office of Academic Technology University of Florida

Introduction

6/10/2012

2

The University of Florida (UF)’s contract with Blackboard WebCT was going to end in October 2010.

In Spring of 2009, UF came up with four options: Sakai, Blackboard, Moodle, and Angel.

UF ran open meetings with each vender.

UF set up test sites for faculty, staff, and students.

UF selected Sakai

Introduction: Planning, Testing, & Implementation

6/10/2012

3

2009 2010 2011 2012

Process Dec Spr Sum Fall Spr Sum Fall Spr

Planning

Pilot Study

Migration

Introd. to UF

Roll Out

Workshops

Open Labs

Webinars

Introduction: Training & Assistant Offer to UF

6/10/2012

4

• Getting Started • Moving On • Evaluating knowledge with online assessment • Begin a new semester • End a semester

Workshops

• 1 h consultation per week • There is no need of appointment • Every Tuesdays through the semester

Open labs

• 30 minutes per week • Every Wednesday through the semester Webinar

Introduction: The Need of Monitoring Sakai Usage

6/10/2012

5

Even though the effort of promoting Sakai carried out from summer 2010 to 2011, we did not have the expected attendance.

We decided to look our records to: Find out the percentage of courses, sections, and instructors

using Sakai per college/year Identify the colleges with percentages less than 50% used of

Sakai Plot the percentage of courses, sections, and instructors per

semester Target colleges with a marketing program

Methodology: Databases

6/10/2012

6

Data was retrieved from the network:

Registrar Office: IBM DB2 database

Middleware: MySQL database

Language used to query the database:

A PHP script queried both database (< 5 seconds).

The same script merged and compared the results (5-20 seconds).

Methods: Points of Interest about Querying

6/10/2012

7

You need to understand the database. You have to find how to best compare the results:

The Registrar DB2 had zero to many instructors associated with a section

The Registrar DB2 used postfix characters in the course name, such as for labs

The results produced from each database were accurate and error free.

Manipulating large data sets and verifying the results were accurate.

Methodology: Calculating Percentages

6/10/2012

8

Registrar (Total)

• Courses • Sections • Instructors

Middleware (Sakai)

• Courses • Sections • Instructors

% of Courses =No. of Courses Middleware

No. of Courses Registrar Xx 100

Methodology: Statistics

6/10/2012

9

Statistical Analysis:

Proportion comparison:

Between 2010 and 2011 for courses, sections, and instructors per college

Between semesters for courses, sections, and instructors

Alpha level = 0.05

Two-tailed test

No. of Courses from the Registrar Office and Middleware in 2010 & 2011

6/10/2012

10 2010 2011

College Registrar Middleware Percentage Registrar Middleware Percentage DESIGN, CONSTRUCTION AND PLANNING 307 91 29.64 348 256 73.56 HEALTH AND HUMAN PERFORMANCE 383 91 23.76 423 282 66.67 ENGINEERING 1,100 218 19.82 1,127 445 39.49 JOURNALISM AND COMMUNICATIONS 311 57 18.33 289 191 66.09 MEDICINE 534 92 17.23 519 116 22.35 PHARMACY 244 42 17.21 296 150 50.68 LIBERAL ARTS AND SCIENCES 3,138 455 14.50 2,872 1,237 43.07 PUBLIC HEALTH & HEALTH PROFESSIONS 457 65 14.22 458 245 53.49 BUSINESS ADMINISTRATION 645 88 13.64 653 406 62.17 AGRICULTURAL AND LIFE SCIENCES 1,602 213 13.30 1,647 573 34.79 CROSS-COLLEGE 36 4 11.11 47 0 0.00 FINE ARTS 1,022 56 5.48 1,037 179 17.26 VETERINARY MEDICINE 259 13 5.02 264 51 19.32 EDUCATION 630 18 2.86 662 54 8.16 DENTISTRY 244 1 0.41 279 5 1.79 ACCOUNTING 64 0 0.00 65 0 0.00 BUILDING CONSTRUCTION 211 0 0.00 218 0 0.00 CONTINUING EDUCATION 209 0 0.00 171 0 0.00 LEVIN COLLEGE OF LAW 190 0 0.00 550 11 2.00 MILITARY 62 0 0.00 55 0 0.00

NURSING 133 0 0.00 130 88 67.69 Total 11,781 1,504 12.77 12,110 4,289 35.42

6/10/2012

11

Courses: Proportion Comparison between 2010 and 2011 per College

6/10/2012

12

Semester π1 π2 Π σ π1- π2 |Z| Zα P-value Result

AGRICULTURAL 0.13 0.35 0.2419 0.0150 14.3034 1.645 < 0.0001 P < 0.05

BUSINESS ADMINISTRATION 0.14 0.62 0.3806 0.0270 18.0054 1.645 < 0.0001 P < 0.05

DENTISTRY 0.00 0.02 0.0115 0.0093 1.4809 1.645 0.0694 P > 0.05

DESIGN, CONSTRUCTION 0.30 0.74 0.5298 0.0391 11.2387 1.645 < 0.0001 P < 0.05

EDUCATION 0.03 0.08 0.0557 0.0128 4.1510 1.645 < 0.0001 P < 0.05

ENGINEERING 0.20 0.39 0.2977 0.0194 10.1481 1.645 < 0.0001 P < 0.05

FINE ARTS 0.05 0.17 0.1141 0.0140 8.4064 1.645 < 0.0001 P < 0.05

HEALTH & HUMAN PERFORMANCE 0.24 0.67 0.4628 0.0352 12.2002 1.645 < 0.0001 P < 0.05

JOURNALISM & COMMUNICATIONS 0.18 0.66 0.4133 0.0402 11.8711 1.645 < 0.0001 P < 0.05

LEVIN COLLEGE OF LAW 0.00 0.02 0.0149 0.0102 1.9640 1.645 0.0250 P < 0.05

LIBERAL ARTS & SCIENCES 0.14 0.43 0.2787 0.0115 25.0227 1.645 < 0.0001 P < 0.05

MEDICINE 0.17 0.22 0.1975 0.0245 2.0872 1.645 0.0188 P < 0.05

NURSING 0.00 0.68 0.3346 0.0582 11.6320 1.645 < 0.0001 P < 0.05

PHARMACY 0.17 0.51 0.3556 0.0414 8.0846 1.645 < 0.0001 P < 0.05

PUBLIC HEALTH 0.14 0.53 0.3388 0.0313 12.5489 1.645 < 0.0001 P < 0.05

VETERINARY MEDICINE 0.05 0.19 0.1224 0.0287 4.9889 1.645 < 0.0001 P < 0.05

2010 0.13 0.50 0.3142 0.0060 61.6931 1.645 < 0.0001 P < 0.05

2011 0.35 0.50 0.4248 0.0064 22.2413 1.645 < 0.0001 P < 0.05

6/10/2012

13

6/10/2012

14

Courses: Proportion Comparison between Semesters

6/10/2012

15

Semester π1 π2 Π σ π1- π2 |Z| Zα P-value Result

Spring 2010 - Summer 2010 0.00 0.04 0.0166 0.0031 13.4318 1.645 < 0.0001 P < 0.05

Summer 2010 - Summer 2011 0.04 0.29 0.1772 0.0102 24.5078 1.645 < 0.0001 P < 0.05

Fall 2010 - Spring 2011 0.30 0.37 0.3344 0.0099 6.4253 1.645 < 0.0001 P < 0.05

Fall 2010 - Summer 2011 0.30 0.29 0.2985 0.0107 1.0054 1.645 0.1562 P > 0.05

Fall 2010 - Fall 2011 0.30 0.38 0.3432 0.0099 8.1293 1.645 < 0.0001 P < 0.05

Spring 2011 - Summer 2011 0.37 0.29 0.3363 0.0111 6.7244 1.645 < 0.0001 P < 0.05

Spring 2011 - Fall 2011 0.37 0.38 0.3752 0.0102 1.6609 1.645 0.0485 P < 0.05

Summer 2011 - Fall 2011 0.29 0.38 0.3469 0.0111 8.2315 1.645 < 0.0001 P < 0.05

6/10/2012

16

No. of Sections from the Registrar Office and Middleware in 2010 & 2011

6/10/2012

17 2010 2011

College Registrar Middleware Percentage Registrar Middleware Percentage

HEALTH AND HUMAN PERFORMANCE 628 169 26.91 894 563 62.98

MEDICINE 604 134 22.19 682 165 24.19

DESIGN, CONSTRUCTION AND PLANNING 577 125 21.66 582 341 58.59 ENGINEERING 2467 500 20.27 2384 784 32.89

PHARMACY 718 131 18.25 735 426 57.96

LIBERAL ARTS AND SCIENCES 7653 1308 17.09 7632 3649 47.81

CROSS-COLLEGE 66 11 16.67 71 0 0.00

JOURNALISM AND COMMUNICATIONS 588 98 16.67 638 327 51.25

BUSINESS ADMINISTRATION 1264 185 14.64 1306 915 70.06

AGRICULTURAL AND LIFE SCIENCES 3220 457 14.19 3870 1144 29.56 PUBLIC HEALTH & HEALTH PROFESSIONS 811 90 11.10 919 347 37.76

FINE ARTS 1843 87 4.72 1975 273 13.82

VETERINARY MEDICINE 325 14 4.31 370 63 17.03

EDUCATION 1299 22 1.69 1389 78 5.62

DENTISTRY 356 1 0.28 372 4 1.08

ACCOUNTING 136 0 0.00 116 0 0.00

BUILDING CONSTRUCTION 269 0 0.00 277 0 0.00 CONTINUING EDUCATION 228 0 0.00 177 0 0.00

LEVIN COLLEGE OF LAW 441 0 0.00 503 11 2.19

MILITARY 90 0 0.00 79 0 0.00

NURSING 279 0 0.00 279 208 74.55

Total 23,862 3,332 13.96 25,250 9,298 36.82

6/10/2012

18

Sections: Proportion Comparison between 2010 and 2011 per College

6/10/2012

19

Semester π1 π2 Π σ π1- π2 |Z| Zα P-value Result

AGRICULTURAL 0.14 0.30 0.2258 0.0100 15.4094 1.645 < 0.0001 P < 0.05

BUSINESS ADMINISTRATION 0.15 0.70 0.4280 0.0195 28.3899 1.645 < 0.0001 P < 0.05

DENTISTRY 0.00 0.01 0.0069 0.0061 1.2973 1.645 0.0968 P > 0.05

DESIGN, CONSTRUCTION & PLANNING 0.22 0.59 0.4021 0.0288 12.8197 1.645 < 0.0001 P < 0.05

EDUCATION 0.02 0.06 0.0372 0.0073 5.3689 1.645 < 0.0001 P < 0.05

ENGINEERING 0.20 0.33 0.2647 0.0127 9.9592 1.645 < 0.0001 P < 0.05

FINE ARTS 0.05 0.14 0.0943 0.0095 9.6172 1.645 < 0.0001 P < 0.05

HEALTH & HUMAN PERFORMANCE 0.27 0.63 0.4809 0.0260 13.8633 1.645 < 0.0001 P < 0.05

JOURNALISM & COMMUNICATIONS 0.17 0.51 0.3467 0.0272 12.7130 1.645 < 0.0001 P < 0.05

LEVIN COLLEGE OF LAW 0.00 0.02 0.0117 0.0070 3.1238 1.645 < 0.001 P < 0.05

LIBERAL ARTS & SCIENCES 0.17 0.48 0.3243 0.0076 40.5675 1.645 < 0.0001 P < 0.05

MEDICINE 0.22 0.24 0.2325 0.0236 0.8508 1.645 0.1917 P > 0.05

NURSING 0.00 0.75 0.3728 0.0409 18.2102 1.645 < 0.0001 P < 0.05

PHARMACY 0.18 0.58 0.3833 0.0255 15.5669 1.645 < 0.0001 P < 0.05

PUBLIC HEALTH & HEALH PROFESSIONS 0.11 0.38 0.2526 0.0209 12.7359 1.645 < 0.0001 P < 0.05

VETERINARY MEDICINE 0.04 0.17 0.1108 0.0239 5.3304 1.645 < 0.0001 P < 0.05

2010 0.14 0.50 0.3213 0.0043 84.9707 1.645 < 0.0001 P < 0.05

2011 0.37 0.50 0.4316 0.0044 28.7675 1.645 < 0.0001 P < 0.05

6/10/2012

20

6/10/2012

21

Sections: Proportion Comparison between Semesters

6/10/2012

22

Semester π1 π2 Π σ π1- π2 |Z| Zα P-value Result

Spring 2010 - Summer 2010 0.00 0.05 0.0163 0.0023 21.5659 1.645 < 0.0001 P < 0.05

Summer 2010 - Summer 2011 0.05 0.24 0.1576 0.0074 25.3576 1.645 < 0.0001 P < 0.05

Fall 2010 - Spring 2011 0.31 0.40 0.3528 0.0068 12.3286 1.645 < 0.0001 P < 0.05

Fall 2010 - Summer 2011 0.31 0.24 0.2850 0.0075 9.7743 1.645 < 0.0001 P < 0.05

Fall 2010 - Fall 2011 0.31 0.42 0.3636 0.0068 15.2238 1.645 < 0.0001 P < 0.05

Spring 2011 - Summer 2011 0.40 0.24 0.3371 0.0079 19.8939 1.645 < 0.0001 P < 0.05

Spring 2011 - Fall 2011 0.40 0.42 0.4058 0.0070 2.7055 1.645 0.0034 P < 0.05

Summer 2011 - Fall 2011 0.24 0.42 0.3514 0.0079 22.3055 1.645 < 0.0001 P < 0.05

No. of Instructors from the Registrar Office and Middleware in 2010 & 2011

6/10/2012

23 2010 2011

College Registrar Middleware Percentage Registrar Middleware Percentage CROSS-COLLEGE 36 10 27.78 37 0 0.00 HEALTH AND HUMAN PERFORMANCE 220 56 25.45 267 175 65.54 ENGINEERING 697 176 25.25 758 417 55.01 DESIGN, CONSTRUCTION AND PLANNING 218 46 21.10 236 129 54.66 BUSINESS ADMINISTRATION 405 80 19.75 414 124 29.95 JOURNALISM AND COMMUNICATIONS 286 56 19.58 318 180 56.60 LIBERAL ARTS AND SCIENCES 3,364 519 15.43 3,843 1,289 33.54 PUBLIC HEALTH & HEALTH PROFESSIONS 403 61 15.14 444 162 36.49 AGRICULTURAL AND LIFE SCIENCES 1,003 147 14.66 1,050 388 36.95 FINE ARTS 621 50 8.05 659 141 21.40 VETERINARY MEDICINE 256 14 5.47 489 32 6.54

MEDICINE 324 16 4.94 541 49 9.06 EDUCATION 583 21 3.60 581 67 11.53 PHARMACY 164 3 1.83 236 8 3.39

DENTISTRY 132 1 0.76 571 4 0.70 ACCOUNTING 40 0 0.00 45 0 0.00 BUILDING CONSTRUCTION 67 0 0.00 82 0 0.00

CONTINUING EDUCATION 12 0 0.00 5 0 0.00 LEVIN COLLEGE OF LAW 256 0 0.00 297 9 3.03 MILITARY 28 0 0.00 31 0 0.00 NURSING 174 0 0.00 205 6 2.93

Total 9,289 1,256 13.52 11,109 3,180 28.63

6/10/2012

24

Instructors: Proportion Comparison between 2010 and 2011 per College

6/10/2012

25

Semester π1 π2 Π σ π1- π2 |Z| Zα P-value Result

AGRICULTURAL 0.15 0.37 0.2606 0.0194 11.5043 1.645 < 0.0001 P < 0.05

BUSINESS ADMINISTRATION 0.20 0.30 0.2491 0.0302 3.3741 1.645 < 0.001 P < 0.05

DENTISTRY 0.01 0.01 0.0071 0.0081 0.0703 1.645 0.4721 P > 0.05

DESIGN, CONSTRUCTION 0.21 0.55 0.3855 0.0457 7.3403 1.645 < 0.0001 P < 0.05

EDUCATION 0.04 0.12 0.0756 0.0155 5.1170 1.645 < 0.0001 P < 0.05

ENGINEERING 0.25 0.55 0.4076 0.0258 11.5416 1.645 < 0.0001 P < 0.05

FINE ARTS 0.08 0.21 0.1492 0.0199 6.6968 1.645 < 0.0001 P < 0.05

HEALTH & HUMAN PERFORMANCE 0.25 0.66 0.4743 0.0455 8.8171 1.645 < 0.0001 P < 0.05

JOURNALISM & COMMUNICATIONS 0.20 0.57 0.3907 0.0398 9.3113 1.645 < 0.0001 P < 0.05

LEVIN COLLEGE OF LAW 0.00 0.03 0.0163 0.0108 2.8082 1.645 0.0025 P < 0.05

LIBERAL ARTS & SCIENCES 0.15 0.34 0.2509 0.0102 17.6964 1.645 < 0.0001 P < 0.05

MEDICINE 0.05 0.09 0.0751 0.0185 2.2242 1.645 0.0132 P < 0.05

NURSING 0.00 0.03 0.0158 0.0129 2.2748 1.645 0.0116 P < 0.05

PHARMACY 0.02 0.03 0.0275 0.0166 0.9387 1.645 0.1736 P > 0.05

PUBLIC HEALTH 0.15 0.36 0.2633 0.0303 7.0459 1.645 < 0.0001 P < 0.05

VETERINARY MEDICINE 0.05 0.07 0.0617 0.0186 0.5791 1.645 0.2810 P > 0.05

2010 0.14 0.50 0.3152 0.0068 52.8065 1.645 < 0.0001 P < 0.05

2011 0.50 0.29 0.3907 0.0065 31.9009 1.645 < 0.0001 P < 0.05

6/10/2012

26

6/10/2012

27

Instructors: Proportion Comparison between Semesters

6/10/2012

28

Semester π1 π2 Π σ π1- π2 |Z| Zα P-value Result

Spring 2010 - Summer 2010 0.00 0.07 0.0235 0.0044 14.5203 1.645 < 0.0001 P < 0.05

Summer 2010 - Summer 2011 0.07 0.23 0.1582 0.0113 14.3211 1.645 < 0.0001 P < 0.05

Fall 2010 - Spring 2011 0.29 0.32 0.3065 0.0103 3.5117 1.645 < 0.001 P < 0.05

Fall 2010 - Summer 2011 0.29 0.23 0.2645 0.0114 5.3643 1.645 < 0.0001 P < 0.05

Fall 2010 - Fall 2011 0.29 0.28 0.2856 0.0099 0.4479 1.645 0.3264 P > 0.05

Spring 2011 - Summer 2011 0.32 0.23 0.2881 0.0115 8.4099 1.645 < 0.0001 P < 0.05

Spring 2011 - Fall 2011 0.32 0.28 0.3030 0.0099 4.0899 1.645 < 0.0001 P < 0.05

Summer 2011 - Fall 2011 0.23 0.28 0.2636 0.0111 5.1051 1.645 < 0.0001 P < 0.05

6/10/2012

29

Discussion

6/10/2012

30

In general, course, sections, and instructors using Sakai are not monitored.

This simple procedure has the following advantages: Provide information about the proportion of

instructors, courses, and sections using Sakai.

It helps to monitor the implementation process.

It creates the database for marketing programs.

It might be used to evaluate success at the college and at university level.

Conclusions

6/10/2012

31

There was significant increase in their proportions of courses, sections, and instructors from 2010 to

2011 in the majority of the colleges.

There was significant increase in their proportions of courses, sections, and instructors per semester.

We have identified the colleges that need more our attention, and we are targeting them with a

marketing program.

Monitoring Sakai usage is important during the implementation and evaluation of the overall

success at the college and university level.