under the hood: using the data generated by sakai

John Leasia, R.P. (Adi) Aditya, Steven Lonn, David Roldan Mar;nez

University of Michigan & Universidade Fernando Pessoa

Under the Hood: Using the Data

Generated by Sakai

11th Sakai Conference -‐ June 15-‐17, 2010

Overview

• Introduc;on to Sakai Data• John Leasia

• Using Sakai Data to Monitor User Ac;vity• Ramaswamy Aditya

• End-‐User Support, Ins;tu;onal Research, etc.• Steven Lonn

• End-‐User Applica;ons Using Sakai Data: SiteStats• David Roldan Mar;nez

2


Introduc<on to Sakai Data

• Why measure Up;me, load, predict support needs, overall usage growth, research

• What Concurrent users, site crea;ons, ave max users, comparisons

• HowCharts, Database queries (SQL), Sakai tools

3


• Admin Online tool

• SiteStats

4


• Tool events lis;ng at

h\p://confluence.sakaiproject.org/display/UDAT/Log+Events+Descrip;ons+-‐+Sakai+2.6.x

• Real;me (almost) charts – later in presenta;onBased on events and app server logs

• Queries on the DatabaseCreate graphs, stats from accumulated data

5


• Event tables

6

sakai_session SESSION_ID SESSION_USER SESSION_IP SESSION_USER_AGENT SESSION_START SESSION_END SESSION_SERVER SESSION_ACTIVE SESSION_HOSTNAME

sakai_event EVENT_ID EVENT_DATE EVENT REF SESSION_ID EVENT_CODE CONTEXT

REF changes depending on event type content.delete /content/group/<site_id>/<file_name> asn.submit.submission /assignment/s/<site_id>/<assignment_id>/<submission_id> site.upd.site.mbrshp /site/<site_id>

11th Sakai Conference -‐ June 15-‐17, 2010 7

DbVisualizer


• Count of sites by type

• Count site crea;ons in a term

8

select type, count(type)from sakai_sitewhere type in ('project', 'course', 'myworkspace', 'GradToolsStudent')group by type;

TYPE COUNT(TYPE)project 34556course 40753GradToolsStudent 2634myworkspace 159743

select count(site_id) All_coursesfrom SAKAI_SITE_PROPERTY_MVW where name='term'and value like 'WINTER 2010';

select count(site_id) Projectsfrom sakai_site where type = 'project'and createdon betweento_date('20100103 00:00','YYYYMMDD HH24:MI') andto_date('20100425 23:59','YYYYMMDD HH24:MI')

All_courses4419

Projects3118


• Who did something – e.g., deleted a site… you know the site_id (look for site.del with a par;cular site_id)

…you know roughly the ;me (look for all site.del events)

9

select sea.event_date, ssa.session_user, sea.event, sea.ref, sea.session_idfrom sakai_event_archive sea, sakai_session_archive ssawhere sea.event = ‘site.del'and ssa.session_id=sea.session_idand sea.event_date > sysdate -‐ 20and sea.site_id = '42544a1-‐f3fd-‐4540-‐00f2-‐3307c295954'

select ssa.session_user, sea.event_date, sea.event, sea.reffrom sakai_session_archive ssa, sakai_event_archive seawhere ssa.session_id = sea.session_id and sea.event_date betweento_date('20090505 00:00','YYYYMMDD HH24:MI') andto_date('20090505 23:59','YYYYMMDD HH24:MI')and event = 'site.del'


sakai_session SESSION_ID SESSION_USER SESSION_IP SESSION_USER_AGENT SESSION_START SESSION_END SESSION_SERVER SESSION_ACTIVE SESSION_HOSTNAME

sakai_event EVENT_ID EVENT_DATE EVENT REF SESSION_ID EVENT_CODE CONTEXT

select sea.event_date, ssa.session_user, sea.event, sea.ref, sea.session_idfrom sakai_event_archive sea, sakai_session_archive ssawhere sea.event = ‘asn.submit.submission'and ssa.session_id=sea.session_idand sea.event_date > sysdate -‐ 20and sea.site_id = '4228a1-‐f6fd-‐4540-‐00f2-‐337c228954'

select se.event_date, ss.session_user, se.event, se.ref, se.session_idfrom sakai_event se, sakai_session sswhere se.event = ‘asn.submit.submissionand ss.session_id=se.session_idand se.event_date > sysdate -‐ 20and se.ref like ‘/assignment/%/4228a1-‐f6fd-‐4540-‐00f2-‐337c228954/%’

sakai_event_archive EVENT_ID EVENT_DATE EVENT REF SESSION_ID EVENT_CODE CONTEXT SITE_ID


• Crunch ;mes – Course Evalua;ons in par;cular

100K+ responses in 8-‐10 days, during study days < finals Queries to monitor rates of responses, create response count over ;me to verify/guage email reminder affects

11

select to_char(end_;me, 'YYYY-‐MM-‐DD HH24:MI:SS') submit_;mesfrom eval_response er, eval_evalua;on eewhere ee.id=er.evalua;on_wand ee.start_date = to_date('20100409 00:15', 'YYYYMMDD HH24:MI')and ee.due_date = to_date('20100421 23:59', 'YYYYMMDD HH24:MI')order by end_;me

submit_<mes2010-‐04-‐09 01:11:042010-‐04-‐09 01:11:482010-‐04-‐09 01:12:002010-‐04-‐09 01:12:482010-‐04-‐09 01:12:582010-‐04-‐09 01:13:002010-‐04-‐09 01:13:312010-‐04-‐09 01:13:482010-‐04-‐09 01:14:342010-‐04-‐09 01:14:372010-‐04-‐09 01:15:10


• Queries to count site crea;ons and manual checking of max concurrent users per day

13


All tables


Evalua;on tables


R.P. (Adi) AdityaUniversity of Michigan

Under the Hood: Using the Data Generated by Sakai

Using Sakai Data to Monitor User Ac<vity

18


Steven LonnResearch Fellow

University of Michigan


End-‐User Support, Ins<tu<onal Research, & Inves<ga<ng Local Sakai Use

19


Different Purposes, Different Goals

20

End-‐User Support Ins

tuonal Research Invesga

ng Local Sakai Use


End-‐User Support Log Searches

• Accidentally Deleted Site

• "Missing" Assignment Submission

21

So far this term, I have had at least 4-5

students swear that they submitted their assignment, but when they later go back, the Assignment

tool says that they did not submit. They have all claimed to have deleted the confirmation email or say that they never received the

email.

The site is PSYCH 240 020 W10. The student is *****. She says she uploaded a submission entitled "8 - Categorical

Perception.doc" for the assignment "Coglab #8: Categorical Perception - ID" on March 29th

around 4am.


"Missing" Assignment

• Look up all ac;vity the user did on the site in the given ;meframe:

select event_date, eid, event, reffrom sakai_event_archive, sakai_session_archive, sakai_user_id_mapwhere eid = '*****'and site_id = '96011a7a-‐7f8b-‐480e-‐a576-‐66a7fd8b2e05'and event <> all ('pres.begin', 'pres.end', 'user.login', 'user.logout')and event_date >= to_date('20100328 00:00','YYYYMMDD HH24:MI')and event_date < to_date('20100330 00:00','YYYYMMDD HH24:MI')and sakai_session_archive.session_id=sakai_event_archive.session_idand session_user=user_id

22



• Result...

23



24

• Can then confirm that the student uploaded a "legi;mate" file (or even send that file to the instructor)



25



ng Local Sakai Use



26

En

d-‐User Sup

port In

s tu onal Research Invesga

ng Local Sakai Use

Some "stakeholders" interested in log data for "ins3tu3onal research:"Individual instructorsAdministra<ve units

Research groups (e.g., higher educa<on research, academic success, etc.)

Sakai managersOthers...


Language Resource Center Audit

• The Language Resource Center (LRC) links language courses to a library of audio files via Sakai.

• Each term, the LRC wants to know how many students downloaded each resource.

27

I work at the Language Resource Center in

the Modern Languages Building. The LRC provides digital resources to many language classes. As instructors become more

comfortable with setting up their own sites, they are able to upload or transfer LRC resources without adding me to their site. The important part of

this equation is: the LRC needs to have usage numbers for all of our resources used through Sakai, whether we have access to the sites or not. This is important for our annual report (funding), and to

track the activity level of our materials.



• LRC provides a list of courses for a given term

• I then run one query coun;ng total downloads and how many UNIQUE users download them.

28

select count (*) as "Total Downloads", count(DISTINCT eid) as "# of Users", reffrom sakai_event_archive, sakai_session_archive, sakai_user_id_mapwhere event = 'content.read'and event_date >= to_date('20090901 00:00','YYYYMMDD HH24:MI')and event_date < to_date('20100101 00:00','YYYYMMDD HH24:MI')and ref like '%.mp3'and site_id in (LIST OF SITE IDs, separated by commas)and sakai_session_archive.session_id=sakai_event_archive.session_idand session_user=user_idgroup by reforder by ref



• Result...

29



• Result...

30


Who's Crea<ng Project Sites & Why?

• Michigan: Any UM faculty, staff, or student can create project sites.

31

0

375

750

1,125

1,500

January February March April May June July August September October November December

2005 2006 2007 2008 2009 2010

Total Sites:

2005: 3,131

2006: 4,890

2007: 6,359

2008: 8,079

2009: 8,714

2010: 3,251 (so far)


• Ability to ask ques;ons during site crea;on process (SAK-‐12912) completed April 2008.

• Two ques;ons:• In what capacity are you crea;ng this site?

• Staff, Faculty, Student

• The primary use for this project site will be:• Learning, Research, Administra;ve, Personal, Student group/organiza;on, Other

32



• USE Lab team analyzed ques;on answers a|er first school year...

33


select sq.ques;on, sa.answer, sua.answer_string, sua.site_id "SITE ID", zz.;tle "SITE TITLE", yy.eid "USER ID", zz.createdon "CREATION DATE"from ssq_answer sa, ssq_ques;on sq, ssq_sitetype_ques;ons ssq, ssq_user_answer sua, sakai_site zz, sakai_user_id_map yywhere sa.ques;on_id = sq.idand sua.answer_id=sa.idand sua.site_id=zz.site_idand zz.createdby=yy.user_idand zz.createdon >= to_date('20080901 00:00','YYYYMMDD HH24:MI')and zz.createdon < to_date('20090901 00:00','YYYYMMDD HH24:MI')



• Data cleaning....

• Analysis....

34



• Data cleaning....

• Analysis....

35


• Something's fishy...

36


Faculty18%

Staff23%

Students59%

Learning ResearchAdmin-‐istra<ve

PersonalStudent Group / Org.

OtherTOTAL

NUMBER OF SITES

23% 38% 24% 3% 4% 8% 1,403

14% 15% 54% 2% 3% 12% 1,835

29% 19% 2% 5% 38% 7% 4,725

Data Source:7,963 Project Sites Created Between

Sept 2008 -‐ Aug 2009



• Data-‐driven decision to change project site crea;on ques;ons.

• New 2nd ques;on:• The primary use for this project site will be:

• Group project (e.g., course project, wri;ng group, professional development)

• Study group (e.g., exam prep, reading group, skill prac;ce)

• Club / organiza<on (e.g., fraternity/sorority, cohort, student government)

• Research (e.g., scholarly lab group, grant proposal)

• Administra<ve (e.g., commi\ees, campus ini;a;ves, special events) • Personal (e.g., file backup)

• Other

37



• Team generally happy with new ques;ons...

38

Faculty17%

Staff21%Students

62%

Group Project

Study Group

ResearchAdmin-‐istra<ve

PersonalClub / Organiza<on

OtherTOTAL # OF SITES

26% 6% 27% 24% 5% 2% 10% 1,093

13% 3% 13% 54% 4% 2% 11% 1,309

69% 7% 8% 1% 4% 8% 3% 3,864

Data Source:6,266 Project Sites Created Between

Sept 2009 -‐ April 2010


User / Non-‐User Analysis

• For annual survey, wanted to run experiment targe;ng users vs. non-‐users• Also was of interest to unit Deans & Sakai managers

• Ran simple query: Did user login at any ;me during date range?

select session_start, eidfrom sakai_session_archive, sakai_user_id_mapwhere eid = 'xxxxxxxx'and session_start >= to_date('20090901 00:00','YYYYMMDD HH24:MI')and session_user=user_idand rownum <= 1;

39



40

Social WorkArchitectureEngineering

BusinessLS&A

EducationPublic HealthArt & Design

LawNursingMusic

MedicineDentistryPharmacy

0% 25% 50% 75% 100%Users Non-‐Users



41

Medicine

Dentistry

Pharmacy

0% 25% 50% 75% 100%

Users Non-‐Users

Who are the users vs. non-‐users in these three units?



Associate Prof.

Assistant Prof.

Clinical Associate Prof

Clinical Assistant Prof

Professor

Adjunct Clinical Asst Prof.

Adjuct Clinical Instructor

0% 25% 50% 75% 100%Users Non-‐Users

Pharmacy



Clinical Assoc ProfClinical LecturerClinical ProfessorClinical Asst ProfAsst Professor

ProfessorAssoc ProfessorAsst Professor

Adjunct Clin Asst ProfessorAdjunct Clinical LecturerAdjunct Clin Assoc Prof

0% 25% 50% 75% 100%Users Non-‐Users

Dental School



Asst ProfessorAssoc Professor

ProfessorClinical ProfessorClinical Assoc ProfClinical LecturerClinical Asst ProfClinical Instructor

Adjunct Clin Asst ProfessorAdjunct Clinical Instructor

Adjunct Clinical Associate Prof.

0% 25% 50% 75% 100%Users Non-‐Users

Medical School



45

En

d-‐User Sup

port In

s tu onal Research Invesga

ng Local Sakai Use



46

End-‐User Support Instu

onal Research Inve

sga

ng Local Sakai Use


Who's Using the Wiki Tool? How? Why?

• Wiki tool use pre\y small compared to "push-‐out" tools like Resources & Announcements

• Need a quick way to guide further inves;ga;on...

47

select count (a.event), a.event, a.ref, a.site_id, b.;tlefrom sakai_event_archive a, sakai_site bwhere a.event = 'wiki.revise'and b.type = 'course'and a.event_date >= to_date('20100101 00:00', 'YYYYMMDD HH24:MI')and a.event_date < to_date('20100501 00:00', 'YYYYMMDD HH24:MI')and a.site_id = b.site_idgroup by a.event, ref, a.site_id, b.;tle



• Results...

48



• Results...

49


Looking Across Courses for Trends...

• Coun;ng all events in a course over the semester

• Compare / Contrast units, instructors, tools, etc.

50


• Basic percentage of use differences between campus types:

51

Looking Across Courses for Trends...

ToolResiden<al Campus

Commuter Campus

Mean Difference

Resources 58% 52% 5.3

Drop Box 5% 3% 2.2

Email Archive 0.5% 0.1% 0.4

Chat 1% 2% 0.9

Forums 1% 3% 1.7

Messages 3% 6% 3.3



52



ng Local Sakai Use


Nuno FernandesUniversidade Fernando Pessoa, Portugal

PhD. David Roldán MarhnezUniversidad Politécnica de Valencia


Site Stats: The power of event tracking at a single click in Sakai

53

The beginning of the project

• Site usage sta;s;cs

• Audience:• Sakai administrators

• Site maintainers (instructors,…)

• Help desk and support staff

• Available for Sakai 2.2+ (2.5+ recommended)

• Based on Sakai Event service

2

Main features

• Summary informa;on about site visits, tool ac;vity and resource ac;vity

• Advanced repor;ng capabili;es• System and user defined reports• Table and chart presenta;on• Data exportable to .csv, .xls and .pdf

• Support for several Sakai tools• Complete feature list at:

• h\p://confluence.sakaiproject.org/display/STAT/Features

3

Site maintainer view

• Tool organized in 3 pages:• Overview(summary informa;on for site visits and ac;vity)

• Reports(detailed site visit, tool and resource ac;vity reports)

• Preferences(configure the tool on a site-‐basis: define tools or events to be included as ac;vity on overview page)

4

OVERVIEW PAGE

Site maintainer view:

5

Site maintainer view: Overview

• Sta;s;cs (totals and averages) about site visits and ac;vity

6


• Visits detail:

7


• Ac;vity detail:

8


• Resources detail:

9

REPORTS PAGE

Site maintainer view:

10

Site maintainer view: Reports

• Generate reports based on:• site visits• tool events• resource ac;vity

• Narrow search by:• specific tools, events or resources• pre-‐defined or custom ;me periods• all/specific users, roles, groups or non-‐matching users

• Excel, CSV and PDF file export

11


12


13


14

CONCLUSIONS

Finishing…

15

Finishing: Conclusions

• Site Stats provides a simple way to get basic informa;on about how your Sakai sites are used

• Useful for help desk and support ac;vi;es

• Highly configurable to adapt to different deployments

• Open to sugges;ons and contribu;ons

16

Thank You!• For more detailed & technical demonstra;ons of Sakai data use, please a\end:#156 "Unraveling the run<me behavior of Sakai"Tomorrow morning, 10:00am, Grand Mesa A

• John Leasia: [email protected]

• R.P. (Adi) Aditya: [email protected]

• Steve Lonn: [email protected]

• David Roldan Mar;nez: [email protected]

69

under the hood: using the data generated by sakai

Documents