oire systems projects department heads meeting, 3/11/2013 suresh nair, ph.d. interim associate vice...
TRANSCRIPT
OIRE Systems Projects
Department Heads Meeting, 3/11/2013
SURESH NAIR, Ph.D.
Interim Associate Vice Provost for Institutional Effectiveness
Professor, School of Business
University of Connecticut, Storrs
1
Overview
Online Student Evaluation of Teaching (SET)
HuskyDM enhancements HuskyAL – Assessment of Learning OIRE Self Service DataMart
2
Online SET – Student Evaluation of Teaching
The new 5 point form required by the Senate is being implemented
It will be an online system Students in courses ending after March
22 will receive online SET links by e-mail.
3
4
History of the SET
We have been using a paper-based 10-point SET Average of ratings is reported
In Fall 2010, a new 5-point SET was approved by Senate after trials—strong reliability and validity http://senate.uconn.edu/evaluations/SEoT.guidelines.pdf Medians of ratings are to be reported
eXplorance Blue was chosen as the vendor for the new SET—the goal was to use paper SET’s with a quick turn-around and electronic reporting
A pilot was run in Fall 2012 on a subset of courses scored using both the 10-point and 5-point SET
5
Problems with Paper SET
Multiple Problems with Paper SETs scoring Printing of paper forms is a logistical problem
due to manual nature of process Resolution of scanning is on new form poor,
requiring constant redos. New scanning software yields 10% uncertainty of student responses requiring manual intervention and “guessing”
Possible inaccuracy when instructor changes after 10th day of class and paper SETs already printed with old name.
Only one vendor was available in RFP process, since almost every major university in the country schools has moved to completely online systems
Online SET – Student Interface6
Online SET – Instructor Report7
Online SET – Instructor Report (contd.)8
9
The path forward
Rollout of the 5-point fully online scoring and reporting system planned for Spring 2013 (opt outs available)
Student scoring Evaluation and comments to be entered online Students may use computers, tablets or smart
phones Strategies to improve student response rate
Push green/sustainability angle Use promotional videos to promote online
scoring Early access to grades (say, two days before
others)
10
The path forward (contd.)
Instructor and DH reporting Will be able to see reports online much sooner
than the paper system Instructors will be able to see comments online Research has shown that the quantity and
quality of text comments online is much better than comments on paper
Opt-out: Faculty may request paper scoring after obtaining consent from DH
HuskyDM status update
The Provost Annual Report (PAR) for last year was created by HuskyDM ~95% faculty entered their information We have noted all the feedback and are
making improvements this year – discussed next
Additionally, HuskyDM can this year be used for Creating PTR form drafts Merit forms
11
Publications upload
Demo
12
HuskyDM enhancements (contd.) Some of the most common complaints were
“data not appearing in proper PAR item cell” Reason – faculty may not choose to enter all the
fields required by the logic for creating PAR Resolution – we are redoing most screens to now
directly ask for the PAR# via a drop-down menu
13
HuskyDM enhancements (contd.)
Similar to Publications Upload, we are planning to upload the following information directly into HuskyDM this year (this work is not complete) Grants and Contracts HR data Advisee data
PTR forms via HuskyDM This has been tested with ~20 TT faculty
this year This is a convenience available to TT
faculty, not a requirement
14
PAR Redesign
The Provost Annual Report (PAR) has remained unchanged for about 20 years.
A committee of DHs and users is currently looking at making it less confusing and more useful
15
HuskyDM enhancements planned (contd.)
Merit forms via HuskyDM We have contacted each school to collect
their Merit forms BUS has been using HuskyDM for merit for
5 years now Again, a convenience, not a requirement
16
HuskyAL – Assessment of Learning
We are working on an online tool to replace OATS
DHs and Program Directors are responsible to continue to Assessing their programs All programs should have Mission and
Purpose, Goals, Objectives/Outcomes and Methods, as before
All programs should track results , and specify follow-up actions, as before
The only difference is, instead of posting into OATS manually, you will upload into HuskyLM using Excel.
17
OIRE DataMart
We are working on a self service DataMart for OIRE data
Departments and Schools should be able to access routine data using a WebFocus online interface.
Will have some functionality to manipulate data somewhat like Pivot tables in Excel
18
Contact Information
Suresh Nair, Interim Associate Vice Provost for Institutional Effectiveness, [email protected]
HuskyDM Alexander del Campo, School of Business,
IT Services, [email protected]
Liming Liu, OIR, [email protected] Online SET
Cheryl Williams, OIR, [email protected] Valorie Elwell, UITS, [email protected]
Georgianne Copley, UITS, [email protected]
19
20
Comparison between 10 and 5 point scales
In Fall 2012, 39 courses were scored on both the 10-point and 5-point scales
The 5-point median scale actually resulted in higher SETs than the old 10-point scale (9% higher)Average of
10-point scale
Twice median of 5-point scale
Count of courses
5-6 73% 5%6-7 118% 5%7-8 103% 13%8-9 112% 77%Total 109% 100%