2015 16th annual pabug conference bursting the bubble [sheets]: moving course evaluations from paper...

33
2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake Andrea Rosenberry

Upload: dana-kennedy

Post on 18-Jan-2016

219 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Bursting the Bubble [Sheets]:Moving Course Evaluations from

Paper to Online

Shippensburg UniversityJohn VandeBrake

Andrea Rosenberry

Page 2: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

General Announcements:• Please turn off all cell phones/pagers

• If you must leave the session early, please do so as discreetly as possible

• Please avoid side conversations during the session

• Questions will be answered …..

Thank you for your cooperation

Page 3: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

To receive CPE credits for this session (if eligible), complete the CPE Attendance Form on the PABUG Annual Conference website

http://pabug.org/pennsylvania-banner-users-group-annual-conference/cpe-credits/

or via the PABUG conference app.

For additional questions please contact

Kim Fremont CPE - Coordinator http://pabug.org/contact-us/

CPE Credits - We’ve gone electronic!

Page 4: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

About

Page 5: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Agenda

OverviewOpportunities

ChallengesKeys to Success

OutcomesExplorance Blue PeakFuture Enhancements

Q & A

Page 6: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Legacy system using Opscan paper bubble sheets• Collective Bargaining Agreement (CBA) specifications• Collaboration between provost and faculty key• SIS integration – Explorance Blue Evaluation chosen

due to Banner integration• Data security: Explorance versus Survey Monkey or

other non-SSO options• Three year pilot launched starting with Athletics

Overview

Page 7: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Paper versus Online• It goes without saying…..lots of manual work!

Overview

Page 8: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Paper process• Pre Printing of the Opscan forms with course information• Distribution of the paper forms to department offices then

class• Collection of paper forms and security concerns• Functional issues with the Opscan machine• Issues with bubble forms filled incorrectly• Issues with pre printing on forms• Computation of results in legacy software• Printing and distribution of paper results• Storage and retrieval of paper results

Overview

Page 9: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Eliminate paper processing saving time for both provost office, secretaries, and IT

• Security of documents improved• Student convenience improved• Student responses more accurate and

thoughtful• Computation, distribution and retrieval of

results easier

Opportunities

Page 10: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• SIS Integration: Banner to Blue• Process changes: Explorance Blue vs paper• Online response and evaluation rate variance• Data integrity and automation for provost

office• CBA requirements & faculty adoption

Challenges

Page 11: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

SIS Integration

• Oracle Views connect to Explorance Blue

• Authentication - SSO using CAS

Challenges

Page 12: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Banner Data Integration

• Data is term driven• Non-distinct CRN

– Numbers reused across terms– Team Taught Courses

• Not all courses should be evaluated

Challenges

Page 13: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Challenges

Page 14: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Process Changes: Explorance Blue • Pre-survey sent to enlist “volunteers” from Blue

– Volunteer versus required to evaluate– Excel flag file data extension block used in Blue

• Volunteer list from survey combined with required• Evaluation survey scheduled for launch• One click thru for students to evaluation

– Easy to use– Mobile enabled– Only one time completion allowed

Challenges

Page 15: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Process limitations• Delivery mechanism changes with online

• Instrument moved to online– Challenge with collection of written responses and privacy

• Process changes– 24/7 access for two week window– Lack of “Captive audience”

• Reporting changes• Mirrored reporting as much as possible• Privacy of written responses

Challenges

Page 16: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Evaluation Rate Variances

• Evaluation rate slightly lower, .2 to .5 on Likert scale. – Announcement made to faculty“…a negative variance in this range may be observed

when comparing online to paper. And that such variation should be considered insignificant and may not be used as the basis for negative evaluation.”

Challenges

Page 17: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Response Rate Variance• Response rates have somewhat dropped

– “Captive audience” effect neutralized– Reminder emails sent to those not responding– Faculty encouraged to announce in class

• Helps increase participation

• More could be done to encourage online participation– Advertising?– Contests?– LMS Integration?

Challenges

Page 18: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Data Integrity and Automation• Paper process had potential inaccuracies

– Paper responses were sometimes incomplete and hard to validate

– Response rates were not always accurate– Paper slugging of forms was not always accurate

• Maintaining data integrity online in Banner• Moving offline data into Blue thru Banner

integration

Challenges

Page 19: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

CBA Requirements – Faculty Adoption• Shippensburg works with unionized faculty

under a collective bargaining agreement (CBA)– Mirrored process as much as possible– Mirrored paper documents as much as possible– Mirrored time lines as much as possible

• Faculty adoption encouraged thru collaboration with representatives and committee as well as through the pilots

Challenges

Page 20: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Communication between provost, faculty, athletics, and IT– Complete support of the Provost– Representation included leadership, athletics, faculty,

union, and IT in forming a process plan– Regular committee meetings to receive input– Implementation of changes as technology permits– Communication to the larger campus community vital

Keys to Success

Page 21: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Three year pilot with three phases– Phase 1: Focus on athletic coach evaluations and early

adoption as well as training• Athletic evaluations established success• Training meetings held for pilot faculty

– Phase 2: Process focused: adjustments and improvements• Meetings held to review process and gather input• Online reporting improved to match paper reporting• Reminders tweaked

Keys to Success

Page 22: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Three year pilot comprised of three phases– Phase 3: Adoption: Improve participation and tweaks

• Encourage participation• Added faculty monitoring of response rates during

evaluation cycle• Supervisory report viewing enhanced

Keys to Success

Page 23: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Budget considerations help drive adoption– Good pilots and minor technical glitches made

adoption more likely– Show how budget savings could benefit faculty

• Software ease of use: Explorance Blue has served us well

• Responsive leadership will help pave the way

Keys to Success

Page 24: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Slight decrease in quantitative responses• Increase in qualitative data

– Written responses more detailed and legible– Quality of responses improved. Longer and more

thoughtful• Use of own time to complete allows for more thought

• Reduces the use of paper and processing• Saves class time for faculty• Insures data integrity• Quick reporting turn around• Distance Ed and offsite course evaluations easier

Outcomes

Page 25: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Evaluation Instrument: sent with specific class details to each student

Explorance Blue Sample

Page 26: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Three reminder emails sent to those not completed

Explorance Blue Sample

Dear Susan Hickenbaugh,

REMINDER: We have not yet received your feedback for these course(s):

-Charles Stanley (CSC570-01 Database Mgt Systems),-Joe Leevy (CSC521-01 Operating Systems),- Joe Leevy (CSC502-01 Automation Theory)

We would really appreciate it if you can take a few minutes now to complete the student course evaluations.

We are using online survey software to make it easier for students to give us their feedback.

Click here now to provide us with your feedback.

Remember: You will need to make sure you click the "Submit" button to complete your evaluation.

If you experience any technical difficulties or have any questions please contact T Pearson in the Provost office at [email protected]

Page 27: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Explorance Blue Sample

Page 28: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Explorance Blue Sample

Page 29: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Subject View: Response rate review during evaluation cycle

Provides the ability for faculty members to view the cumulative response rate as the evaluations is being administered.

Explorance Blue Sample

Page 30: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Mobile enabled for more input options– Potential for classroom

administration– Easy access

Explorance Blue Sample

Page 31: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

• Further automation in the provost office– Volunteer and required selection process in SIS

• LMS Integration• Participation promotion• CBA requirements reassessment • Parts of term and interactive start/stop times• Long term storage of reporting• Survey add-on tool for Title IX, Snyder File V data

collection, and other surveys

Future Enhancements

Page 32: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Open to the Floor

• Questions• Comments

Page 33: 2015 16th Annual PABUG Conference Bursting the Bubble [Sheets]: Moving Course Evaluations from Paper to Online Shippensburg University John VandeBrake

2015 16th Annual PABUG Conference

Please take time to complete our Conference and Session

evaluations.

YOUR INPUT MATTERS!!!