Annual Report on Annual Report on Institutional Effectiveness:Institutional Effectiveness:Pathway to ExcellencePathway to Excellence

Contributing ColleaguesDr. Julia Pet-Armacost, IAA

Dr. Taylor Ellis, UACDr. Michael Johnson, UAC

Dr. Paula Krist, OEASDr. Angela Albert, OEAS

Uday Nair, OEAS

August 30, 2004

UAC Annual Report 2August 30, 2004

Overview

institutional effectiveness at UCFenhancements made this yearstatus report on program assessmentnational visibilitynext steps

UAC Annual Report 3August 30, 2004

UCF’s Integrated Approach to Institutional Effectiveness

ProgramAssessment

Unit and Program Reviews

StrategicPlanning

linkages•share information•inform budget process

differences•different cycles•additional data elements•different purposes

•continuous improvement•evaluation•planning

UAC Annual Report 4August 30, 2004

Program Assessmentformative evaluation process designed to support improvementcontinuous

focused on improvingstudent learningstudent developmentservices and operations

Do

Act

Plan Check

UAC Annual Report 5August 30, 2004

Brief History of Assessmentat UCF

SACS reaffirmation visit, little evidence of assessment

Nichol’s modelIs adopted

Programs asked to conduct assessment

annually and submit for review every three years

OEAS Office isestablished and

President announcesannual review cycle

Web-based submission and review, Divisional Review Committees,

Web access to surveys

UAC is established and SACS accepts

UCF report

1995 1996 1997 19981998 19991999 20002000 20012001 20022002 20032003 20042004

UAC Annual Report 6August 30, 2004

This Year’s Enhancements

one-time submissionassessment handbooksUAC & DRC policies and proceduresenhanced focus on student learning outcomesnew partnership with FCTLNSSE and FSSEenhanced access to surveys

UAC Annual Report 7August 30, 2004

Status Reportsubmissionsmethods of assessmentstudent learning outcomeschanges made on the basis of assessment

academic programsadministrative units

UAC Annual Report 8August 30, 2004

Number of Programs and Units Conducting Assessment

202

254278 296 310

0

50

100

150

200

250

300

350

2000 2001 2002 2003 2004

Year

UAC Annual Report 9August 30, 2004

Submission Rates

Submission of Assessment Plans

81.0%92.0% 97.5% 99.0%

98.3%

0%

20%

40%

60%

80%

100%

1999 2000 2001 2002 2003

Year

Subm

issi

on R

ate

UAC Annual Report 10August 30, 2004

Assessment Tools Used in 2004

3.3%

11.3%

12.7%

18.3%

21.1%

21.1%

22.1%

61.5%

0% 10% 20% 30% 40% 50% 60% 70%

Scoring Essays

Other

Capstone Project or Performance Evaluations

Standardized (state normed) Exams

Standardized (nationally normed) Exams

Evaluation of Portfolios

Other Evaluations Using Scoring Rubrics

Locally Developed Tests

Percent UsageAcademic Administrative

UAC Annual Report 11August 30, 2004

Surveys Used in 2004

0.0%

9.4%

11.3%

55.4%

57.3%

0% 10% 20% 30% 40% 50% 60% 70%

National Surveys

Employer Surveys

Alumni Surveys

Other Locally (program or unit)Developed Surveys

UCF's Graduating Student Surveys

Percent UsageAcademic Administrative

UAC Annual Report 12August 30, 2004

Percent of Programs with 1 to 9 Program Outcomes

74.4%

37.5%

70.0%

7.1%

30.8%

52.3%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

CAS CBA CED CECS COHPA UNIVERSITY

1 2 3 4 5 6 7 8 9

UAC Annual Report 13August 30, 2004

Percent of Undergraduate Programs Addressing Particular Student Learning Outcomes

0%10%20%30%40%50%60%70%80%90%

100%

Discipline Knowledge,Skills, Attitudes, &

Behaviors

Communication Skills Critical Thinking Skills

CAS CBA CED CECS COHPA

UAC Annual Report 14August 30, 2004

Percent of Undergraduate Programs Addressing 0,1,2, or 3 Student Learning Outcomes (SLOs)

0% 20% 40% 60% 80% 100%

CAS

CBA

CED

CECS

COHPA

None 1 SLO addressed 2 SLOs addressed 3 SLOs addressed

UAC Annual Report 15August 30, 2004

Curricular Changes Made to Academic Programs

26%

24%

23%

12%

0% 10% 20% 30% 40%

Revise and/or EnforcePrerequisites or Course

Sequence

Revise Course Content

Add Course(s)

Delete Course(s)

2003-04 2002-03

UAC Annual Report 16August 30, 2004

Process Changes Made to Academic Programs

15%

13%

16%

6%

12%

9%

0% 10% 20% 30% 40% 50%

Modify Frequency or Schedule ofCourse Offerings

Made Technology RelatedImprovements

Made Personnel Related Changes

Implement Additional Training

Revise Advising Standards orProcesses

Revise Admission Criteria

2003-04 2002-03

UAC Annual Report 17August 30, 2004

Changes Made by Academic Programs to Assessment Plans

21%

28%

18%

26%

0% 20% 40% 60% 80% 100%

Revise StudentOutcome Statement(s)

Revise MeasurementApproach(es)

Collect and AnalyzeAdditional Data and

Information

Change Method(s) ofData Collection

2003-04 2002-03

UAC Annual Report 18August 30, 2004

Process Changes Made by Administrative Units

43%

57%

36%

43%

36%

9%

21%

0% 10% 20% 30% 40% 50% 60% 70%

Revamp Services

Make Technology RelatedImprovements

Make Personnel Related Changes

Implement Additional Training

Add New Service(s)

Delete Service(s)

Other Change(s)

2003-04 2002-03

UAC Annual Report 19August 30, 2004

Changes Made by Administrative Units to Assessment Plans

40%

43%

23%

37%

0% 10% 20% 30% 40% 50%

RestructureObjective

Statement(s)

Revise MeasurementApproach(es)

Collect and AnalyzeAdditional Data and

Information

Change Method(s) ofData Collection

2003-04 2002-03

UAC Annual Report 20August 30, 2004

Examples of Quality Improvements (Academic Programs)

implemented curricular changes that enhanced students’ writing skills (CAS, Chemistry, BA)

increased course offerings; also increased student enrollment (COHPA, Health Information Management Program, BS)

achieved 100% pass rate among program completers on the Florida Teacher Certification Exam in this specialty; also activated graduate certificate program (CED, Instructional Technology/Media, Educational Media, MEd (Online)

UAC Annual Report 21August 30, 2004

Examples of Quality Improvements (Academic Programs)

making changes to improve critical thinking, problem solving, presentation, and writing skills(CECS Industrial Engineering and Management Systems)

intensified effort on developing student learning outcomes; also implemented balanced scorecard that resulted in creation of stronger collaborative environment (CBA, Academic Programs & Undergraduate Student Services)

UAC Annual Report 22August 30, 2004

Examples of Quality Improvements(Administrative Units)

developed web application that improves access and process turn-around time (A&F: Facilities Planning)

improved training and customer service (SDES: Student Academic Resource Center)

enhanced faculty perception of the value of diversity in the curriculum (President’s Division: Diversity Initiatives)

enhanced capability to provide services (Academic Affairs: University Analysis and Planning Support)

increased client base, quality, awareness, and involvement (Sponsored Research: Technology Incubator)

UAC Annual Report 23August 30, 2004

Examples of Quality Improvements(Administrative Units)

shifted focus to outcomes related to increasing number and types of employers recruiting UCF students (SDES: Career Resource Center)

focused on new ways of educating clients by offering workshops on how to use services; also enhanced communication with clients (SDES: Student Union)

increased student satisfaction regarding safety in their living environments (SDES: Housing)

UAC Annual Report 24August 30, 2004

Increasing Our National Prominence in Assessment

conference presentations 28 presentations (15 last year)

1 invited plenary1 invited paper1 selected as “best paper”

at 18 different conferences (7 last year)primarily by 3 colleges and IAA staff

workshops presented6 workshops (5 last year)

1 paper published in J. of General Education

UAC Annual Report 25August 30, 2004

28 Conference Presentations

AACTE (1)AACSB International(1)AACSB Undergraduate (1)AAHE Assessment (1)AIR (4)FAIR (3)FATE (1)FIE (1)LiveText (2)

NCCI (1)NCSU Symposium (1) invitedSACCR (1) invited plenarySBA(1)SACS (2)SAIR (3)SCUP National (1)SCUP Regional (1): selected as best paperSESUG (2)

UAC Annual Report 26August 30, 2004

Workshops Conducted

This Year = 6AIR (2) BAP VI (1) invitedSACS (2)SAIR (1)

Accepted for Next YearSAIR (1)SACS (2)

UAC Annual Report 27August 30, 2004

External Consultations andSite Visits

Consultations and AssistanceOur Lady of the LakeNorthwestern State UniversityEmbry RiddleDBCCUSFFAUBOG and DCU

SACS Visiting Team 2004Clayton College

Planned ConsultationsFAMU UNF UWF

SACS Visiting Team 2005Texas Tech

UAC Annual Report 28August 30, 2004

Planned Enhancements for Next Year

develop stronger partnership with FCTLincrease emphasis on learning outcomes in assessment plansimplement Academic Learning Compactsrevise alumni survey with program-level dataimplement recognition for best practicesexpand national reputation

increased participation in conference presentationspeer reviewed assessment-related publications

UAC Annual Report 29August 30, 2004

UCF is Moving Toward Excellence!

SACS reaffirmation visit, little evidence of assessment

Nichol’s modelIs adopted

Programs asked to conduct assessment

annually and submit for review every three years

OEAS Office isestablished and

President announcesannual review cycle

Web-based submission and review, Divisional Review Committees,

Web access to surveys

UAC is established and SACS accepts

UCF report

1995 1996 1997 19981998 19991999 20002000 20012001 20022002 20032003 20042004

Focus on student learning

Partner w/FCTL

Annual Report on Annual Report on Institutional Effectiveness:Institutional Effectiveness:Pathway to ExcellencePathway to Excellence

Contributing ColleaguesDr. Julia Pet-Armacost, IAA

Dr. Taylor Ellis, UACDr. Michael Johnson, UAC

Dr. Paula Krist, OEASDr. Angela Albert, OEAS

Uday Nair, OEAS

August 30, 2004