Program Review by p.mery, dean of institutional effectiveness march 2015

Visiting Team Report – Word Counts

1+ 4001

100 130

200 37

300 9

400 6

500 2

1000 1

Grand Total 4186

number of

words

times they

appear

Visiting Team Report – Word Counts

1+ 4001

100 130

200 37

300 9

400 6

500 2

1000 1

Grand Total 4186

number of

words

times they

appear

“evidence” = 405

“student” = 472 “planning” = 306

“Plan” = 335

“college” = 985

“program review” = 159

“data” = 160

“institution” = 521

Most words occur very infrequently 1496 occur once 2300 occur < 20x

~Top 50 words used to create the “Wordle” (see prior slide) appear far more often than the 4000+ words not included in the Wordle.

“Program Review” is prominent among the ~Top 50.

Quick Quiz

What is the purpose of program review?

Quick Quiz

The purpose of program review is...

◦ Unit level improvement

◦ Collectively achieving institutional goals

◦ Fulfilling our mission

◦ Reaching toward our vision

◦ Meeting educational needs of students

◦ And yes, connection to budget… but as a means not an end.

Self Evaluation

Where are we really?

What examples of fully looped CQI can you cite

based on this year’s program reviews?

Self Evaluation

Where are we really?

Improvements Made as a Result of Programmatic Assessments (I.B.1. excerpt, pages 44-45)

DEPARTMENT/UNIT IMPROVEMENTS

Engineering & Technology As a result of SLO assessment, the Engineering and Technology degree was

revised into two distinct majors with refined major descriptions and course

requirements. It now offers an Electronic Engineering AS and a Mechanical

Engineering AS.

Mathematics Assessment of different delivery modes highlighted that a hybrid, team

teaching method of instruction resulted in higher pass rates. More hybrid

sections are now being scheduled.

Automotive/ Motorcycle

Technology, Construction

& Building Maintenance

Now offers six stackable certificates building up to an AS degree that allows

students to complete a section and return to a work site with recognized

national certification skill sets.

Culinary Arts & Hospitality

Studies Improved and better aligned student expectations and skill building by turning

a 3-unit class into 3 one-unit classes. These one-unit courses drilled down

subject matter and allowed students to tailor skills and build a specialized

culinary portfolio.

Speech Communication After direct assessment of student speeches, students struggled to articulate

their own credibility through research and experience. Custom instructional

materials were created using highly relevant, localized, and current subject

matter.

VTR Evaluation

Where are we really?

Overall, we need to…

make better use of data

maintain a learning outcome focus

get more employees involved

clarify some of the criteria

yes... connect it to budget

overall… build it into our DNA

VTR Evaluation

Where are we really?

I.B.1. The college will be able to maintain compliance assuming it continues its ongoing dialogue about continuous improvement of student learning and institutional processes; and the current and planned staffing levels in the Office of Research and Planning are achieved and maintained so that plans to enhance access to disaggregated program-level data are implemented, the use of data in program review is more consistent across all of the programs, and the assessment question in program review is further clarified.

VTR Evaluation

Where are we really?

I.B.4. In order to remain in compliance, the

college needs to continue efforts to engage more employees in department and college planning activities, determine and implement processes to reallocate existing resources as appropriate.

VTR Evaluation

Where are we really?

II.A.2.c

While the college's program review process is sufficient, a process that made more robust use of data would lead to better-informed continuous improvement.

VTR Evaluation

Where are we really?

II.A.2.e

There is evidence that program review in general at the college is just starting to take shape and that faculty are only starting to fully understand the relevance of the process and importance of planning.

VTR Evaluation

Where are we really? II.B.3. SLO assessments and program review analysis

are taking place, but it is not clear that robust discussions and collaboration between the Ocean Campus and the centers are taking place. As mentioned in Standard II.B.1, student services programs have developed SLOs and program reviews which require the inclusion of SLOs in order to be submitted, but the assessments of these SLOs are in different phases depending on services and locations.

VTR Evaluation

Where are we really? III.D.4. The College’s departments and units develop budget

plans through the use of program review documents completed each year during cycles that integrate with the College’s financial planning calendar. Although the College has been conducting Program Review for many years now, the college is still making strides in ensuring full participation by all programs and completion of all requested elements of the Program Review. The first year to allocate resources using program review was FY 2013-14. This allocation was only of additional funding or staffing and did not tie base allocations to program review.

VTR Evaluation

Where are we really?

III.D.4., continued

During 2014-15, the College will assess the allocations as to the effectiveness and improvements the College achieved. The College is conducting these assessments during the Fall 2014 semester.

VTR Evaluation

For additional analysis…

Members of the Academic Senate including the

Senate President mined the VTR and developed Program Review Processes Consideration, an extensive document called to create dialogue around our Program Review processes.

https://docs.google.com/document/d/1GObJKoqBjatgOMXQnxSRen-8naU_IHKPwS8vhKoGw0U

Your next steps

Your top six Ways to increase (focus on) institutional

effectiveness and CQI: 1. 2. 3. 4. 5. 6.

Your next steps

Your top six Some possible ideas: 1. Become more data-informed 2. Seed some future “looped” examples 3. Collect evidence (follow the loop) 4. Encourage involvement 5. Do nuts & bolts, e.g., budget workshops 6. Be the culture (A-Team models!)

Keep Moving & Modeling

How can you inspire others? List some things you do every day to model

the principles of CQI: 1. 2. 3. 4. 5. 6.

Keep Moving & Modeling

What gets in the way? Some common issues:

◦ Silos ◦ Distrust ◦ Everyday tasks ◦ You name it! ◦ The impediments won’t vanish but we need to keep coaxing, pushing, and modeling CQI

Integrated Assessment, Planning, and Budgeting

A presentation on Program Review is never complete without the flowchart!

Shown on next slide. Note the multiple,

interconnected CQI and assessment wheels beginning with SLOs which inform program review and annual planning, all the way up through long-range, institutional planning.

Quick Reminder