moving from “buy-in” to “process improvement” and to “planned change”

Moving from “buy-in” to “process improvement” and to “planned change”Jack Phadungtin & Darla CalvetNational University, La Jolla, CA

LOGOAbout National University

Type: Private, Not-for-ProfitCampuses: 28Students: 27,000Faculty: 3,100 (full-time & adjunct)Degree programs: 98 (grad & undg)Classes: 800-1000/monthNewly admitted students: 17,000/YrIR & Assessment staff: 5

LOGOMomentum to SuccessMomentum to Success

ContinuousImprovement

Planned Change

ProcessImprovement

Buy-In

Institutional Effectiveness

LOGOIngredients of Winning Planned Change

Assessment Infrastructure Support from Administration Capability of the IR Office and Faculty

Assessment Process Assessment Cycle Governance bodies and their responsibilities

Effective Use of Assessment Results

LOGOIngredients of Winning Planned Change

Assessment Infrastructure Support from Administration Capability of the IR Office and Faculty

Assessment Process Assessment Cycle Governance bodies and their responsibilities

Effective Use of Assessment Results

Buy

In

LOGOIngredients of Winning Planned Change

Assessment Infrastructure Support from Administration Capability of the IR Office and Faculty

Assessment Process Assessment Cycle Governance bodies and their responsibilities

Effective Use of Assessment ResultsLess

Res

ista

nce

to C

hang

e

LOGOAssessment Goals at NU

Enhance assessment qualityImprove assessment process

Utilize assessment results

LOGOHow To Initiate Planned Change

Administration Faculty

Who Originates Change?

LOGOHow To Initiate Planned Change

Administration Faculty

Office of IR & Assessment

LOGOHow To Initiate Planned Change

Administration Faculty

Office of IR & Assessment

ChangePersonnel, Structure, Mind

ChangePersonnel, Structure, Mind

LOGOGetting Started

Stop the “Resistance-to-Change” talk Isolate Problems & Prioritize Tasks

1. Assessment Cycle 2. The needed “buy-in”

– Support from Administration» Capability of the IR Office

– Support from Faculty» Understanding Benefit of Assessment

3. Governance bodies and their responsibilities

LOGOBuy the “Buy-In”

Administration Cost and Benefit Analyses Recommendations Successful Pilot Projects

Faculty Remove Barriers Simplify Tasks Identify Project Champion Compensation

LOGOBuy the “Buy-In”

Amplify Benefits of ChangePromote Effective CommunicationUnderstand Each Functions Involved in the

Process

Sample??

LOGOAssessment Time Table (No longer used)

LOGORevised Assessment Cycle

LOGOMoving from Buy-In to Process Planning

Create Master PlansIdentify Key Indicators/Measures

Academic Non-Academic

Plan for Data Collection and AnalysisEstablish practices of how results will be

used

LOGOIdentify Key Indicators - Academic

Student Surveys

LOGO

Non-Academic Assessment Business Quality Survey Staff Survey

Identify Key Indicators – Non-Academic

LOGONon-Academic Measure 2008

Frequency of Survey – 4/Year 2 Quality of Business Survey (QBS) 2 HR Surveys

Jan

Pre

Feb Mar

QB

S

May Jun

Pos

t

JulQ

BS

–Foc

used

Aug

Pre

Sep Oct Nov

Pos

t

DecApr

Pre Post

Control Group 1 Control Group 2

LOGOUsing Results to Drive Improvement

To Promote Positive Planned ChangeGive Credit to Project Originator/OwnerShare Results With StakeholderKeep Track of How Results Are UsedPrevent Reprimand for Inability to Achieve

the Set Goals

LOGOInformation Optimization

LOGOTracking Information Usage

LOGOPromote Use of Evidence for Process and Structure ChangesChange Process to Support Change

Simplify and Minimize Steps Emphasize Quality Control

Change Structure to Support Change Enhance Institutional Capability

LOGO

Process Improvement2006

2008

LOGOProcess / Structure Improvement:Academic Program Reviews

2006

2008

LOGOProcess & Assign Responsibilities:Academic Program Reviews

Program LeadFaculty

UniversityAssessment

Council

Department Chair

Provost

School Assessment Committee (SAC)Members are appointed by the school Dean

School Dean

Undergraduate Council

President

Process Coordination

Process Coordination

Process Coordination

Graduate Council

Process Coordination

Process Coordination

LOGOStructure / Capability Improvement

2006

2008

LOGOImprovement Results

Academic Improvement 2007

LOGOMAT EXIT SURVEY: Fiscal Year 2006 Graduates

ABSTRACTWithin the context of an academic program review at National University a

web-based survey was conducted asking recent graduates of the Masters of Arts in Teaching (MAT) program about the value of core MAT courses and whether they believed program outcomes were met. The survey was the second annual exit survey administered. Respondents rated the value of core courses, and rated agreement about whether program outcomes were met. Respondents were also given opportunity to provide open-ended rationale for their ratings.

PURPOSEThe purpose of the study was to discover, through perceptions and experiences

of recent graduates of the Masters of Arts in Teaching (MAT) program, the value of core MAT courses and whether graduates believed MAT program outcomes were achieved. Data was gathered by way of a web-based survey.

LITERATURE REVIEW

The purpose of an internal program review is to provide opportunity for continuous improvement through a systematic and cyclical process in which strengths of a program and areas for growth are identified (Kornuta, 2007; Daniels, 2005; LSU Planning & Review, 2006). Data for program reviews should come from multiple sources (NCATE, 2003; SIEU, 2006), and involve multiple measures such as teacher candidates' grades, scores on standardized entrance and exit exams and portfolios; community service data; employer surveys; and the extent to which students are meeting learning outcomes (Kornuta, 2007; Hess, 2002; University of Delaware, 2006).

Outcomes assessment allows institutions to measure goals unique to particular programs and identify successes and the need for growth without regard to the size or prominence of the institution (Jones, 1996; NCATE, 2003; SIEU, 2006; Western Association of Schools and Colleges, 2001).

BACKGROUNDAcademic programs are reviewed at the University on a five year cycle. The

comprehensive review process addresses the adequacy, viability, and necessity of the programs. During the program review process, a cross-check occurs to ensure that program goals and learning outcomes are aligned with the mission of the University, the mission of the School of Education, and requirements of both national and state accrediting agencies.

Within this context, a second exit survey was administered to recent graduates of the Masters of Art in Teaching (MAT) program. The data collected is compared to the previous survey, added to longitudinal data, and will contribute to the MAT Program Review.

Ron Germaine, Ed.D.Contributors: Jack Phadungtin, DBA; Cynthia Schubert, Ed.D.

DATA ANALYSIS

Population = 1648 N=290 (18% response rate) Not all respondents answered all questions; however, reliability is strong because respondents were representative of the geographic and online distribution of students.

Response to Question:“Rank the value of each of the six MAT core courses in

terms of how valuable each course was to the development of the your teaching ability.”



For additional information please contact:Ron Germaine, Teacher Education Department, National Universitygermaine@nu.edu

METHODOLOGY• A web-based survey was distributed in August 2006 to all students who

completed the capstone course MAT 640 between July, 2005 through June, 2006.

• Participants were provided with a unique personal identification number (PIN) code.

• Respondents were asked to use a Likert-scale to rank the value of each of the six MAT core courses in terms of how valuable each course was to the development of the their teaching ability;

• Respondents were asked to rank how strongly they agreed that the MAT Program helped to develop their knowledge, skills, and abilities in regard to each of the learning outcomes.

• The order of individual questions in each section was randomized to minimize the risk of survey fatigue occurring and contributing to skewed results.

• Validity of the survey was addressed by composing questions based on course learning outcomes and MAT program goals.

MAT CORE COURSE

# of + Resp.(# of +

Comments)

# of – Resp.(# of -

Comments)

Ratio+ : - Resp(+ : - com.)

% of +

Resp.

1. EDT 612: Creating Meaningful Learning with Technology

147(140)

29(25)

5:1(6:1)

85

2. MAT 641: Cultural Democracy: Contemporary, Local, and Global Issues

120(111)

38(33)

3:1(3:1)

76

3. MAT 642: Program Design: Curriculum Theory, Design, and Assessment

138(124)

25(25)

6:1(5:1)

85

4. MAT 643: Models of Teaching, Theories, Applications, and Practice

148(134)

16(15)

9:1(11:1)

90

5. MAT 644: Foundations and Principles of Curriculum

139(125)

27(26)

5:1(5:1)

84

7. MAT 640: Applications of Research for the Art of Teaching

183(171)

26(27)

7:1(6:1)

88

Average 146

(134) 20

(25) 7:1

(7:1) 85

Themes from open-ended responses: “Rank the value of each of the six MAT core courses in

terms of how valuable each course was to the development of the your teaching ability.”

Positive response themes (Note: Positive responses to the value of MAT core courses outnumbered negative responses by ~ 7:1)

• Content was relevant • Example comment: “Information presented correlated with real life instruction.”

• Instructors/instruction was good• Example comment: “Professor very accessible and responsive.”

Negative response themes (Note: Negative responses to the value of MAT core courses were outnumbered by positive responses by ~ 7:1)

• Content was not meaningful• Example comment: “We taught the course by presenting chapters. Few practical

connections to the classroom were made.”• Poor instructor/instruction

• Example comment: “Neither the learning outcomes nor the teacher's expectations were clear.”

Recommendations from Findings • Individually, we (faculty) need to reflect on our own teaching to ask whether we are

modeling best teaching practices and providing clear emphasis on relevant course and program learning outcomes.

• Supervision and professional development of adjunct instructors should take place annually for the purpose of enhancing the quality of teaching and instruction.

• For further study: Comments specific to each course should be reviewed by faculty who teach the course to look for ways to enhance student learning.

MAT Program Learning Outcomes: The MAT Program built my competency to…

# of + Resp.(# of Pos.

Comments)

# of – Resp.(# of Neg.

Comments)

Ratio+ : - Resp.(+ : - com)

% of +

Resp.

1. Reflect on my own teaching practice.

202(184)

8(8)

25:1(23:1)

96

2. Build professional relationships.

179(164)

31(31)

6:1 (5:1) 85

3. Build learning communities within my school and/or classroom.

190(168)

20(20)

9:1(8:1)

90

4. Analyze cross-cultural educational issues.

193(175)

17(16)

11:1(11:1)

91

5. Identify, describe, and apply theories of curriculum as they relate to State approved standards.

184(157)

26(25)

7:1(6:1)

88

6. Implement standards-based assessment in my educational setting.

185(157)

25(23)

7:1(6:1)

88

7. Use technology in my teaching.

187(172)

23(23)

8:1(7:1)

89

8. Use multiple teaching models.

196(184)

14(14)

14:1(13:1)

93

9. Implement, assess and evaluate standards-based curriculum.

182(156)

28(25)

7:1(6:1)

87

10. Use multiple assessment strategies

189(159)

21(21)

9:1(8:1)

90

11. Conduct action research in my teaching

194(163)

16(12)

12:1(14:1)

92

Average Number of Responses & Response Rate

189(167)

20(20)

9:1(8:1)

90

Themes from Open-Ended Responses:“Rank how strongly you agree that the MAT Program

helped to develop your knowledge, skills, and abilities in regard to each of the learning outcomes.”

Positive response to themes: (Note: Positive responses to program goals being met outnumbered negative responses by ~ 8:1)•The learning outcomes were met.

• Example statements: • “I had ample opportunity to effectively put into practice the learning

outcomes that were discussed in my masters classes”• “The entire program added to my capacity to implement State Standards.”• “The courses/professors contributed significantly to my ability to effectively

implement, assess, and evaluate the curriculum currently in use.”

Negative response themes: (Note: Negative responses to program goals not being met were outnumbered by positive responses by ~ 8:1)•Content was not meaningful; Courses are too short

• Example statements: • “One month is just not enough time to absorb and really develop these

learning outcomes”… ; “The course is not long enough…”•Greater need for emphasis on practical application of theory

• Example statement: “The program does not go far enough to take the learning out of books and into the classroom.”

Response to Question:“Rank how strongly you agree that the MAT Program

helped to develop your knowledge, skills, and abilities in regard to each of the learning outcomes.”

Example of Descriptive Statistics Used to Organize Data

Example of Descriptive Statistics Used to Organize Data

0 20 40 60 80 100 120 140 160 180 200 220

The MAT program developed my knowledge skills and abilities to:

Reflect on my teaching practice.

202 or 96%

8 or 4%

# of Students

Strongly Disagree

Disagree

Strongly Agree

Agree

Total # respondents to the question:

210

ReferencesDaniels, M. (2005). Student affairs program review, Ohio State University. Retrieved March 13, 2007 from

http://studentaffairs.osu.edu/programreview/static/Program_Review_Overview.pdfHess, F. (2002). The accreditation game. Education Next. Retrieved March 19, 2007 from

http://www.thefreelibrary.com/_/print/PrintArticle.aspx?id=91821095Jones, E. A. (1996). National and state policies affecting learning expectations. In M. Kramer (Series Ed.) & E.

A. Jones (Volume Ed.), New directions for higher education: Vol. 24. Preparing competent college graduates: Setting new and higher expectations for student learning. (pp. 7-17). San Francisco: Jossey-Bass.

Kornuta, H. (2007). Program review handbook: California Lutheran University. Retrieved March 19, 2007 from http://www.callutheran.edu/assessment/cycle/documents/ProgramReviewHandbookJan102007.pdf

LSU planning and review. (2006). Retrieved March 13, 2007 from http://appl003.lsu.edu/acadaff/lsustrategicres.nsf/$Content/Program+Review?OpenDocument#purpose

National Council for Accreditation of Teacher Education. (2001). Professional standards for the accreditation of schools, colleges, and departments of education. Washington, DC: NCATE.

SIEU. (2006). Report of the University Committee on Assessment: A proposed plan for assessment of undergraduate education at Southern Illinois University at Edwardsville. Retrieved March 13, 2007 from http://www.siue.edu/assessment/plan.html#Principle

University of Delaware. (2006). Periodic review report, p. 67 Retrieved March 13, 2007 from http://www.udel.edu/provost/FinalPRRMay2006.pdf

Western Association of Schools and Colleges. (2001). WASC handbook of accreditation. Alameda, CA: WASC.

“Rank the value of MAT640, Application of Research for the Art ofTeaching, in terms of how valuable the course was to the development

of your teaching ability.”

“The MAT program developed my knowledge, skills, and abilities to reflect on my teaching practice.”

Recommendations from Findings• Learning outcomes for each course should be reviewed for clarity, precision,

relevance, and alignment with program learning outcomes.• Learning outcomes, assignments, and evaluation for each course should be reviewed

for alignment with appropriate program learning outcomes.

0 20 40 60 80 100 120 140 160 180 200 220

MAT 640:Applications of Research for the Art of Teaching

How valuable was this course to the development of your teaching ability?

183 or 88%

26 or 12%

# of Students

209

Not at all Valuable

Not too Valuable

Very Valuable

Somewhat Valuable

Total # respondents to the question:

LOGOMomentum of Improvement

Minimize Resistance to Change Create Successful Pilot Projects “Snow Ball” Effect

Learn from MistakesProvide Continuing Support

Customized HelpRewards and Pressure

Positive Reinforcement Peer Pressure

LOGO

Let’s Recap

LOGOGetting Started

Stop the “Resistance-to-Change” talk Isolate Problems & Prioritize Tasks

1. Assessment Cycle 2. The needed “buy-in”

– Support from Administration» Capability of the IR Office

– Support from Faculty» Understanding Benefit of Assessment

3. Governance bodies and their responsibilities

LOGOGetting Started

Stop the “Resistance-to-Change” talk Isolate Problems & Prioritize Tasks

1. Assessment Cycle 2. The needed “buy-in”

– Support from Administration» Capability of the IR Office

– Support from Faculty» Understanding Benefit of Assessment

3. Governance bodies and their responsibilities

LOGOGetting Started

Stop the “Resistance-to-Change” talk Isolate Problems & Prioritize Tasks

1. Assessment Cycle 2. The needed “buy-in”

– Support from Administration» Capability of the IR Office

– Support from Faculty» Understanding Benefit of Assessment

3. Governance bodies and their responsibilities

DON’T GIVE UP

National University