INSPIRE-G and INSPIRE-PP Results, 2015-2016 and 2016-2017

Dr. Diana PounderDr. Yongmei Ni

UCEA plenum ReportNov 14, 2018Houston, TX

Purposes• Whether educational leadership preparation programs with “High” &

“Low” quality differ on • Program elements • Specific graduate learning outcomes, and • Career plans.

• Whether educational leadership preparation programs with “High” & “Low” quality differ on• specific leadership preparation program features as described by program

coordinators/faculty

Data • INSPIRE Graduate survey (INSPIRE-G) data collected in 2016 and

2017. • Over 1600 graduates, representing 41 leader preparation programs (mostly

UCEA member institutions)

• Merged INSPIRE-G data in 2016 and INSPIRE Preparation Program (INSPIRE-PP) data collected in 2016.• Over 1000 graduates, representing 29 leader preparation programs

INSPIRE-G (graduates’ assessment)

•

Program ElementsFaculty Quality (5-items) (α=.889)Program Rigor & Relevance (5-items) (α=.931)Internship Quality (7-items) (α=.875) Program Accessibility (5-items) (α=.713)Peer Relationships (3-items) (α=.881)

Standards-Based Learning OutcomesProfessional & Organizational Culture (9-items) (α=.952)Supportive & Equitable Learning Environment (4-items) (α=.917) Ethical & Professional Norms (7-items) (α=.928)Strategic Leadership (5-items) (α=.921) Instructional Leadership (9-items) (α=.951)Operations & Management (6-items) (α=.909)Family & Community Engagement (6-items) (α=.950)

Career Plans (5-items)

INSPIRE-PP (by program coordinators)Curricular Emphasis (5-items) (α=.737)Technology Use (12-items) (α=.853) Local Partnership (7-items) (α=.848)Practitioner Roles (8-items) (α=.932)Admission (18-items) (α=.828)Instructional Practices (6-items)– Applied Problem-Solving (4-items)– Interactive Discussions (2-items)

(α=.695)(α=.769)(α=.402)

Internship Mentor(s) (5-items) (α=.946)Internship Characteristics (7-items) (α=.920)Internship Design/Goals (7-items) (α=.965)Knowledge Assessment (4-items) (α=.714)Comprehensive Assessment Practices (4-items) (α=.810)Assessment Formative (10-items) (α=.830)Assessment Principles (3-items) (α=.722)Licensure Assessment – Capstone Project or paper (2-items) (α=.534)Licensure Assessment – Portfolio (1-item)Licensure Assessment – Exam (Comprehensive or Standardized) (2-items) (α=.651)

Method• Question 1—Do programs with “High” & “Low” quality differ in terms of

graduates’ assessment of program features, learning outcomes, and career plans?• Use INSPIRE-G data for 2016 & 2017, computed mean & SD of “Overall

Program Quality” item by institution/program (41 programs)• Comparison groups:

• “HI”-- 5 institutions/programs that were approximately 0.5 SDs ABOVE the mean • “LO”-- 5 institutions/programs that were approximately 0.5 SDs BELOW the mean

• Unit of analysis• Individual graduates. • Sample sizes roughly equal between HI & LO groups. • N varies depending on responses on different items.

• MANOVAs to test for differences between “HI” and “LO” rated programs on • Graduates’ assessments of program features, learning outcomes, and career plans

Program FeaturesNot significant

3.5

3.7

3.9

4.1

4.3

4.5

4.7

4.9

faculty quality program rigor &relevance

internship quality program accessibility peer relationships

LO HIN=114 for LO; N=118 for HIAll differences are statistically significant between LO and HI groups

Learning outcomes

3.5

3.7

3.9

4.1

4.3

4.5

4.7

4.9

Prof. & org.culture

Supp. & equ.learning env.

Ethical & prof.norms

strategicleadership

instructionalleadership

operations &management

family & comm.engagement

LO HI

• N=130 for LO; N=126 for HI

• All differences are statistically significant b/w LO & HI groups

Career plans – likelihood of pursuing leadership career

3.5

3.6

3.7

3.8

3.9

4.0

4.1

4.2

4.3

4.4

4.5

LO HI

• N=46 for LO; N=132 for HI

• The difference is statistically significant b/w LO & HI groups

Summary of findings (INSPIRE –G)• Program Elements: There are statistically significant differences between “higher” versus

“lower” overall rated programs on all program elements EXCEPT “peer relationships”. • Higher rated programs have significantly higher ratings on faculty quality, program rigor and relevance,

internship quality, and even program accessibility. • Only peer relationships was not significantly reflective of overall program quality ratings.

• Learning Outcomes: There are statistically significant differences between “higher” versus “lower” overall rated programs on all specific standards-based learning outcomes. • Including learning in professional and organizational culture, supportive and equitable learning

environment, ethical and professional norms, strategic leadership for school improvement, instructional leadership, operations and management, and family and community engagement.

• Career Plans: Graduates from the “higher” quality programs had stronger intentions or plans to become school leaders/administrators than did those from “lower” quality programs.• However, due to the disparate size of the two groups, these results may be less reliable than that of the

previous analyses • All program element variables met the assumption of homogeneity of variance for MANOVA

analyses between groups EXCEPT that of “peer relationships” and “ethical and professional norms”.

Method• Question 2– Do programs with “High” & “Low” quality differ on specific

program features as described by program coordinators/faculty?• Merged data of INSPIRE-G (2016) and INSPIRE-PP (2016)• 1000+ graduates representing 29 leader preparation programs• Use INSPIRE-G data for 2016, computed mean & SD of “Overall Program

Quality” item by institution/program (29 programs)• Comparison groups:• “HI”-- 4 institutions/programs that were approximately 0.5 SDs ABOVE the

mean • “LO”-- 5 institutions/programs that were approximately 0.5 SDs BELOW the

mean• Test the differences b/w “HI” and “LO” programs on program features• Descriptive –compare each program feature b/w Hi and LO at program level• MANOVAs –graduate level (75 in “LO” + 98 in “HI”).

Program Features LO (N=4) HI (N=5) DifferenceCurricular Emphasis 3.96 4.48 0.52Technology Use 2.69 3.23 0.55Practitioner Roles 3.09 3.74 0.65Admission 3.14 3.62 0.48Instructional Practices 3.29 3.61 0.32– Applied Problem-Solving 3.13 3.86 0.73– Interactive Discussions 4.50 4.61 0.11Internship Mentor(s) 2.35 3.82 1.47Internship Characteristics 3.86 4.21 0.36Internship Design/Goals 2.62 2.47 -0.15Knowledge Assessment 2.75 3.55 0.80Comprehensive Assessment Practices 3.42 4.19 0.78Assessment Formative 3.40 3.56 0.16Assessment Principles 3.75 4.41 0.66Licensure Assessment – Capstone Project or paper 2.25 2.09 -0.16Licensure Assessment – Portfolio 4.00 3.40 -0.60Licensure Assessment – Exam 1.88 2.98 1.11

Program features (MANOVA)

2

2.5

3

3.5

4

4.5

5

Curricular Emphasis Technology Use Practitioner Roles Admission

LO HI

Not significant

Instructional practice (MANOVA)

2

2.5

3

3.5

4

4.5

5

Instructional Practices – Applied Problem-Solving – Interactive Discussions

LO HI

Internship (MANOVA)

2

2.5

3

3.5

4

4.5

5

Internship Mentor(s) Internship Characteristics Internship Design/Goals

LO HI

Assessment (MANOVA)

1

1.5

2

2.5

3

3.5

4

4.5

5

KnowledgeAssessment

Comp.Assessment

Practices

AssessmentFormative

AssessmentPrinciples

Lic. Assess. –Capstone Project

or paper

Lic. Assess. –Portfolio

Lic. Assess. –Exam

LO HI

Summary of findings (INSPIRE-PP)• There are statistically significant differences between “higher” versus “lower” overall

rated programs on all program elements EXCEPT the Admissions program feature scale.

• Higher quality programs had more favorable features in the areas of curricular emphasis, technology use, practitioner roles, and applied problem-solving instructional strategies.

• Internship experiences--higher quality programs had more favorable features in the areas of internship mentoring, internship characteristics, and internship design/goals.

• Assessment practices-- higher quality programs had more favorable features in student knowledge assessment, comprehensive assessment practices, assessment principles, licensure assessment – capstone project or paper, and licensure assessment – comprehensive exam or standardized exam.

• By contrast, lower quality programs had more favorable features in the areas of interactive discussions as an instructional strategy, formative assessment practices, and licensure assessment– portfolio.