Noyce Program Evaluation Group

• Our evaluation project has four major components:– Preparation of an extensive literature review

pertaining to recruitment and retention– Thematic synthesis through content analysis of

project information– Statistical analyses to produce quantitative models

of the program development– Execution of an overall program evaluation plan

through collaboration (participatory approach) with existing projects.

http://www.nsf.gov/start.htm

Methodology: Data sources• Surveys

– PI online survey (N=66)– Scholar online survey (N=555)– Disciplinary faculty online survey (N=80)

• Interviews– Scholars in progress (N=12)– Districts in progress (N= 17)

• Monitoring data collected for NSF• Self-response bias

Methodology: Analyses

• Frequencies, cross tabs, means • Open-ended responses categorization• Factor and cluster analyses• Regression and HLM analyses using 3 outcome

variables:– Factor score: Commitment to teaching in high needs school (Influence

of Noyce)

– Would you have become a teacher if you hadn’t received the Noyce scholarship?

– Would you have decided to teach in a high need school if you hadn’t participated in the Noyce scholarship program?

Project overview

75 active Noyce projects with 88% of the projects responding– Of the 9 non-responding PIs, 7 were in their first or second year and

subsequently had no data to report

• Projects in 29 states responded, with 12 responding from Texas

• Total of 141 strands reported with an average 1 or 2 teacher education strands per responding PI– 8 strands was the maximum– Strand breakdown:

• 49 Undergraduate program leading to a bachelor’s degree (34.8%)• 17 Teaching credential (no degree) (12.1%)• 27 Post-bac or graduate program (no master’s awarded) (19.1%)• 27 Graduate program (19.1%)• 21 Other (14.9%)

Funded activities

Miscellaneous Meetings Mentoring

PI Sec II Q 1, number of respondents for each item ranged from 65 to 56.

33%

37%

39%

54%

34%

54%

58%

67%

52%

82%

97%

0.0% 20.0% 40.0% 60.0% 80.0% 100.0%

Mentoring by individuals not in school

Mentoring by experienced teachers

Mentoring by teachers in the same discipline

Mentoring in the schools

Meetings with district personnel and/or teachers

Meetings of the Noyce scholars after the program

Meeting with university or other instructional faculty

Meetings of Noyce scholars during the program

Special educational programs or sessions

Data collected about scholars

Scholarships or stipends

Project overview

• Of the 555 scholars responding to the survey:– 46% were teaching full-time/part-time

– 31% were still in their certification program but not yet a full-time teacher

– 13% were still in their program but also teaching full-time

– 8% completed a program but never taught

– 1% left their program without completing certification

– 1% taught after being certified and were working in education but not as teachers

– 1% taught after being certified but were no longing working in education

Effect of Noyce scholarship on teaching

Would you have become a teacher if you had not received the Noyce Scholarship? Would you have taught in a high needs school if you hadn’t participated in the Noyce program?

Yes Possibly No I will not teach in a high needs school

Scholar Sec IV Q6 (n = 543) & Q7 (n = 542)

3.8%

61.3%

20.0%

15.0%

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

Higher ability Similar ability Lower ability Cannot compare

Ability of STEM majors who intend to teach compared to

those that do not

Fac Q4 (n = 80)

Career changers and high needs teaching

Teaching high needs school Teaching another type of school

Career changers Non-career changers

Scholar Sec III Q1 & Sec V Q4, career changers (n=267), non-career changers (n=265)

Program type and timing of teaching decision

Scholars Sec I Q1 &Sec IV Q 4, Before (n=74), After (n=469)

21%19% 19%

37%

4%

29%

10%

22%

34%

6%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

Undergraduateprogram

Teachingcredential

Post-bac orgrad program(no masters)

Grad program(mastersawarded)

Other

Before After

Influence of Noyce money on potential commitments

Scholar Sec IV Q8, respondents vary from 543 to 534

Education about how to work in high needs schools

Yes to high needs education No to high needs education

High needs student teaching experience

No high needs student teaching experience

Scholar Sec II Q1, high needs experience (n = 163); no high needs experience (n = 380)

Factor analysis results of Noyce scholar surveyConstructed Factor Cronbach’s

AlphaNumber of Items

Commitment to teaching in a high needs school (influence of Noyce)

0.893 6

Preparation for high needs school 0.730 13

Path to teaching 0.722 7

District/school high needs environment 0.716 5

Personal beliefs towards teaching 0.611 8

School teaching environment 0.775 4

Mentoring experience 0.724 6

* Please see the attached laminated sheet for more details.

Influence of funding on timing of teaching decision

21.6%

1.5%

31.1%

14.1%

47.3%

84.4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Before After

Did you first learn about the Noyce scholarship before or after you decide to become a teacher?

Yes

Possible

No

2

Would you have becomea teacher if you had not received the Noyce scholarship?

(2, N=542) = 83.296, p<0.001

(Scholar Sec IV Q4)

(Scholar Sec IV Q6)

School district representatives perceptions of Noyce

* Please see the attached laminated sheet for more details.

District model highlights• “Recruitment pipeline”

– Many participants commented that they valued the Noyce program for bringing a higher number of qualified STEM teachers into their schools.

• Valued qualities of Noyce scholars: sense of purpose and “mission”– Some participants mentioned they perceived a stronger sense of

purpose in Noyce scholars compared to other teachers which they believed positively affected their teaching in an urban setting and commitment to urban education.

• Relationship with Noyce programs– Many participants commented that their relationship with the teacher

preparation programs had not changed or only changed minimally with the Noyce program. The relationship in some cases was “enhanced”, but did not affect the districts significantly.

District interview quotes“Recruitment pipeline”One participant mentioned that without the Noyce program, “It would limit the pool of highly qualified math and science teachers and to me, it would hurt because the pool has been shrinking or has shrunk for math/science teachers. The Noyce scholarship just gives us another avenue of having a pool of qualified candidates to be able to become highly qualified teachers in our schools”

Valued qualities of Noyce Scholars: Sense of purpose and “mission”“I have not met one that isn’t truly, I don’t want to use the word idealistic, but they want to teach and they have made commitments to go into such a school and that shows a lot about the quality of the person.”

Relationship with Noyce ProgramRegarding how Noyce had changed the district’s work with the teacher preparation program, one participant commented, “From our end, not so much, because already [the teacher preparation institution] has a pretty strong commitment or partnership with the district... It’s pretty strong. Already, we get quite a few teachers from that program.”