academic program assessment report - kean universityncate/caep accreditation 2017/slo... ·...

of 20

Academic Program Assessment Report INSTRUCTIONS: Page 1 of this document serves as the program’s annual assessment plan. Please complete page 1 by October 31, 2015. Pages 2-3 serve as the program’s annual assessment report. Please complete pages 2-3 by June 30, 2016.

COLLEGE: College of Education

ENTER PROGRAM NAME: Elementary Education Initial Certification (K-6, K-6/5-8)

ACADEMIC YEAR: 2015-2016

REPORT AUTHOR: Gail Verdi

PROGRAM STUDENT LEARNING OUTCOMES (CHECK OFF THE SLOs BEING ASSESSED):

☐ SLO1: Knowledge (KU 2, 4; GE S4) -- Candidate has understanding and knowledge of subject matter, and of national and New Jersey state standards.

☐ SLO2: Knowledge (KU 2, 4; GE S4, V4) -- Candidate demonstrates knowledge of how elementary students learn and develop.

☐ SLO3: Skills (KU 2, 4; GE S4) -- Candidate plans instruction based on knowledge of subject matter, and of national (CCSS), professional and New Jersey Core Curriculum Content Standards.

☐ SLO4: Skills (KU1, 4; GE S4, V4) -- Candidate uses formal and informal assessment strategies to measure student progress and adjust instruction as needed.

☐ SLO5: Dispositions (KU1, 2, 4; GE K2, S4, V4) – Candidate appreciates individual, cultural, and linguistic differences.

☐ SLO6: Dispositions (KU 2, 4; GE S@, S4, V2, V4) – Candidate takes responsibility for establishing a positive climate in the classroom. DIRECT MEASURE:

☐ SLO1: Knowledge (KU 2, 4; GE S4) -- Candidate has understanding and knowledge of subject matter, and of national and New Jersey state standards.

☐ SLO2: Knowledge (KU 2, 4; GE S4, V4) -- Candidate demonstrates knowledge of how elementary students learn and develop. Praxis II In order for K-6 Elementary Education teacher candidates to qualify for their professional internship semester (student teaching), they must take and pass the Praxis II: Multiple Subjects Test (5001) designed by ETS (http://www.ets.org/s/praxis/pdf/5001.pdf)/. This standardized test measures teacher candidates’ content knowledge of Language Arts, Mathematics, Social Studies, and Sciences, as well as their understanding of how students at different grade levels acquire and develop specific skills. For example, when reviewing descriptors for foundational skills in reading, it is noted on page 6 of the Praxis Study Companion (see link above) that teacher candidates “must understand the role of phonological awareness in literacy development” as well as “identify and provide examples of phonemes, syllables, onsets, and rimes.” Therefore, this assessment not only assesses teacher candidates understanding and knowledge of subject matter linked to national and NJ State standards, but that they also have to demonstrate how

http://www.ets.org/s/praxis/pdf/5001.pdf)/

of 20

students learn and develop these forms and knowledge and skills over time. The Praxis II: Multiple Subject test is a requirement of the New Jersey Department of Education for all Elementary Education K-6 initial teacher candidates. TARGET:

Minimal scores required for initial teacher certification candidates in the state of New Jersey are as follows:

Content Test Test Code Minimal Passing Score

1. Reading and Language Arts Subtest (5002) 157

2. Mathematics (5003) 157

3. Social Studies (5004) 155

4. Science (5005) 159

Please note that these are minimal scores required as of November 9, 2015. The cut off scores may be revised during a given academic year. Therefore, scores may change in subsequent reports

DATA COLLECTION AND RESULTS:

Over the 2012-2013, 2013-2014, 2014-2015 academic years, ETS revised the Elementary Education Praxis II exam three

times. Below you will find Table 1, listing a breakdown of the years each exam was counted towards NJ Elementary

Education K-6 certification, the passing scores for the three different versions of the test, and the years that data for the

each exam were collected:

Elementary Education Content Knowledge Test (0014/5014)

Elementary Education Multiple Subject Tests (5031-combined) a. Language Arts (5032) b. Math (5033) c. Social Studies (5034) d. Science (5035)

Elementary Education Multiple Subject Test Revised LA & Math (5001 combined) e. Language Arts (5002) f. Math (5003) g. Social Studies (5004) h. Science (5005)

of 20

Table 1.

Academic Years Test Administered 2012-2013 2013-2014 2014-2015

Elementary Education Content Knowledge (0014/5014) Passing score = 141

I - Reading/Language Arts X X X

II - Mathematics X X X

III - Social Studies X X X

IV - Science X X X

Elementary Education Multiple Subject Language Arts (5032) Passing Score = 165

I - Reading X X X

II - Language Writing & Communication X X X

Elementary Education Multiple Subject Mathematics (5033) Passing Score = 164

I - Number Operations and Algebraic Thinking X X X

II - Geometry, Measure Data and Interpretation X X X

Elementary Education Multiple Subject Social Studies (5034) Passing Score = 155

I - US History, Government and Citizenship X X X

II - Geography, Anthropology and Sociology X X X

III - World History and Economics X X X

Elementary Education Multiple Subject Science (5035) Passing Score = 159

I - Earth Science X X X

II - Life Science X X X

III - Physical Science X X X

Elementary Education Multiple Subject Language Arts Subset (5002) Passing Score = 167

I - Reading X

of 20

II - Language Writing & Communication X

Elementary Education Multiple Subject Mathematics Subset (5003) Passing Score = 167

I - Numbers and Operations X

II - Algebraic Thinking X

III - Geometry, Measure Data and Interpretation X

Elementary Education Multiple Subject Social Studies Subset (5004) Passing Score = 155

I - US History , Government and Citizenship X

II - Geography, Anthropology and Sociology X

III - World History and Economics X

Elementary Education Multiple Subject Science Subset (5005) Passing Score = 159

I - Earth Science X

II - Life Science X

III - Physical Science X

Narrative Discussion of Changes Across Versions of Elementary Education Praxis II Exams

1.1 Elementary Education Content Knowledge Praxis II (0014/5014): The passing score for both Kean University and

The State of New Jersey for this version of the test is 141. This test, like the later versions of Multiple Subject Exam, was

designed for prospective teachers of children in primary through upper elementary school grades. The 120 multiple-

choice questions focused on four major subject areas: language arts/reading, mathematics, social studies, and science.

There are 30 questions for each subject area. Test questions were arranged in the test book by subject area. Each of the

four content areas constitutes 25 percent of the test. This test is no longer being administered because it was found that

the composite score of 141 did not require teacher candidates to exhibit proficiency in all four content areas. In other

words, you could score high on the language arts, social, studies, and science portions of the exam, but miss a majority

of math questions and still pass the test by receiving a combined score of 141. This would mean that elementary

teachers would be certified even though they lacked the appropriate background knowledge in one or more of the four

key content areas assessed through the Elementary Education Content Knowledge Praxis II (0014/5014):

Praxis II Content Categories No. of Questions % of Exam

Reading/Language Arts 30 25%

Mathematics 30 25%

of 20

Social Studies 30 25%

Science 30 25%

1.2 Elementary Education Multiple Subject Praxis II Subtests (5031 Series/5032, 5033, 5034, 5035) were instituted in

the 2012-2013 academic year: The passing scores for both Kean University and the State of New Jersey for this version

of the test were now disaggregated by content area (subtests) to resolve the issues found in the previous Content

Knowledge Praxis II:

Content Area Passing Score Code #Qs % of Exam

Language Arts 165 (5032) 65 100%

Math 164 (5033) 40 100%

Social Studies 155 (5034) 55 100%

Science 169 (5035) 50 100%

The major shift in the Praxis II 5031 Series test construct was that it required teacher candidates to pass each section

separately to qualify for state certification. However, ETS now made it easier for universities and employers to review

data that indicated the specific strengths and weaknesses under subcategories of a content area by cohort. Table 1

above shows how each content area section of the Praxis II tests specific skills. For example, under Language Arts (5032)

teacher candidates are tested on their knowledge of reading as well as language, writing, and communication. When

looking at the actual data, we can see the scoring trends of a university cohort under subcategories of the Language Arts

(5032) subtest.

1.3 Elementary Education Multiple Subject Praxis II Exams (5001 Series/5002, 5003, 5004, 5005) were instituted in the

2014-2015 academic year: For the third and most recent version of the Elementary Education Praxis II, passing scores

for both Kean University and the State of New Jersey for this version of the test continued to be disaggregated by

content area (subtests) to resolve the issues found in the Content Knowledge Praxis II (0014/5014) and continued to

include the same subcategories of knowledge under each subtest, According to a 2014 Multistate Standard-Setting

Technical Report issued by Educational Testing Services regarding the Language Arts and Math Subtests, the Language

Arts and Math subtests were revised to reflect the new National Common Core Standards; therefore, the number of

items of each subtest for LA and Math were increased and the scores on both these tests were lowered from 165 to 157

for Language Arts & from 164 to 157 for Math to reflect the revisions made for this purpose. The Social Studies and the

Science subtests scores were not changed:

of 20

Content Area Passing Score Code #Qs 5

Language Arts 157 (5002) 80

Math 157 (5003) 50

Social Studies 155 (5004) 55

Science 169 (5005) 50

1. Alignment with national ACEI Standards

Table 2

Academic Years 2012-2013 2013-2014 2014-2015 ACEI Standards Qs Percent Qs Percent Qs Percent

Elementary Education Content Knowledge (0015/5014) Passing score = 141

I Language Arts 30 25% 2.1

II - Mathematics 30 25% 2.3

III - Social Studies 30 25% 2.4

IV - Science 30 25% 2.2


I - Reading 32 49% 2.1

II – Lang. Writing & Communication

33 51% 2.1


I – Number Operations and Algebraic Thinking

26 65% 2.3

II - Geometry, Measure Data and Interpretation

14 35% 2.3


I - US History, Govern. & Citizenship

25 45% 2.4

II - Geography, Anthrop. and Sociology

16 30% 2.4

III - World History and Economics

14 25% 2.4

Elementary Education Multiple Subject Science (5035) Passing Score = 159

I - Earth Science 16 32% 2.2

II - Life Science 17 34% 2.2

III - Physical Science 17 34% 2.2


I - Reading 38 47% 2.1

II - Language Writing & Communication

42 53% 2.1


I - Numbers and 20 40% 2.3

of 20

Operations

II - Algebraic Thinking 15 30% 2.3

III - Geometry, Measure Data and Interpretation

15 30% 2.3


I - US History , Govern. Citizenship

25 45% 2.4

II - Geography, Anthrop. and Sociology

16 30% 2.4

III - World History and Economics

14 25% 2.4

Elementary Education Multiple Subject Science Subset (5005) Passing Score = 159

I - Earth Science 16 32% 2.2

II - Life Science 17 34% 2.2

III—Physical Science 17 34% 2.2

As indicated in Table 2 above, all three of the Praxis II tests: Praxis II Elementary Content Knowledge (0014/5014), Praxis

II Elementary Education Multiple Subjects (Series 5031), and Revised Praxis II Elementary Education Multiple Subjects

(Series 5001) are aligned with the Association of Childhood Education International (ACEI) Specialized Professional

Association (SPA) standards.

Content Categories ACEI Standards

Reading/Language Arts 2.1

Mathematics 2.3

Social Science 2.4

Science 2.2

Similarly, the subcategories of the 4 content areas have remained consistent across the three tests:

Language Arts covers Foundations of Reading, Language in Writing, Communication Skills (Speaking,

Listening, and Viewing).

Mathematics covers Mathematical Processes, Number Sense and Numeration, Algebraic Concepts,

Informal Geometry and Measurement, and Data Organization and Interpretation.

Social Studies covers Geography, Anthropology, Sociology, World History, United States History,

Government, Citizenship, and Economics

Science covers Earth Science, Life Science, and Physical Science

However, what did change drastically was the shift in test construction from one test (0014/5014) to 4

separate tests (5031) of equal weight, to a realignment of the Language Arts and Math tests (5002) and (5003)

to the National Common Core Standards.

of 20

DISCUSSION OF FINDINGS

2. Brief analysis of data findings A review of the Praxis II data from 2012-2015 indicates that our candidates achieved a 100% pass rate for all three

versions of the Praxis II Elementary Education Exams. Therefore, it can be argued that our candidates possess the

overall content knowledge to score well on the Praxis II and that currently we are well placed in terms of our passing

rate when compared to state and national averages. The results reported in Table 3 below focus on the number of

correct answers on each of the subtests of each version of the Praxis II

Table 3: Test Results 2012-2015 - Percent of Correct Answers

Test Results (Percent Correct Answers) National New Jersey Kean University Elementary Education Content Knowledge (0014/5014) Passing score = 141 (2012-2015) N=143 I - Reading/Language Arts 77 68 75 II - Mathematics 75 66 74 III - Social Studies 64 57 68 IV - Science 70 61 72 Elementary Education Multiple Subject Reading Arts (5032) Passing Score = 165 (2012-2015) N=95 I - Reading 76 71 80 II - Language Writing & Communication 78 74 82 Elementary Education Multiple Subject Mathematics (5033) Passing Score = 164 (2012-2015) N=95 I - Number Operations and Algebraic Thinking 73 70 79 II - Geometry, Measure Data and Interpretation 67 63 75 Elementary Education Multiple Subject Social Studies (5034) Passing Score = 155 (2012-2015) N=95 I - US History, Government and Citizenship 66 61 69 II - Geography, Anthropology and Sociology 71 67 75 III - World History and Economics 68 61 67 Elementary Education Multiple Subject Science (5035) Passing Score = 169 (2012-2015) N=95 I - Earth Science 66 62 62 II - Life Science 76 73 77 III - Physical Science 58 54 54 Elementary Education Multiple Subject Language Arts (5002) Passing Score = 167 (2014-2015) N=3 I - Reading 67 64 68 II - Language Writing & Communication 65 61 64 Elementary Education Multiple Subject Mathematics (5003) Passing Score = 167 (2014-2015) N=3 I - Numbers and Operations 74 68 75 II - Algebraic Thinking 63 60 63 III - Geometry, Measure Data and Interpretation 68 65 69 Elementary Education Multiple Subject Social Studies (5004) Passing Score = 155 (2014-2015) N=3 I - US History , Government and Citizenship 60 55 75 II - Geography, Anthropology and Sociology 70 66 57 III - World History and Economics 57 54 90 Elementary Education Multiple Subject Science (5005) Passing Score = 159 (2014-2015) N=3 I - Earth Science 67 65 68 II - Life Science 76 75 69 III - Physical Science 61 59 72

of 20

Comparison of Number of Correct Answers from National, State, and Kean University Data 2012-2015

As noted in Table 3 above, the aggregated data outlines the number of teacher candidates from Kean Union, Kean

Ocean County, and Kean’s Post Bac Program that took one of the following three tests: For 014 the total number of test

takers is 143, for the 5031 series the total is 95, and for 5001 series the total is 3. Given the extremely low number of

test results from the 5001 series, the data from the newest version of the Praxis II Multiple Subjects test is the least

reliable, but when we compare data from the 5031 series and the 5001 we can see that Kean’s teacher candidates

remain on par with national averages and score higher on the majority of the subtests than their New Jersey state peers.

Language Arts Data (Number of Correct Answers) Across Exams (014, 5032, and 5002)

In 2009, when comparing the number of correct answers on the 014 Language Arts subtest to National data, it

was reported that 5% of Kean teacher candidates scored below the national average. In the years 2012-2015 only 2% of

teacher candidates taking the 014 Language Arts subtest scored below the National average and 10% of Kean teacher

candidates scored above the New Jersey State average. Therefore, Kean teacher candidates test score results on this

portion of the 014 Praxis II exam rose 3% when compared to the nation and state data showed that our students held a

significant lead in results when compared to their counterparts at other New Jersey Teacher Education Programs.

When reviewing the data from 5032 Language Arts subtest, the data shows that Kean graduates scored 5%

higher than the national average and scored 12% higher compared peers at other higher education institutions in New

Jersey. Similarly, the data from the 5002 Language Arts subtest demonstrate that our Kean University students

outperform their state cohort (6% higher) and are on par with the national cohort.

Math Data (Number of Correct Answers) Across Exams (014, 5032, and 5002)

In 2009, when comparing scores on the 014 Math subtest to National data, it was reported that 5% of Kean

teacher candidates scored below the national average. In the years 2012-2015 only 1% of teacher candidates taking the

014 Math subtest scored below the National average and 12% of Kean teacher candidates scored above the New Jersey

State average. Therefore, Kean teacher candidates test score results on this portion of the 014 Praxis II exam rose 4%

when compared to the nation and state data showed that our students held a significant lead in results when compared

to their counterparts at other New Jersey Teacher Education Programs.

When reviewing the data from 5032 Math subtest, the data shows that Kean graduates scored 10% higher than

the national average and scored 1% higher compared peers at other higher education institutions in New Jersey.

Similarly, the data from the 5002 Math subtest demonstrate that our Kean University students outperform their state

cohort (7% higher) and are on par with the national cohort.

Social Studies Data (Number of Correct Answers) Across Exams (014, 5032, and 5002)

of 20

In 2009, when comparing scores on the 014 Social Studies subtest to National data, it was reported that 6% of

Kean teacher candidates scored below the national average. In the years 2012-2015 only 6% of teacher candidates

taking the 014 Social Studies subtest scored above the National average and 19% of Kean teacher candidates scored

above the New Jersey State average. Therefore, Kean teacher candidates test score results on this portion of the 014

Praxis II exam rose 12% when compared to the nation and state data showed that our students held a significant lead in

results when compared to their counterparts at other New Jersey Teacher Education Programs.

When reviewing the data from 5032 Social Studies subtest, the data shows that Kean graduates scored 2%

higher than the national average and scored 11% higher compared peers at other higher education institutions in New

Jersey. Similarly, the data from the 5002 Social Studies subtest demonstrate that our Kean University students

outperform their state cohort and are on par with the national cohort - except in one instance. The three students that

took the 5002 Social Studies subtest scored 22% lower than the national average and 15% lower than the state cohort

on the subcategory of Geography, Anthropology and Sociology.

Science Data (Number of Correct Answers) Across Exams (014, 5032, and 5002)

In 2009, when comparing scores on the 014 Science subtest to National data, it was reported that 6% of Kean

teacher candidates scored below the national average. In the years 2012-2015 2% of Kean teacher candidates taking the

014 Math subtest scored above the National average and 18% of Kean teacher candidates scored above the New Jersey

State average. Therefore, Kean teacher candidates test score results on this portion of the 014 Praxis II exam rose 8%

when compared to the nation and state data showed that our students held a significant lead in results when compared

to their counterparts at other New Jersey Teacher Education Programs.

When reviewing the data from 5032 Science subtest, the data shows that Kean graduates scored 4% lower than

the national average and scored 1% higher compared peers at other higher education institutions in New Jersey.

Similarly, the data from the 5002 Science subtest demonstrate that our Kean University students outperform their state

cohort (4% higher) and are on par with the national cohort.

CLOSING THE LOOP

3. Interpretation of how data provides evidence for meeting ACEI standards: The consistent pass rate of 100% indicates the strengths of the Elementary Education Program and our candidates to

provide evidence of their ability to meet ACEI Standards. These results are indicative of our candidates’ ability to

perform well on a nationally standardized exam. While the data shows our students possess the content knowledge to

score well on the Praxis II, this is not a performance-based assessment of the candidates’ abilities to demonstrate

mastery of the integration of this knowledge with teaching or candidates’ professional attitudes. It also shows that we

have steadily improved as result of steps taken, e.g., more opportunities on-campus for Praxis preparation, an increased

focus on content standards have impacted the scores of Kean’s teacher candidates. What this doesn’t show us,

however, are the scores of students that have not passed the Praxis II exam. This is important data for the NJDOE to

supply to colleges of education across the state so that we can realistically accommodate the needs of students that do

not meet the academic requirements of their teacher education programs. The pass rate is 100% because students do

of 20

not become program completers if they do not do so. This ultimately skews the data and denies us from getting a truly

clear picture of how our students are doing on these exams. Questions arise here such as: What do we do to assist our

teacher education candidates that do not pass one of more sections of the Praxis II? Faculty need to be involved in

analyzing the types of thinking and the forms of knowledge teacher candidates need to have acquired to pass the Praxis

II Multiple Subjects Exam. For example, what can be done to increase students knowledge of the Geographical,

Anthropological, and Sociological Concepts need to increase scores on that section of the Social Studies exam? One

place Kean’s College of Education has begun to explore answers to these questions is by having faculty take the exams

their students must take in order to become teachers. This has helped us begin to take part in valuable dialogues

around how teacher testing and evaluation impede or inform new teacher preparation programs.

SUPPORTING EVIDENCE: SEE ATTACHMENT A

of 20

DIRECT MEASURE:

☐ SLO3: Skills (KU 2, 4; GE S4) -- Candidate plans instruction based on knowledge of subject matter, and of national (Common Core for Math and Language Arts), professional and New Jersey Core Curriculum Content Standards.

TWS

In Professional Internship (Student Teaching), a candidate’s instructional planning based on subject matter and national (Common Core) and NJDOE Standards is assessed using the COE Teacher Work Sample (TWS) Assessment Rubric. Course instructors and clinical supervisors use the TWS Rubric to measure each teacher candidate’s ability to develop appropriate learning goals for a given grade level (including differentiation), construct an assessment plan to determine if students have met goals, design an instructional unit based on those learning goals. Therefore, I am going to focus on three subsections of the Teacher Work Sample Assessment Rubric to analyze data from Direct Measure SLO3 listed above. The two subsections I will focus on for the 2015-2016 academic year are:

1. Learning Goals: The teacher candidate sets significant, challenging, varied, and appropriate learning goals

2. Assessment Plan: The teacher candidate uses multiple assessment modes and approaches aligned with learning goals to assess student learning before, during, and after instruction.

3. Design for Instruction: The teacher candidate designs instruction for specific learning goals, student characteristics and needs, and learning contexts.

The rubric consists of 10 components/criteria (Intro to Portfolio, Philosophy Statement, Contextual Factors, Learning Goals, Assessment Plan, Design for Instruction, Instructional Decision Making, Analysis of Student Learning, Reflection and Self-Evaluation and Writing Mechanics and Appearance). Each component includes 3-4 descriptors to assist course instructors and clinical supervisors in providing detailed feedback to our Professional Interns as they work towards becoming highly qualified teachers.

TARGET:

***For the 2014-2015 Elementary Education Annual Academic Program Assessment Report. I used the following criteria to determine whether teacher candidates met minimum competency on a dispositional measure based on the total score:

A candidate must meet a minimum score between 40-44 (Capable) to be recommended for K-6 Initial certification in the state of New Jersey. Since this is a culminating experience of all K-6 teacher candidates, it is expected that 100% of Professional Intern Candidates meet the minimal competencies outlined in the TWS Assessment Rubric. If a Professional Intern fails to meet the minimal criteria (Capable), a remediation plan is outlined by the cooperating teacher, the clinical supervisor, the Senior Seminar course instructor, and the Director of the Teaching Performance Center. Generally, a Professional Intern in need of a remediation plan will be expected to hold back graduation, and to repeat the Professional Internship experience with further support and guidance.

of 20

However, this year I plan to review individual components of the TWS since I believe that select component focus on a various SLOS. Therefore, for 2015-2016 Elementary Education Annual Academic Program Assessment Report I plan to look at data for:

1. Learning Goals: The teacher candidate sets significant, challenging, varied, and appropriate learning goals

2. Assessment Plan: The teacher candidate uses multiple assessment modes and approaches aligned with learning goals to assess student learning before, during, and after instruction.

3. Design for Instruction: The teacher candidate designs instruction for specific learning goals, student characteristics and needs, and learning contexts.

This data will specifically align with SLO 3 listed above. This data will be assessed on a 5 point scale for each item listed above. The criteria are as follows:

A description of how this assessment specifically aligns with the standards

The Level III Teacher Work Sample Portfolio is aligned with both the College of Education Learning Goals based on

the SPECTRUM Model (Knowledge, Skills and Dispositions) and the ACEI standards. The alignment of the

components of the TWS and the College of Education Learning Goals is reflected in the document on page 28 of A

Guide for Compiling a Teacher Work Sample Portfolio titled “Integration of Teacher Work Sample Process and COE

Learning Goals: Teaching Processes Assessed by the Kean COE Teacher Work Sample. The ACEI standards are

indicated in the rubrics used to evaluate each component of the TWS.

For example, the Philosophy Statement aligns with ACEI Standard 1 that requires candidates to know and

understand major concepts, principles and theories related to the development of children and adolescents. The

Contextual Factors assignment spans two ACEI standards 3 and 5: Standards 3.2 and 3.3 requiring clinical interns to

make adaptations for diverse learners and to understand the need to use a variety of teaching strategies to support

learning and to make accommodations for students with specific learning needs. Standard 5.2 require clinical

interns to demonstrate an understanding of the needs of the community they work in and to develop opportunities

1 = Unacceptable

(Not Competent)

Teacher candidate

demonstrates little

or no competence.

2 = Beginning

(Beginning Competence)

Teacher candidate

demonstrates competence

with significant assistance

and prompting.

3 = Developing

(Developing Competence)

Teacher candidate

demonstrates developing

competence with some

assistance and

prompting.

4 = Capable

(Competent)

Teacher candidate

consistently

demonstrates

competence without any

assistance or prompting.

5 = Accomplished

(Highly Competent)

Teacher candidate

consistently demonstrates a

high degree of competence

functioning independently.

of 20

to engage parents, colleagues and community members in conversations about teaching and learning. The

Contextual Factors piece helps candidates to compile information they can use to create communication plans and

to help parents and community members become partners in students’ education.

The Learning Goals and Assessment Plan supports ACEI standards 2.1 to 2.7 by aligning learning goals and

assessment plans for tasks to specific content area standards or by integrating cross-curricula units of study and ACEI

standard 4.0 by designing both formal and informal assessments to collect and analyze data from pre and post

assessments and formative assessments.

Similarly, the Design for Instruction covers ACEI standards 2.1-2.7 and 3.1, 3.2, and 3.4. In developing units of

instruction, clinical interns must decide what content area or cross-curricular thematic unit they will teach. In doing

this, the clinical interns will cover one or more of the ACEI standards that focus on curricular design. In addition, this

task invites clinical interns to explore a variety of instructional techniques, therefore, utilizing ACEI standard 3.1

(Integrating and applying knowledge of instruction) and ACEI standard 3.3 (Developing critical thinking and problem

solving). When clinical interns complete the section of the TWS on Instructional Decision-Making, they must

consider the types of modifications they might need to make for their original Design for Instruction based on the

background knowledge students bring to the lessons (ACEI standard 5.1), the need for individualizing instruction

based on findings (ACEI standard 3.2), and the units alignment with learning goals and standards (ACEI standards

1.0, 3.2, & 5.1).

Through the Analysis of Student Learning clinical interns continue to use assessment to plan and evaluate

instructions (ACEI 4.0) and through the Reflection and Self-Evaluation Task they cover ACEI standard 5.1

(Candidates are aware of and reflect on their practice) and continue to explore standards 2.1-2.7 (Curriculum design)

by rethinking the methods and materials used and consider alternative actions that might lead to more probable

success for student learning.

of 20

DATA COLLECTION AND RESULTS

3. Brief Analysis of Data Findings

Section I

The data set focusing on clinical interns’ performance during Level III Teacher Work Sample was collected using The

Teacher Work Sample (TWS) Portfolio Assessment Form (attached). The following data table (see Teacher Work

Sample Portfolio Assessment Data attached) were designed by the Teacher Performance Center to summarize

findings collected over four semesters (spring 2014 to fall 2015):

Scores comparing competency levels from both the University Supervisor (EMSE 4800 Professional Internship) and the Course Instructor for Senior Seminar (EMSE 4900)

Scores comparing undergraduate (UG) and post-baccalaureate (PB) clinical interns competency levels Scores comparing competency levels from the Union Campus (Union) and Ocean County Campus (OCC)

Each table includes the ten Process Indicators described in sections 1 and 2 of this report as well as the number of

candidates and competency levels from each subset listed above.

Section II

In addition, during the most recent program review cycle, the elementary education program and the Teaching

Performance Center collected a second source of data for assessment 5 to determine the impact instruction by

clinical interns had on students’ learning. Therefore, for each semester (spring 2015 to fall 2015) data was collected

on the following:

Instructional Unit Number of lessons

taught in the Unit

PreTest Average

Class Score

(Percent Correct)

PostTest Average

Class Score

(Percent Correct)

Percent Increase

(Decrease) in P-12

Student Learning

However, since both data sets (TWS and Student Learning Pretest/Postest Comparisons) are collected via cohorts

and there is no access to individual clinical intern data, we can not determine if there is a correlation between

specific scores on the TWS and the success or failure to produce higher posttest scores by any individual clinical

intern.

of 20

4. DISCUSSION OF FINDINGS: Interpretation of the data and how it provides evidence of meeting standards

Section I

As noted in sections 1 & 2 of this report, the ten process indicators listed in The TWS Portfolio Assessment are closely

aligned with the ACEI standards. Therefore, the links between the indicators, descriptors and the ACEI standards

show that the data collected represent high levels of content validity. The cumulative scores submitted by

instructors and supervisors for all subsets described above (UG, PB, Union, OCC) from the spring of 2014 to the fall

of 2015 indicate that majority of competency levels ranged from 4 (Capable) to 5 (Accomplished) for individual

Process Indicators 1-10 with a small percentage of undergraduate clinical interns from both the Union and Ocean

Campuses being scored 3 (Developing). No clinical interns in the Post Baccalaureate Program received a score lower

than a 4 (Capable).

However, upon review of the data across semesters (spring 2014 to fall 2015) there are no trends in specific areas

that need improvement for the majority of undergraduate clinical interns at either the Union or OCC Campuses. For

example, the data doesn’t show that the number of students scored a 3 (Developing) on the Introduction to

Portfolio in the Spring 2014 remained constant over time (to fall 2015). If it had, this would have informed the

faculty that more emphasis would have to be placed on how to write an Introduction in the EMSE 4900 course

during class workshops. Although individual students, may need more support when constructing the TWS, the

cohort as a whole performed at a level consistent with a highly qualified clinical intern.

Sample of Undergraduate Program Data from the Union Campus

Below you will find a sampling of data for two components – Assessment Plan and Design for Instruction –

submitted by the EMSE 4900 Course Instructors and the University Supervisors—for the Undergraduate Program on

the Union Campus from spring 2014 to fall

2015:

Process/Component S14

Instr

S14

Super

F14

Instr

F14

Super

S15

Instr

S15

Super

F15

Instr

F15

Super

Assessment Plan 2

5%

5

11%

0 2

5%

0 1

3%

0 0

Design for

Instruction

1

3%

1

2%

0 2

5%

0 0 0 0

of 20

As indicated earlier in this report, university supervisors have a tendency to score the TWS a bit lower than the EMSE

4900 Course Instructor, but this is not significant in the overall assessment of individual cohorts over time. When we

look at the data across semesters, there actually appears to be an improvement in scoring over time. However, this

may be misleading and depend more on variables that may not be determined through this data such as classroom

observations and input from the cooperating teacher that the Course Instructor may not have access to. In addition,

the Course Instructor spends most of his/her time assisting the clinical interns in drafting and organizing the TWS

Portfolio where the University Supervisor’s evaluation of the TWS may be colored by classroom observations of the

clinical intern. Course Instructors tend to view themselves more as advocates for the clinical interns while the

University Supervisor may perceive their role as more of a judge. Since it is difficult to truly determine what is going

on in the minds of the Course Instructor and the University Supervisor, it might be more helpful if we can follow a

few individual students over time to get a sense of how they are progressing and deepening their skills as they move

through the program. It is our hope that the new assessment system, EdTPA, that the NJDOE will be requiring for

clinical interns to complete in order to get certified will help close the gap in our methods of gathering data so that

we can get a better sense of how to measure an individual clinical interns impact on student learning.

Sample of Undergraduate Program Data at the Union and OCC Campuses

Below you will find a sampling of data for two components – Assessment Plan and Design for Instruction –

submitted by University Supervisors—for the Undergraduate Programs on both the Union and OCC Campuses from

spring 2014 to fall

2015:

Process/Component S14

Union

S14

OCC

F14

Union

F14

OCC

S15

Union

S15

OCC

F15

Union

F15

OCC

Assessment Plan 4

9%

1

10%

1

3%

1

10%

1

3%

0 0 0

Design for

Instruction

1

2%

0 1

3%

1

10%

0 0 0 0

As you can see, there are semesters when clinical interns at both campuses receive 3 (Developing) scores on the

Assessment Plan and Design for Instruction components. Given that the data shifts to 0% at both campuses in the

Fall of 2015 might indicate that there is a trend towards students being more prepared for clinical internship prior to

entering their student teaching semester.

of 20

These findings make sense in light of the increased GPA required for students to declare Elementary Education as a

major. It also might hint at how the integration of components of the Teacher Work Sample into other courses

starting in the sophomore field experiences have improved clinical interns’ ability to score higher on these

assessments. The faculty learned, from the previous assessment cycle, that it is important to introduce clinical

interns to the components of the final TWS Portfolio that they are expected to complete during their Professional

Internship prior to their final semester:

1. Assessment #3-The Interdisciplinary Unit/Level II Teacher Work Sample requires teacher candidates in Pre-Professional Internship to Develop a Unit of Study and to teach a part or all of the unit to students.

2. Assessment #7-The Language Arts and Literacy Development Case Study is an assignment that assists teacher candidates in acquiring the skills of assessing student language proficiency and developing a plan to help students improve reading and writing.

Although instructors and supervisors may not always agree on whether Kean clinical interns are producing

accomplished work, the general trend in the Final Recommendation scores indicate that Kean is producing clinical

interns that can develop curriculum, assess student learning, and reflect on their teaching in order to improve

instruction. Across the four semesters of data presented above, the majority of clinical interns, UG and PB, as well

as UGs at Union and OCC have received Final Recommendation scores at either the Accomplished (5) or Capable (4)

levels. When revisiting scores for the spring 2014 through fall 2015 semesters percentages of students reaching

either the Accomplished or Capable competency levels ranged from between 95% and 100% across subsets (UG &

PB) and (Union & OCC). Therefore, 100% of clinical interns reported on in this report, both Undergraduates and

Post-Baccalaureates at the Union Campus as well as 100% of clinical interns at OCC were recommended for

certification.

Section II

At the end of the 2013 academic year, faculty determined that although the TWS was an excellent way of aligning

the ACEI Standards with the work clinical interns were completing in the field, they also came to the conclusion that

the assessment of the components of TWS was not enough to measure Kean clinical interns’ impact on student

learning. Given the extensive reform efforts taking place at both the national and local levels requiring teachers to

measure growth via SGOs (Student Growth Outcomes), the COE Teaching Performance Center began collection data

on Pretest/Posttest Comparison Scores to determine whether the instruction taking place during the Professional

Field Experience was showing that students’ were acquiring the knowledge and skills necessary to prepare them for

the next grade and to be College and Career Ready. Below you will see a table reviewing the Average growth % for

the total cohort for semesters Spring 2014 to Fall 2015 (Full Section II Reports are found of the end of this report):

of 20

Semester Elementary Programs

UG/PB or Union & OCC

# of Clinical Interns Average % of Score

Increase

Spring 2014 UG/Post Bac/OCC 51 45%

Fall 2014 UG/Post Bac/OCC 44 39%

Spring 2015 UG/Post Bac/OCC 42 48%

Fall 2015 UG/Post Bac/OCC 22 45%

The data collected from Spring 2014 to Fall 2015 is not disaggregated by the subsets of Undergraduate/Post Bac and

Union/OCC; therefore, the data doesn’t present differences in the percentage of average score across the subsets of

UG, PB, and OCC. When averaging the percentage of score increase on posttest scores across the 4 semesters listed

above, the average increase equals 44%. When looking at the data by individual unit topic and Pretest/Posttest

scores, we can reason that, for the most part, the higher the pretest scores the lower the percentage of score

increase. This makes sense when you consider that the more knowledge a student already possess on a given topic,

the less he/she needs to learn in order to pass an assessment. Here are some examples of units that produced low

percentage increases across semesters – leading to a decrease in the overall percentage of average increase:

Semester Topic PreTest Score PostTest Score Increase %

Spring 2014 Geomentry 89.77 90.16 0%

Spring 2014 Reading Non-

Fiction

77 85 9%

Fall 2014 Weather 84 94 11%

Fall 2014 Health-Identifying

Personal Traits

83 92 10%

Fall 2014 Social Studies:

Map Skills

83 93 10%

Spring 2015 Butterflies 74 88 16%

Spring 2015 Making

Connections

78 95 18%

Fall 2015 Objects in the Sky 82 85 4%

of 20

CLOSING THE LOOP

Although the data indicates that students working with our clinical instructors increase the understanding of content

by an average of 44%, that doesn’t really tell us what individual students are experiencing in these classrooms. One

option might be for clinical interns to move away from the Pretest/Posttest model to one of working with Student

Growth Objectives geared towards developing realistic instructional goals for students at various levels of

proficiency at a given grade level.

Below is the link to the Achieve NJ Student Growth Objectives Guidebook that outlines procedures for creating

outcomes that run across a continuum of learning. This enables a student that enters a teachers class with little to

no knowledge or skill in a particular area to still show marked improvement over a marking period:

http://www.state.nj.us/education/AchieveNJ/teacher/SGOGuidebook.pdf

Here are examples of different types of Student Growth Objectives (p.4 – SGO Guidebook):

This doesn’t mean that our clinical interns aren’t doing this when they are teaching their units. They are being

required to measure student learning based on these design criteria, but we can’t access that data from the

information we have collected. Therefore, as indicated earlier in this report, it would be helpful if we can find a way

to follow several students longitudinally and find a way to correlate the true impact individual candidates have on

student learning through the TWS. It is important to note that the TWS will be replaced by edTPA in the fall of 2017

as a certification requirement. We will revise our imbedded assessments to assistant clinical interns in designing

and implementing the edTPA, but the edTPA portfolio will be assessed by trained outside evaluators.

SUPPORTING EVIDENCE: SEE ATTACHMENT B

http://www.state.nj.us/education/AchieveNJ/teacher/SGOGuidebook.pdf

academic program assessment report - kean universityncate/caep accreditation 2017/slo... ·...

Documents