prior year assessment history - ps.edu€¦ · prior year assessment history phoenix seminary...

PRIOR YEAR

ASSESSMENT

HISTORY

PHOENIX SEMINARY

2015-17 Comprehensive Assessment Plan

ACADEMIC PROGRAMS

Assessment Plans

2013-15

PHOENIX SEMINARY


2015-17 Phoenix Seminary Comprehensive Assessment Plan Page 115

Degree Program Student Learning Outcomes 2013‐15 Direct Measure Results

The following chart gives an overview of assessment results per student learning outcome in the MDiv, MAM and DMin programs. This assessment includes a weighted average of all direct measure results. A criterion of 80% success rate was set by the Assessment Committee. For more detailed analysis, see individual program assessment plans. Master of Divinity Core

1. Know and apply Scripture using proper hermeneutical principles.

88%

2. Articulate sound doctrine according to historic Christian orthodoxy.

91%

3. Explain how to shepherd people with biblical wisdom, compassion and justice.

97%

4. Demonstrate healthy relationships with God and neighbor.

95%

Master of Divinity in Biblical Communication

5. Demonstrate knowledge of the biblical languages for exegesis.

100%

6. Communicate Scripture clearly according to proper homiletical principles.

100%

Master of Divinity in Counseling and Family

5. Counsel according to biblical principles.

100%

6. Design and assess programs for biblical counsel and care.

99%

Master of Divinity in Leadership Development

5. Articulate how to replicate ministry leadership based on biblical principles.

100%

6. Demonstrate the ability to provide biblical leadership.

70%

Master of Arts in Ministry Core The student should be able to:

1. Know and apply Scripture using proper hermeneutical principles.

68%

2. Articulate sound doctrine according to historic Christian orthodoxy.

78%

3. Demonstrate ministry competency in an applied setting.

100%

4. Demonstrate healthy relationships with God and neighbor.

100%

Doctor of Ministry The student should be able to:

1. Produce biblical and theological solutions to challenges in ministry culture.

45%

2. Demonstrate advanced ministry knowledge in at least one of four areas of ministry: Preaching and Teaching, Ministry Leadership, Pastoral Care and Christian Formation.

50%

3. Exercise library and research skills appropriate for increasing ministerial competence.

100%

Master of Divinity Program 2013-15

Master of Divinity Core

Biblical Communication

Counseling and Family

Leadership Development

PHOENIX SEMINARY



Master of Divinity Program Summary of Assessment 2013-14

Assessment Results The results demonstrate that the Student Learning Outcomes (SLOs) in the Master of Divinity Core program are being met. New student learning outcomes were created last year and this is the first year of assessment. Overall, the assessment results are the most positive in several years. The MDiv Core has four student learning outcomes (SLOs). Following are the results per outcome. The benchmark for success is 80%:

1. Know and apply Scripture using proper hermeneutical principles. 89% 2. Articulate sound doctrine according to historic Christian orthodoxy. 92% 3. Explain how to shepherd people with biblical wisdom, compassion and justice. 97% 4. Demonstrate healthy relationships with God and neighbor. 95%

Highlights include:

Strengths: All student learning outcomes were met -- well above the 80% benchmark. This is the second year in a row that 100% of the students have passed the comprehensive

theological oral exam the first time. This year nearly half of these students did so with an “exemplary pass”.

Two new juried review measures were inaugurated: 1) MS508 funeral sermon which has proven effective in measuring all four core SLOs and 2) Field Education “360” review used in several SLOs and other degree programs.

Most assessment instruments are working well, particularly the juried reviews. Graduate feedback is consistently positive in all areas. Results in M.Div. concentration areas are mostly positive, especially in Biblical Communication. Action Items are linked to the budget process as necessary (see below). Action Items are linked to the Strategic Priorities they support and fulfill. Improvements to be addressed: SLO #1-2 – The Standardized Bible and Theology Test needs to be evaluated and possibly revised. MDiv CF direct measures need to be re-evaluated.

For detailed information, see the individual Degree Program Assessment Charts.

Budget Implications The Assessment Committee will consider paying a stipend to Faculty if extensive changes and additions are needed for the Standardized Bible and Theology Test. Strategic Plan Implications It is important to note that the Action Items from the degree program assessment plans have been aligned to specific Strategic Priorities to show the relationship between assessment and strategic planning and therefore the effectiveness of the institution itself.


Those Strategic Priorities most often aligned in degree program assessment are Strategic Priority #3 “Develop the Seminary’s Academic Programs” and Strategic Priority #5 “Develop an institutional culture committed to continuous improvement through assessment.”

It is our expectation that though the Strategic Priorities are well-developed and cover all aspects of the seminary, it is conceivable that results from degree program assessment could engender significant change or addition to the Strategic Priorities. In the meantime, the Action Items support and fulfill the Strategic Priorities (see M.Div. Assessment Charts). Direct and Indirect Measures Phoenix Seminary currently has very effective direct measures for the MDiv program. (A minor exception can be seen in the “Improvements” section.) As noted in the “Strengths” section, two new juried review measures were inaugurated: 1) MS508 funeral sermon which has proven effective in measuring all four core SLOs and 2) Field Education “360” review used in several SLOs and other degree programs. Assessment Process The Director of Assessment collects all results from the Direct and Indirect Measures, making a brief, initial analysis. The degree program assessment data and this initial analysis are reviewed by Faculty in late August. The Faculty makes a comprehensive analysis and suggests change for the future. Revised plans are finalized by the Director of Assessment and presented to the Assessment Committee for final approval. Following approval, the Director of Assessment creates an action list and follows through on all action items for the year.

Conclusion Assessment demonstrates the strength of the M.Div. program. The M.Div. program has benefited from revised and reduced direct measures. A new multi-year cycle will benefit the M.Div. program by providing more accurate assessment results. The linkage to the strategic plan and budget is important for overall institutional effectiveness.

Assessment Plan and Record for Year: 2013-14

Institutional Mission: To equip godly leaders to serve Jesus Christ effectively in a diverse and changing world by providing theological education with personal mentoring and ministry training in partnership with the local church.

Academic/Administrative/Student Services Program: Master of Divinity Core

Program Purpose: The Master of Divinity degree is designed to equip students for professional, vocational ministry in church and para-church settings. This may include a broad range of occupations, including the pastorate, intercultural service, Christian leadership, pastoral counseling, chaplaincy, evangelism, and youth ministry. In many churches and denominations, the M.Div. degree has been recommended for those preparing for ordination. It can also provide the academic basis for the D.Min., Th.M. or Ph.D. degrees. Assessment Plan (“beginning the loop”) Assessment Record (“closing the loop”) Student Learning Outcomes (SLO)

Criteria & Procedures Assessment Results Analysis and Use of Results Strategic Planning and/or Budget Action

Responsible for Follow-up

1. The student should be able to know and apply Scripture using proper hermeneutical principles

Aligned to: ATS Standard A.2.2 - Religious Heritage

Direct Measures Comprehensive Theology Oral Exam Score. Success will be achieved if 75% of students receive an unconditional or conditional pass on the oral exam the first time. Standardized Bible and Theology Test administered at graduation. Success will be achieved if 85% of students achieve a score of 84 or higher. Also evaluate improvement rate between entering and graduating. Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. BL501 Final Exam Theological Oral Exam BC only - BC502 Sermon BC only - OT/NT Exegesis Paper 30-Hour Review of Students Success will be

achieved if 60% of students receive a “meet” or “exceeds” for this outcome.

60-Hour Review of Students Faculty Review of Graduates Indirect Measures Annual Alumni Survey – “I am satisfied with my

ability to use and interpret the Scriptures.” Success will be achieved if 80% “strongly agree” or “agree.”

ATS GSQ Table 13 - Ability to use/interpret Scriptures – success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage;

ATS AQ Table 23 - Importance of biblical studies and spiritual directions/faith practices in professional work - success for both AQ tables will be achieved if the average score is a 3 (effective/somewhat important) or higher. Maximum score is a 4.

Direct Measure Summary - 89% achieved benchmark or higher (summary of weighted measures) Comprehensive Theology Oral Exam Score - 100% (45% exemplary passes) Standardized Bible and Theology Test – 50% rec’d 84 pts or higher Juried Reviews BL501 Final Exam – 86% Oral Exam – 95% BC only - BC502 Sermon – 75% BC only - OT/NT Exegesis Paper

- both were 100% 30-Hour Review of Students

MDiv BC – 87% MDiv CF – 100% (small sample) MDiv LD – 100% (small sample)

60-Hour Review of Students MDiv BC – 100% MDiv CF – No results (there were no 60-hour review students) MDiv LD – 100%

Faculty Review of Graduates– 99%

Included in juried review but not listed as direct measure for this SLO: MS508 funeral sermon – 88% Indirect Measures Annual Alumni Survey – not

conducted ATS GSQ Table 13 – Ability to

use and interpret Scriptures – 4.9 ATS AQ Table 21 – Effectiveness

in education: comprehensive understanding of religious heritage – 4.0

ATS AQ Table 23 - Importance of biblical studies – 3.0; and spiritual directions/faith practices in professional work – 2.6

Results indicate the SLO was met. The Bible and Theology test continues to have low scores. The concerns were discussed early in 2013 but due to faculty load, changes were unable to be made. Concerted effort should be taken in the 2014-15 year to enact the changes that will enhance the learning process and gain more accurate results.

ACTION: Strategic Priority #5 “…continuous improvement through assessment” – Work on changes to the Bible & Theology test so it’s a learning tool for the students and therefore hopefully presents more meaningful assessment results. Budget implication – consider paying a stipend to Faculty who make these detailed revisions.

Responsible Party: Roma Royer Follow-up Date: Fall 2014

2015

-17

Pho

enix

Sem

inar

y

Pag

e 11

9 C

ompr

ehen

sive

Ass

essm

ent P

lan

2. The student should be

able to articulate sound doctrine according to historic Christian orthodoxy.


Direct Measures Comprehensive Theology Oral Exam Score. Success will be achieved if 75% of students receive an unconditional or conditional pass on the oral exam the first time. Standardized Bible and Theology Test (theology portion only) administered at graduation. New success measure using only theology portion will need to be determined. Also evaluate improvement rate between entering and graduating. Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Field Education Theological Oral Exam BC only - BC502 Sermon BC only - OT/NT Exegesis Paper 30-Hour Review of Students Success will be


60-Hour Review of Students Faculty Review of Graduates Indirect Measures Annual Alumni Survey – 1) As a result of my

training, I’m able to identify key theological concepts; 2) My training in theology has given me confidence in understanding the theological issues o the day and their importance in ministry; Success will be achieved if 80% “strongly agree” or “agree.”

Student survey after oral exam; one question addresses this outcome. Success will be achieved if 90% “strongly agree” or “agree.”

ATS GSQ Table 15 - Knowledge of church doctrine; Ability to think theologically - success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage

ATS AQ Table 23 - Importance of biblical studies, theology and ethics and moral theology in professional work - success for both AQ tables will be achieved if the average score is a 3 (effective/somewhat important) or higher. Maximum score is a 4.

Direct Measure Summary - 92% achieved benchmark or higher (summary of weighted measures) Comprehensive Theology Oral Exam Score - 100% (45% exemplary passes) Standardized Bible and Theology Test (theology portion only) – 55% rec’d 21 or higher Juried Reviews Field Education – 100% Oral Exam – 95% BC only - BC502 Sermon – 83% BC only - OT/NT Exegesis

Paper- both were 100% 30-Hour Review of Students




Included in juried review but not listed as direct measure for this SLO: MS508 funeral sermon – 88% Indirect Measures Annual Alumni Survey – not

conducted ATS GSQ Table 13 –

Knowledge of church doctrine – 4.7; Ability to think theologically – 4.7

Student survey after oral exam - -95%

ATS GSQ Table 15 - Knowledge of church doctrine – 4.7 Ability to think theologically – 4.7

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage – 4.0

ATS AQ Table 23 - Importance of biblical studies (3.0), theology (3.0) and ethics and moral theology in professional work (2.6)

Results indicate the SLO was met.

ACTION: Strategic Priority #5 “…continuous improvement through assessment” – Work on changes to the Bible & Theology test so it’s a learning tool for the students and therefore hopefully presents more meaningful assessment results. Budget implication – consider paying a stipend to Faculty who make these detailed revisions.

Responsible Party: Roma Royer Follow-up Date: Fall 2014

2013

-15

Pho

enix

Sem

inar

y

Pag

e 12

0 C

ompr

ehen

sive

Ass

essm

ent P

lan


able to explain how to shepherd people with biblical wisdom, compassion and justice.

Aligned to: ATS Standard A.2.4 - Personal & Spiritual Formation ATS Standard A.2.5 - Capacity for ministerial and public leadership

Direct Measures Comprehensive Theology Oral Exam Score. Success will be achieved if 75% of students receive an unconditional or conditional pass on the oral exam the first time. Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. MS508 funeral sermon Theological Oral Exam BC only - BC502 Sermon 30-Hour Review of Students Success will be


60-Hour Review of Students Faculty Review of Graduates Indirect Measures Evaluation from internship supervisor which

includes a question addressing this outcome. Success will be achieved if 80% of respondents choose “Strongly Agree” or “Agree”.

Annual Alumni Survey – My training equipped me to shepherd God’s people with wisdom and compassion; Success will be achieved if 80% “strongly agree” or “agree.”

ATS GSQ Table 14 - Empathy for poor and oppressed; Concern for social justice; Insight into troubles of others; success will be achieved if the average score is a 4 (stronger) or higher. Maximum score is a 5.

ATS GSQ Table 15 – Ability to lead others; Ability to give spiritual direction; Ability in pastoral counseling; success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage; understanding cultural context of religious work; development in personal & spiritual formation

ATS AQ Table 23 - Importance of spiritual directions/faith practices, pastoral counseling and ethics and moral theology in professional work - success for both AQ tables will be achieved if the average score is a 3 (effective/somewhat important) or higher. Maximum score is a 4.

Direct Measure Summary - 97% achieved benchmark or higher (summary of weighted measures) Comprehensive Theology Oral Exam Score - 100% (45% exemplary passes) Juried Reviews MS508 funeral sermon – 88% Oral Exam 95% BC only - BC502 Sermon – 100% 30-Hour Review of Students



Faculty Review of Graduates – 99% Indirect Measures Evaluation from internship

supervisor – this question was not addressed but will be in the future.

Annual Alumni Survey – not conducted.

ATS GSQ Table 14 - Empathy for poor and oppressed – 4.5; Concern for social justice – 4.4; Insight into troubles of others – 4.6

ATS GSQ Table 15 – Ability to lead others – 4.2; Ability to give spiritual direction – 4.4; Ability in pastoral counseling 4.3;

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage – 4.0; understanding cultural context of religious work – 3.2; development in personal & spiritual formation – 3.6

ATS AQ Table 23 - Importance of spiritual directions/faith practices 2.6; pastoral counseling – 2.6 and ethics and moral theology in professional work – 2.6.


No further action is needed.

20

13-1

5 P

hoen

ix S

emin

ary

P

age

121

Com

preh

ensi

ve A

sses

smen

t Pla

n


able to demonstrate healthy relationships with God and neighbor. Aligned to: ATS Standard A.2.3 - Cultural context ATS Standard A.2.4 - Personal & Spiritual Formation ATS Standard A.2.5 - Capacity for ministerial and public leadership

Direct Measures Comprehensive Theological Oral Exam Success will be achieved if 75% of students receive an unconditional or conditional pass on the oral exam the first time. Results of 30/60 hour reviews Success will be achieved if 80% of students pass the first time. Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Field Education Theological Oral Exam 30-Hour Review of Students Success will be


60-Hour Review of Students Faculty Review of Graduates Indirect Measure Internship Supervisor Evaluation – Question addressing

this outcome. Success will be achieved if 80% of respondents choose “Strongly Agree” or “Agree”.

Annual Alumni Survey – My training has enabled me to demonstrate this outcome; My experience…has made me more aware of issues related to ministry among people of diverse gender/diverse ethnicity; diverse culture. Success will be achieved if 80% “strongly agree” or “agree.”

ATS GSQ Table 14 – Trust in God; Ability to live one’s faith in daily life; Strength of Spiritual life; Empathy for poor and oppressed; Concern for social justice; Insight into troubles of others; success will be achieved if the average score is a 4 (stronger) or higher. Maximum score is a 5.

ATS GSQ Table 15 - Ability to relate social issues to faith; success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage; understanding cultural context of religious work; development in personal & spiritual formation

ATS AQ Table 23 - Importance of spiritual directions/faith practices in professional work - success for both AQ tables will be achieved if the average score is a 3 (effective/somewhat important) or higher. Maximum score is a 4.

Direct Measure Summary - 95% achieved benchmark or higher (summary of weighted measures) Comprehensive Theological Oral Exam - 100% (45% exemplary passes) Results of 30/60 hour reviews Juried Reviews Field Education – 75% Oral Exam – 95% 30-Hour Review of Students




Included in juried review but not listed as direct measure for this SLO: BC only - BC502 Sermon – 100% Indirect Measure Internship Supervisor Evaluation –

100% ATS GSQ Table 14 – Trust in God –

4.6; Ability to live one’s faith in daily life – 4.6; Strength of Spiritual life – 4.3; Empathy for poor and oppressed – 4.5; Concern for social justice – 4.4; Insight into troubles of others – 4.6;

ATS GSQ Table 15 - Ability to relate social issues to faith – 4.4;

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage – 4.0; understanding cultural context of religious work – 3.2; development in personal & spiritual formation – 3.6

ATS AQ Table 23 - Importance of spiritual directions/faith practices in professional work - 2.6



2013

-15

Pho

enix

Sem

inar

y

Pag

e 12

2 C

ompr

ehen

sive

Ass

essm

ent P

lan



Academic Program: Master of Divinity – Biblical Communication Concentration Program Purpose: The Biblical Communication concentration equips students for life-changing, expository preaching and teaching of the Old and New Testaments, and it can provide the basis for further academic study. In many churches and denominations, this degree has been recommended for those preparing for ordination. It can also provide the academic basis for the D.Min., Th.M., or Ph.D. degrees. Assessment Plan (“beginning the loop”) Assessment Record (“closing the loop”) Student Learning Outcomes (SLO)

Criteria & Procedures Assessment Results Analysis and Use of Results Planning and/or Budget Action Responsible for Follow-up

In addition to the M.Div. outcomes 1-4 that apply to all M.Div. programs:

5. The student should be able to demonstrate knowledge of the biblical languages for exegesis.


Direct Measures Comprehensive Theology Oral Exam Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. OT503/NT503 Exegesis Paper Field Education 60-Hour Review of Students Faculty Review of Graduates Annual Alumni Survey – My training in

Hebrew/Greek has given me confidence in my exegesis and proclamation of the OT/NT; Success will be achieved if 80% “strongly agree” or “agree.”

Student Course Evaluations from NT503 and OT503. Success will be achieved if 90% of the students mark “Strongly Agree” or “Agree” that all of the course outcomes were met in the course.

Direct Measure Summary - 100% achieved benchmark or higher (summary of weighted measures) Comprehensive Theology Oral Exam

– 100% (45% exemplary passes) Juried Reviews OT503/NT503 Exegesis Paper - both

were 100% Field Education – 100% 60-Hour Review of Students – 100% Faculty Review of Graduates– 99% Indirect Measures Student Course Evaluations from NT503 and OT503 – 90%

Results indicate the SLO was met. Historical results for NT/OT503 Juried Review papers: Fall 2013 - OT503 – 100% Fall 2012 - OT503 – 67% Fall 2011 - OT503 – 92% Fall 2013 - NT503 – 100% Fall 2012 - NT503 – 100% Fall 2011 - NT503 – n/a

ACTION: Strategic Priority #5 “…continuous improvement through assessment” Remove 30-hour review as direct measure; this was an oversight from the previous year’s assessment when it was discovered that students at the 30-hour mark won’t have had many concentration courses. There are no budget implications at this time.

Responsible Party: Roma Royer Follow-up Date: May 1, 2015

6. The student should be able to communicate Scriptures clearly according to proper homiletical principles.

Aligned to: ATS Standard A.2.5 - Capacity for ministerial and public leadership

Direct Measures Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. BC502 Sermon Field Education 60-Hour Review of Students Faculty Review of Graduates Indirect Measures Annual Alumni Survey – My training has provided

me with 1) adequate sermon development skills 2) adequate training in skills of public communication; Success will be achieved if 80% “strongly agree” or “agree.”

ATS GSQ Table 13 – Ability to preach well; (though this is not only BC students) – success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

Direct Measure Summary - 100% achieved benchmark or higher (summary of weighted measures) Juried Reviews BC502 Sermon – 100% Field Education – 100%

60-Hour Review of Students MDiv BC – 100%

Faculty Review of Graduates– 99% Indirect Measures ATS GSQ Table 15 – Ability to preach

well; (though this is not only BC students) – 4.3

Results indicate the SLO was met. Historical results for BC502 Sermon Juried Review: Spring 2014 – 100% Spring 2013 - 72% Spring 2012 – 89%



2015

-17

Pho

enix

Sem

inar

y

Pag

e 12

3 C

ompr

ehen

sive

Ass

essm

ent P

lan

Approved by the Assessment Committee – 9/9/14 Assessment Plan and Record for Year: 2013-14


Academic Program: Master of Divinity – Counseling & Family Concentration Program Purpose: The Counseling and Family concentration prepares students to provide biblical guidance, sustenance, healing, and reconciliation to individuals, couples, and families in either the church or para-church settings.

Assessment Plan (“beginning the loop”) Assessment Record (“closing the loop”) Intended Program Outcomes Criteria & Procedures Assessment Results Analysis and Use of Results Planning and/or Budget Action Responsible for Follow-up


5. The student should be able to counsel according to biblical principles.

Aligned to: ATS Standard A.2.2 - Religious Heritage ATS Standard A.2.5 - Capacity for ministerial and public leadership

Direct Measures Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Faculty Review of Graduates 30-Hour Review of Students Success will be achieved if

60% of students receive a “meet” or “exceeds” for this outcome.

60-Hour Review of Students Additional direct measures in development Indirect Measures Alumni survey question.

Direct Measure Summary - 100% achieved benchmark or higher (summary of weighted measures) Juried Reviews. Faculty Review of Graduates –

99% 30-Hour Review of Students –

should be removed as direct measure; 30 hours is too early to measure

60-Hour Review of Students – No results (there were no 60-hour review students)

Additional direct measures in development

Results indicate the SLO was met. A direct measure is still needed for this SLO. The committee will need to review direct measures for MDiv CF students during the next review.

ACTION: Strategic Priority #5 “…continuous improvement through assessment” Remove 30-hour review as direct measure; this was an oversight from the previous year’s assessment when it was discovered that students at the 30-hour mark won’t have had many concentration courses. ACTION: Strategic Priority #5 “…continuous improvement through assessment” – A direct measure needs to be developed for this SLO to be used in 2015-16 assessment. There are no budget implications at this time.

Responsible Party: Roma Royer Follow-up Date: May 1, 2015 Responsible Party: Justin Smith Follow-up Date: May 1, 2015

6. The student should be able to design and assess programs for biblical counsel and care.


Direct Measures Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. CF507 Project Faculty Review of Graduates 30-Hour Review of Students Success will be achieved if

60% of students receive a “meet” or “exceeds” for this outcome.

60-Hour Review of Students Indirect Measures In development

Direct Measure Summary - 99% achieved benchmark or higher (summary of weighted measures) CF507 Project – Review not conducted since two of the three projects were done by students in other programs. We are saving papers and will hopefully have enough to review in 2015 when the course is next offered. Juried Reviews CF507 Project Faculty Review of Graduates– 99% 30-Hour Review of Students –


60-Hour Review of Students – No results (there were no 60-hour review students)

Results indicate the SLO was met. CF507 Project is a good measure but there were not enough results to assess (see previous column). This will be reviewed in 2015-16. The committee will need to review direct measures for MDiv CF students during the next review.

ACTION: Strategic Priority #5 “…continuous improvement through assessment” Remove 30-hour review as direct measure; this was an oversight from the previous year’s assessment when it was discovered that students at the 30-hour mark won’t have had many concentration courses. ACTION: Strategic Priority #5 “…continuous improvement through assessment”- Review direct (and indirect) measures for MDiv CF in light of small enrollment. There are no budget implications at this time.

Responsible Party: Roma Royer Follow-up Date: May 1, 2015 Responsible Party: Assessment Committee Follow-up Date: September 1, 2015

2015

-17

Pho

enix

Sem

inar

y

Pag

e 12

4 C

ompr

ehen

sive

Ass

essm

ent P

lan



Academic Program: Master of Divinity – Leadership Development Concentration Program Purpose: The purpose of the Leadership Development concentration is to prepare for vocational ministry positions in church and para-church ministries, wit the exception of preaching. This degree can provide the basis for further academic study. Assessment Plan (“beginning the loop”) Assessment Record (“closing the loop”) Student Learning Outcomes (SLO)



5. The student should be able to articulate how to replicate ministry leadership based on biblical principles.


Direct Measures LD502 Pre/Post Test (questions addressing this outcome) Success is achieved if 90% of the students have a “B” grade or higher. Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Faculty Review of Graduates 30-Hour Review of Students Success will be achieved

if 60% of students receive a “meet” or “exceeds” for this outcome.

60-Hour Review of Students Indirect Measures Annual Alumni Survey – Student should be able to

explain the aspects of leadership from the Bible; Success will be achieved if 80% “strongly agree” or “agree.”

Direct Measure Summary - 100% achieved benchmark or higher (summary of weighted measures) LD502 Pre/Post Test – 100% of students received an “A” Juried Reviews Faculty Review of Graduates –



60-Hour Review of Students – 100%




6. The student should be able to demonstrate the ability to provide biblical leadership.


Direct Measures Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. MS501 Paper Field Education Faculty Review of Graduates 30-Hour Review of Students Success will be achieved

if 60% of students receive a “meet” or “exceeds” for this outcome.

60-Hour Review of Students Indirect Measures In development

Direct Measure Summary - 70% achieved benchmark or higher (summary of weighted measures) Juried Reviews. MS501 Paper – 0% received meet or

exceeds; measure will change as class is discontinued

Field Education - 100% Faculty Review of Graduates –



60-Hour Review of Students – 100%

Results indicate the SLO was not met. The MS501 paper – The jury scored the paper as not applicable to the SLO. It is MS501 is being merged with LD501 to be MS509 Competencies in Ministry Leadership. It will have an assignment specific to this SLO. Future assessment results should be better.

ACTION: Strategic Priority #5 “…continuous improvement through assessment” Remove 30-hour review as direct measure; this was an oversight from the previous year’s assessment when it was discovered that students at the 30-hour mark won’t have had many concentration courses. ACTION: Strategic Priority #5 “…continuous improvement through assessment” Change MS501 Paper to MS509 Paper given the merging of courses. There are no budget implications at this time.

Responsible Party: Roma Royer Follow-up Date: May 1, 2015 Responsible Party: Roma Royer Follow-up Date: May 1, 2015

2015

-17

Pho

enix

Sem

inar

y

Pag

e 12

5 C

ompr

ehen

sive

Ass

essm

ent P

lan

Master of Arts in Ministry

Program

2013-15

PHOENIX SEMINARY



Master of Arts in Ministry Program Summary of Assessment 2013-15

Assessment Results The 2014-15 assessment cycle is the first one for the Master of Arts in Ministry (MAM). The program began two years ago in Fall 2013. It is early days and another cycle of assessment will be helpful in more accurately assessing this program. Two of these outcomes were met by 100% results. This is the first MA program the seminary has offered with a significantly reduced number of hours -- 48. Seeing an opportunity to finish sooner, a number of Master of Divinity students changed to the MAM, some in hopes of returning for their M.Div. These assessment results include a number of these students. The MAM has four student learning outcomes (SLOs). Following are the results per outcome. The benchmark for success is 80%:

1. Know and apply Scripture using proper hermeneutical principles. 68% 2. Articulate sound doctrine according to historic Christian orthodoxy. 78% 3. Demonstrate ministry competency in an applied setting. 100% 4. Demonstrate healthy relationships with God and neighbor. 100%

Highlights of the results are as follows:

Strengths: Three of the four SLOs are close to or well above the benchmark of 80% Juried reviews of course-embedded assignments continue to result in more precise and objective

results for each SLO. The juried review of field education produced a 100% result in SLOs #2-4. The Faculty Review of Graduates (The Faculty’s professional judgement) consistently produced a

100% result across all student learning outcomes. Action Items are linked to the budget process as necessary (see below). Action Items are linked to the Strategic Priorities they support and fulfill.

Improvements to be addressed: SLO #1-2 Faculty agree that the Standardized Bible and Theology test (for entering and graduating

students) should be evaluated to make sure it’s assessing the knowledge we hope graduates readily have upon completion of their program. Historically, students have not done well in this test. There are multiple reasons why this may be the case including the fact that there is no grade requirement. The Assessment Committee will be evaluating this test in the coming year.

SLO #2 – The Assessment Committee will determine if the research paper in TH503 would be a more rigorous and effective direct measure as compared to the current TH503 doctrinal statement.



Budget Implications Results do not indicate any needed budget changes for 2015-17. Strategic Plan Implications It is important to note that the Action Items from the degree program assessment plans have been aligned to specific Strategic Priorities to show the relationship between assessment and strategic planning and therefore the effectiveness of the institution itself.


It is our expectation that while the Strategic Priorities cover all aspects of the seminary, it is conceivable that results from degree program assessment could engender significant change or addition to the Strategic Priorities. In the meantime, the Action Items support and fulfill the Strategic Priorities (see MAM Assessment Charts). Direct and Indirect Measures The direct measures for MAM assessment are appropriate in kind and number for the student learning outcomes. As noted under “Improvements”, the Standardized Bible and Theology Test needs to be evaluated and possibly revised. It was noted that the results of the ATS Graduating Student Questionnaire are generally lower than our benchmark and lower than MDiv results which consistently rate in the “4’s”. This may be due to the fact that the MAM is a much shorter program than the MDiv. Assessment Process The Director of Assessment collects all results from the Direct and Indirect Measures, making a brief, initial analysis. The degree program assessment data and this initial analysis are reviewed by Faculty in late August. The Faculty makes a comprehensive analysis and suggests change for the future. Revised plans are finalized by the Director of Assessment and presented to the Assessment Committee for final approval. Following approval, the Director of Assessment creates an action list and follows through on all action items for the year.

Conclusion The MAM program is brand new. The student learning outcomes are appropriate for the degree program. A comprehensive assessment plan is in place and the multi-year cycle will benefit the MAM program by providing more accurate assessment results. As noted, attention will be given to the Standardized Bible and Theology Test. The linkage to the strategic plan and budget is important for overall institutional effectiveness.



Academic Program: Master of Arts in Ministry Program Purpose: The Master of Arts in Ministry prepares students for Christian ministry other than pulpit/preaching ministries. It specifically equips students for a variety of professional church and para-church ministries or as church lay leaders. It is expressly not designed to prepare students academically for doctoral studies or for ordination in those churches that require the M.Div. degree. Assessment Plan (“beginning the loop”) Assessment Record (“closing the loop”) Student Learning Outcomes (SLO)


1. The student should be able to know and apply Scripture using proper hermeneutical principles.

Aligned to: ATS Standard B.2.2 - Religious Heritage

Direct Measures Standardized Bible and Theology Test administered at graduation. Success will be achieved if 85% of students achieve a score of 84 or higher. Also evaluate improvement rate between entering and graduating. Entire test – (success if 85% of students achieve a score of 84 [out of 100] or higher) Bible only – (success if 85% of students achieve a score of 63 [out of 75] or higher) Theology only – (success if 85% of students achieve a score of 20 [out of 24] or higher) Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. BL501 Final Exam 30-Hour Review of Students Faculty Review of Graduates Indirect Measures Annual Alumni Survey – “I am satisfied with my ability to

use and interpret the Scriptures.” Success will be achieved if 80% “strongly agree” or “agree.”

Internship Supervisor Evaluation ATS GSQ Table 15 - Ability to use/interpret Scriptures –

success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage; providing for develop. in personal spiritual formation

ATS AQ Table 23 - Importance of biblical studies and spiritual directions/faith practices in professional work -

success for both AQ tables will be achieved if the average score is a 3 (effective/somewhat important) or higher. Maximum score is a 4.

Direct Measure Summary - 68% achieved benchmark or higher (summary of weighted measures)

Standardized Bible and Theology Test – MAM Graduating 2014 & 2015: (20 Graduates) --Entire test – 20% --Bible only – 40% --Theology only – 25% --Increase from Incoming to Graduating average scores was 37% Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. BL501 Final Exam – 50% Combined doctrinal statement –


100% but small sample size due to the program being initiated one year before.

Faculty Review of Graduates – 100%

Indirect Measures Annual Alumni Survey – not conducted Internship Supervisor Eval – results not

available ATS GSQ Table 15 – Ability to use and

interpret Scriptures – 4.7 ATS AQ Table 21 & 23– too soon for

MAM results on the AQ.

Results indicate the SLO was not met. Bible & Theology Test – Results continue to be disappointing. Even with it being “required” for graduation, students may not be taking it seriously. Faculty discussed the poor results across most programs and agreed that the test should evaluate what students readily know without needing to study for the test. A review of the test, along these lines, was suggested. Other options from previous discussions should be included as the Assessment Committee discusses the possibilities. BL501 Exam - Previous years have shown that this may not be the best artifact to measure this SLO given that it’s most students’ first course and doesn’t give a culminating assessment of what students know in this area. Faculty discussed the possibility of placing the BL501 course at the end of the curriculum sequence but Faculty were not in agreement, seeing the benefits of having it at the beginning as well as the end. Faculty believe it’s a good measure for this outcome with the exception that it’s not culminating. TH503 doctrinal statement –The 100% were all “meets” with no “exceeds”. Due to their brevity, it was difficult to know what an “exceeds” paper would look like. This is the first time this instrument was used. The Assessment Committee believes that consideration should be given to replacing this with the TH503 Research Paper in order to have more substantive results. Faculty Review of Graduates – a larger number than expected this early in the program because a number of MDiv students transferred into the program.

ACTION: Strategic Priority #5 “…continuous improvement through assessment” – Bible & Theology Test – The Assessment Committee will review the possibilities of improving the test so that it measures what it should. (see previous column) ACTION: Strategic Priority #5 “…continuous improvement through assessment” – TH503 Research Paper should be considered to replace the TH503 doctrinal statements. ACTION: Strategic Priority #5 “…continuous improvement through assessment” – Add Juried Review of TH503 Paper to Criteria column should it be approved by the Assessment Committee. There are no budget implications at this time.

Responsible Party: Roma Royer Follow-up Date: December 31, 2015 Responsible Party: Roma Royer Follow-up Date: November 30, 2015 Responsible Party: Roma Royer Follow-up Date: December 31, 2015

2015

-17

Pho

enix

Sem

inar

y

Pag

e 12

9 C

ompr

ehen

sive

Ass

essm

ent P

lan


able to articulate sound doctrine according to historic Christian orthodoxy.

Aligned to: ATS Standard B.2.2 - Religious Heritage

Direct Measures Standardized Bible and Theology Test (theology portion only) administered at graduation. New success measure using only theology portion will need to be determined. Also evaluate improvement rate between entering and graduating. Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Combined doctrinal statement juried review Field Education 30-Hour Review of Students Faculty Review of Graduates Indirect Measure Alumni Survey – several questions ATS GSQ Table 13 – Knowledge of church doctrine;

ability to think theologically – success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage

ATS AQ Table 23 - Importance of biblical studies and theology & ethics & moral theology in professional work - success for both AQ tables will be achieved if the average score is a 3 (effective/somewhat important) or higher. Maximum score is a 4.

Direct Measure Summary - 78% achieved benchmark or higher (summary of weighted measures) Standardized Bible and Theology Test (theology portion) – Graduating 2014 & 2015: --Theology only – 25% --Increase from Incoming to Graduating average scores was 42% Juried Review: Combined doctrinal statement

juried review – 100% Field Education – 100% 30-Hour Review of Students –



Indirect Measures Annual Alumni Survey – not conducted Internship Supervisor Eval – 100% ATS GSQ Table 15 – Know. of church

doctrine – 4.1; Ability to think theologically – 4.7

ATS AQ Table 21 & 23– too soon for MAM results on the AQ.

Results indicate the SLO was not met. MAM students take the same Theology survey courses as MDiv but they don’t take the oral exam which may be a factor in their low scores in the Bible & Theology test. Bible & Theology Test – See SLO #1. TH503 doctrinal statement – See SLO #1 Field Education – The AC should give consideration as to whether SLO #2 should be measured by this instrument. It was a challenge last year for the MDiv juried review to get SLO#2 information from this instrument as it was this year with the MAM. Faculty Review of Graduates – a larger number than expected this early in the program because a number of MDiv students transferred into the program.

ACTION: Strategic Priority #5 “…continuous improvement through assessment” – Bible & Theology Test – The Assessment Committee will review the possibilities of improving the test so that it measures what it should. (see previous column) ACTION: Strategic Priority #5 “…continuous improvement through assessment” – TH503 Research Paper should be considered to replace the TH503 doctrinal statements. ACTION: Strategic Priority #5 “…continuous improvement through assessment” – Add Juried Review of TH503 Paper to Criteria column should it be approved by the Assessment Committee. There are no budget implications at this time.

Responsible Party: Roma Royer Follow-up Date: December 31, 2015 Responsible Party: Roma Royer Follow-up Date: November 30, 2015 Responsible Party: Roma Royer Follow-up Date: December 31, 2015

3. The student should be able to demonstrate ministry competency in an applied setting..

Aligned to: ATS Standard B.2.4 - Personal & Spiritual Formation ATS Standard B.2.5 - Specialization

Direct Measures Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Field Education 30-Hour Review of Students Faculty Review of Graduates Indirect Measures Annual Alumni Survey – several questions ATS GSQ Table 14 – Empathy for poor & oppressed;

concern for social justice; insight into troubles of others - success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS GSQ Table 15– Ability to lead others; ability to give spiritual direction; ability in pastoral counseling - success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage; understanding cultural context of religious work; develop. in personal & spiritual formation

ATS AQ Table 23 - Importance of biblical studies & spiritual directions/faith practices and pastoral counselling and ethics and moral theology in professional work - success for both AQ tables will be achieved if the average score is a 3 (effective/somewhat important) or higher. Maximum score is a 4.

Direct Measure Summary - 100% achieved benchmark (summary of weighted measures) Juried Reviews: Field Education – 100% 30-Hour Review of Students –



Indirect Measures Annual Alumni Survey – several

questions Internship Supervisor Eval – 100% ATS GSQ Table 14 – Empathy for poor

& oppressed – 3.5; concern for social justice – 3.5; insight into troubles of others – 3.7

ATS GSQ Table 15– Ability to lead others – 3.8; ability to give spiritual direction – 3.8; ability in pastoral counseling – 3.5


Results indicate the SLO was met. Faculty Review of Graduates – a larger number than expected this early in the program because a number of MDiv students transferred into the program. Results of ATS GSQ are lower than MDiv results which consistently rate in the “4’s”. This may be due to a much shorter program than the MDiv.


2013

-15

Pho

enix

Sem

inar

y

Pag

e 13

0 C

ompr

ehen

sive

Ass

essm

ent P

lan


able to demonstrate healthy relationships with God and neighbor.

Aligned to: ATS Standard B.2.3 - Cultural context ATS Standard B.2.4 - Personal & Spiritual Formation

Direct Measures Results of 30- hour reviews Success will be achieved if 80% of students pass the first time. Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Field Education 30-Hour Review of Students Faculty Review of Graduates Indirect Measures Annual Alumni Survey – several questions ATS GSQ Table 14 – Trust in God; ability to live one’s faith

in daily life; strength of spiritual life; empathy for poor and oppressed; concern for social justice; insight into troubles of others - success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS GSQ Table 15 – Ability to relate social issues to faith - success will be achieved if the average score is a 4 (satisfied) or higher. Maximum score is a 5.

ATS AQ Table 21 – Effectiveness in education: comprehensive understanding of religious heritage; understanding cultural context of religious work; develop. in personal & spiritual formation

ATS AQ Table 23 - Importance of spiritual directions/faith practices in professional work - success for both AQ tables will be achieved if the average score is a 3 (effective/ somewhat important) or higher. Maximum score is a 4.

Direct Measure Summary - 100% achieved benchmark (summary of weighted measures) Results of 30- hour reviews – 100% Juried Reviews: Field Education – 100% 30-Hour Review of Students –



Indirect Measures Annual Alumni Survey – several

questions Internship Supervisor Eval – 100% ATS GSQ Table 14 – Trust in God – 3.8;

ability to live one’s faith in daily life –4.0; strength of spiritual life – 3.6; empathy for poor and oppressed – 3.5; concern for social justice – 3.5; insight into troubles of others – 3.7

ATS GSQ Table 15 – Ability to relate social issues to faith – 4.2


Results indicate the SLO was met. Faculty Review of Graduates – a larger number than expected this early in the program because a number of MDiv students transferred into the program. One faculty member noted that the Faculty Reviews are repeatedly higher than other results and a less strenuous measure than the juried reviews or data from tests.

ACTION: Strategic Priority #5 “…continuous improvement through assessment” – Assessment Committee should review the use of the Faculty and Graduate review measure (see previous column). There are no budget implications at this time.


2013

-15

Pho

enix

Sem

inar

y

Pag

e 13

1 C

ompr

ehen

sive

Ass

essm

ent P

lan

Doctor of Ministry Program

2013-15

PHOENIX SEMINARY



Doctor of Ministry Program Summary of Assessment 2013-15

Assessment Results Prior to the current assessment, DMin was assessed annually. Due to the current two-year cycle, more data was received for analysis. The analysis has been helpful in demonstrating the need for a required Research Methods course in order to better prepare DMin students for their Project. This course is in process and will likely be offered in 2016. In addition, the assessment process revealed the need for DMin course papers to consistently include a biblical/theological rationale. This will be required in every course paper. For this current cycle, having more data has been helpful but results demonstrate the importance of the next cycle – the planned for four-year cycle of DMin assessment. Four years should give the quantitative data we need to insure more accurate assessment, given the structure of the DMin degree. While two of the three student learning outcomes were not met – see below – the analysis and use of the results is proving effective in improving the program for student learning success as noted in the first paragraph of this document. (For detailed information, see the individual Degree Program Assessment Charts.) The DMin has three student learning outcomes (SLOs). Following are the results per outcome. The benchmark for success is 80%:

1. Produce biblical and theological solutions to challenges in ministry culture. 45% 2. Demonstrate advanced ministry knowledge in at least one of four areas of ministry:

Preaching and Teaching, Ministry Leadership, Pastoral Care and Christian Formation. 50% 3. Exercise library and research skills appropriate for increasing ministerial competence. 100%

Highlights of the results are as follows:

Strengths: A juried review of the DMin Project was used for the first time. While lengthy, it appears to be a

good measure of all the student learning outcomes. Student feedback indicates that students are generally satisfied with what they are learning in the

program. Action Items are linked to the budget process as necessary (see below). Action Items are linked to the Strategic Priorities they support and fulfill.

Improvements to be addressed: SLO #1 – Due to low assessment results, the DMin Director and Faculty agree that it will be

important to require that every DMin Project and DMin course paper include biblical and theological rationales for their subject.

SLO #1 – The DM711 paper, used in a juried review, will be re-created as a capstone assignment which will help to better measure this outcome.

SLO #3 – Previously, library and research skills were taught in one lecture of a required course that covered many topics. A new required course will be created – Research Methods – which will address the student’s need for better understanding and skill in the area of library and research.



Budget Implications Results do not indicate any needed budget changes for 2015-17. Strategic Plan Implications It is important to note that the Action Items from the degree program assessment plans have been aligned to specific Strategic Priorities to show the relationship between assessment and strategic planning and therefore the effectiveness of the institution itself.


It is our expectation that while the Strategic Priorities cover all aspects of the seminary, it is conceivable that results from degree program assessment could engender significant change or addition to the Strategic Priorities. In the meantime, the Action Items support and fulfill the Strategic Priorities (see DMin Assessment Charts). Direct and Indirect Measures The direct and indirect measures for DMin assessment are appropriate in kind and number for the student learning outcomes. As noted under “Improvements”, the DM711 paper will be revised to better assess SLO #1. Assessment Process The Director of Assessment collects all results from the Direct and Indirect Measures, making a brief, initial analysis. The degree program assessment data and this initial analysis are reviewed by the DMin Director and Faculty in late August. The Faculty makes a comprehensive analysis and suggests change for the future. Revised plans are finalized by the Director of Assessment and presented to the Assessment Committee for final approval. Following approval, the Director of Assessment creates an action list and follows through on all action items for the year.

Conclusion While the results were generally not positive, the information gained and the planned improvements have been beneficial for the DMin program. The upcoming four-year cycle of assessment should provide even more accurate results, placing the DMin Program assessment in a stable position.



Academic Program: Doctor of Ministry

Program Purpose: The purpose of the Doctor of Ministry degree program is to prepare students currently in vocational ministry with the highest level of expertise in the biblical and theological application of God’s truth to their contextual ministry setting in the church and para-church. The Ph.D. degree, by comparison, purposes primarily to equip students to engage in scholarly research and teaching.

Assessment Plan (“beginning the loop”) Assessment Record (“closing the loop”) Intended Program Outcomes Criteria & Procedures Assessment Results Analysis and Use of Results Planning and/or Budget Action Responsible for Follow-up 1. The student should be able to produce biblical and theological solutions to challenges in ministry culture.

Aligned to: ATS Standard E.2.1

Direct Measures Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Project DM711 Contemporary Theological Issues Paper Indirect Measure Exit Survey and Student Survey (every two years) that

includes the question “The D.Min. program surfaced challenges in my ministry and gave me solutions for the challenges.” Success will be achieved if 80% of the graduates respond by “strongly agree” or “agree” to the statement.

Direct Measure Summary – 45 % achieved benchmark or higher (summary of weighted measures) Project – 50% DM711 Paper – 40% Indirect Measure Exit Survey not conducted. Student Survey – Question was changed to two questions: 1) “To what extent do you believe the DMin program has made you aware of new challenges in your ministry? 2) To what degree do you think your studies in the DMin program are helping you to develop new biblical and theological solutions for new and existing ministry challenges? Question #1 – 100% Question #2 – 91.7%

Results indicate the SLO was not met. Intended program outcome not met possibly due to small sample size in DMin Project measure (2). However, it’s interesting to note that student opinion indicates they are receiving what they need in the DMin program to meet this outcome. After discussion, it’s clear that more can be done to ensure that students better understand how to “produce biblical and theological solutions to challenges in ministry culture.” DM711 Paper – The poor results may have to do with the fact that the paper wasn’t geared towards measuring this outcome. This course was restructured prior to these assessment results but after the previous DM711 papers were evaluated. The new DM711 course will have a formal and traditional approach with a strong focus on contemporary theological issues. A capstone course assignment will be created that will help to measure this outcome. DM711 will next be offered in June 2016. Indirect results – In most cases, students’ perceptions don’t match the direct assessment results It will be interesting to observe whether the result is the same in the next cycle. It may be because of a smaller sample size on the Project.

ACTION: Strategic Priority #3 “Develop . . . academic programs” and Strategic Priority #5 “…continuous improvement through assessment” - 1) Require of every DMin Project a biblical/ theological rationale for study – in the proposal or a separate chapter of the Project. 2) Require biblical/ theological rationales in practical ministry course papers as well as in biblical/theological papers proper. ACTION: Strategic Priority #3 “Develop . . . academic programs” and Strategic Priority #5 “…continuous improvement through assessment” - Work with DM711 Professor to create a capstone assignment that will help to measure this outcome. There are no budget implications at this time.

Responsible Party: Chip Moody Follow-up Date: May 31, 2016 Responsible Party: Chip Moody Follow-up Date: December 31, 2015

2. The student should be able to demonstrate advanced ministry knowledge in at least one of four areas of ministry: Preaching and Teaching, Ministry Leadership, Pastoral Care and Christian Formation.

Aligned to: ATS Standard E.2.1 and E.2.2

Direct Measures Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Project Indirect Measure Exit Survey and Student Survey (every two years) that

includes questions addressing this outcome. Success will be achieved if 80% of the graduates respond by “strongly agree” or “agree” to the statement.

Direct Measure Summary - 50% achieved benchmark or higher (summary of weighted measures) Project – 50% Indirect Measure Exit Survey not conducted. Student Survey – 100% The question was “How satisfied are you that you are gaining advanced ministry knowledge (knowledge that you didn’t have before beginning the DMin program) in at least one of the following . . .”

Results indicate the SLO was not met. Sample sizes are simply too small to measure this outcome adequately. However, it’s interesting to note that student opinion indicates they are satisfied in their receipt of “advanced ministry knowledge”. The cycle timeline for this program should allow for a sufficient sample size during the next assessment. Indirect results – In most cases, students’ perceptions don’t match the direct assessment results It will be interesting to observe whether the result is the same in the next cycle. It may be because of a smaller sample size on the Project.


2015

-17

Pho

enix

Sem

inar

y

Pag

e 13

5 C

ompr

ehen

sive

Ass

essm

ent P

lan

3. The student should be able to exercise library and research skills appropriate for increasing ministerial competence.

Aligned to: ATS Standard E.2.2 and E.2.4

Direct Measures Juried Reviews Success will be achieved if 80% of students receive a “meet” or “exceeds” for this outcome. Project Indirect Measure Exit Survey and Student Survey (every two years) that

includes questions addressing this outcome. Success will be achieved if 80% of the graduates respond by “strongly agree” or “agree” to the statement.

Direct Measure Summary - 100% achieved benchmark (summary of weighted measures) Project – 100% Indirect Measure Exit Survey not conducted. Student Survey – 25% The question was “If you are working on your dissertation, to what degree are you satisfied that you have the library and research skills appropriate for increasing your ministry competence?”

Results indicate the SLO was met. While results show that the SLO was met, only 25% of those students who responded are satisfied with their library and research skills. (The research component of the DMin program has been handled as one lecture period of DM711.) In addition, while the juried review results of the DMin Project indicated a 100% result, it’s clear from the jury’s comments that further work needs to be done in this area, specifically: “Research skills need to include research design; a course in research design should be required.” The DMin Director was already on track to create a required Research Methods course. These results confirm that it is indeed a necessary addition to the DMin curriculum. Indirect results – In most cases, students’ perceptions don’t match the direct assessment results It will be interesting to observe whether the result is the same in the next cycle. It may be because of a smaller sample size on the Project.

ACTION: Strategic Priority #3 “Develop . . . academic programs” and Strategic Priority #5 “…continuous improvement through assessment” - Create a Research Methods course that will be a required part of the curriculum ready for offering at least by 2017. There are no budget implications at this time as the research course will be offered as a regular part of the curriculum already budgeted. If this course is offered earlier (perhaps in 2016 as desired by the program director) student fees will cover the course on a cash in-cash out basis.

Responsible Party: Chip Moody Follow-up Date: June 1, 2016

2013

-15

Pho

enix

Sem

inar

y

Pag

e 13

6 C

ompr

ehen

sive

Ass

essm

ent P

lan

Areas of Assessment

Evidence of Assessment

STRENGTHS AND/OR WEAKNESS OF CURRENT PLAN IMPROVEMENTS TO BE IMPLEMENTED Effectiveness

Was each step accomplished effectively? Logic

Was each step logically rigorous? Comprehensiveness

Was each step addressed in a comprehensive way? Action Steps

Based on the foregoing outcomes, what change should be made?

Person(s) Responsible to Implement Change

Who will take action?

Step 1 – Determine and refine desired outcomes

A major revision of outcomes was completed in 2012-13 (based on HLC, ATS and PS input). During the 2013-15 period, only minor revisions were made to three outcomes. The MAC program and the MABTS were added during this time and appropriate outcomes were created by the Faculty.

As degree program assessment was conducted in 2013-15, the newly revised outcomes were found to be well-suited for each program.

The student learning outcomes for the new MAC and MABTS were created by faculty using their professional judgment and knowledge of the new degree programs.

The student learning outcomes for the new degree programs were created by comprehensively considering each degree program -- it’s purpose and curriculum.

No further action needed at this time.

Step 2 – Identify and enrich assessment instruments

Based on 2012-13 assessment, a major thrust of 2013-15 was to be the reduction of the number of direct measures throughout the degree programs (also a suggestion of HLC and ATS) for the purpose of greater sustainability and strategic effectiveness of each instrument. While progress has been made, the securement of a consultant to specifically help with this process fell through due to the consultant’s inability to fulfill the contract. In spite of this, the Assessment Committee has been strategic in the use and reduction of direct measures.

As noted above in “Step 1 – Effectiveness”, the MAC program and the MABTS were added during this time. Appropriate direct measures for each outcome were created or secured by the Faculty keeping in mind our need for sustainability.

Direct measures were appropriately aligned to new degree program student learning outcomes by the faculty using their professional judgment and knowledge of the new degree programs.

Consistent attention has been given to the effectiveness of the direct measures and if ineffective, they have been replaced (e.g. a juried review of a doctrinal statement in the MAM was replaced with a juried review of a TH503 paper.)

The direct measures for the new programs took into consideration each program’s purpose and curriculum as well as the need for sustainability moving forward.

During assessment review, a member of the Assessment Committee suggested that the Graduate and Faculty juried review of students should be reviewed for its effectiveness since results differ from other instrument results.

ACTION: Strategic Priority #7 “…continuous improvement through assessment” – In the 2015-16 academic year, the Assessment Committee will review the Graduate and Faculty juried review instruments.

Responsible Party: Roma Royer Follow-up Date: March 31, 2015

Step 3 – Implement data collection

Data was systematically collected for assessment purposes.

Data collection was based on the Assessment Calendar for Degree Programs, which is a comprehensive list of all instruments used annually.

Data collection was comprehensive in scope.

The Assessment Director has a basic system for collection that is working well.


Step 4 – Review and interpret results

All results are reviewed by the Assessment Director and the Assessment Committee or Faculty. Pertinent revisions are recorded in the minutes and applied to the appropriate degree program assessment plan by the Assessment Director. In addition, results are applied to other areas not directly tied to the assessment plans themselves.

In 2012-13 cycle, weights were added to each direct measure in order to enhance the accuracy of the summary of results. This has continued to be helpful as results are weighted based on the type of instrument.

Results were reviewed and interpreted by the Faculty and Assessment Committee. More than one person is involved with each instrument and reviewers consist of Faculty related to the particular discipline as well as those who are not, in order to foster greater objectivity.


Assessing the Phoenix Seminary 2013-15 Comprehensive Assessment Plan Purpose: To ensure that the Phoenix Seminary Comprehensive Assessment Plan is effective in accomplishing its purpose.

Process: The Assessment Committee will review the assessment plan and process every two years, usually early in the fall semester, and make recommendations to the faculty and administration for change as appropriate. Once approved, changes will be incorporated into the next year’s comprehensive assessment plan.

Procedure: The steps of the Feedback Loop for Continuous Improvement, the essential element of the comprehensive assessment plan, will be evaluated along with the overall plan structure.

2015

-17

Pho

enix

Sem

inar

y

Pag

e 13

7 C

ompr

ehen

sive

Ass

essm

ent P

lan

Areas of Assessment

Evidence of Assessment

STRENGTHS AND/OR WEAKNESS OF CURRENT PLAN IMPROVEMENTS TO BE IMPLEMENTED Effectiveness

Was each step accomplished effectively? Logic

Was each step logically rigorous? Comprehensiveness

Was each step addressed in a comprehensive way? Action Steps

Based on the foregoing outcomes, what change should be made?

Person(s) Responsible to Implement Change

Who will take action?

Step 5 – Determine change needed to improve outcomes

2013-15 is the first cycle of a multi-year calendar of assessment of degree programs. While it may be too early to ascertain the effectiveness of a multi-year cycle process, it is clearly more viable than assessing each program every year -- addressing concerns of accreditors regarding sustainability.

The multi-year cycle has enhanced the number of samples available for each instrument.

The instrument and its results were reviewed and considered in light of the specific outcome as well as the degree program as a whole.

Change resulting from assessment is included in the Action Items document for follow-up as well in “Summary of Change Resulting from Assessment” which is included in the Comprehensive plan and presented to the Board.


Step 6 – Incorporate, as necessary, into Strategic Plan

Implementing change to strategic planning is crucial to the assessment process. Therefore, Phoenix Seminary has a specific plan to insure that changes resulting from assessment are linked directly to strategic planning. In addition, there is a master calendar which coordinates assessment, strategic planning, budget and board meetings.

Since the Strategic Priorities are well-developed and cover all aspects of the seminary, it is less likely that results from degree program assessment will regularly engender significant change or addition to the Strategic Priorities.

Action Items from the degree program assessment plans have been aligned to specific Strategic Priorities to show the relationship between assessment and strategic planning and therefore the effectiveness of the institution itself. While there has not yet been significant change or addition to the Strategic Priorities through degree program assessment, the Action Items support and fulfill the Strategic Priorities.

See previous columns. No further action needed at this time.

Step 7 – Implement necessary budget change

Each step of assessment took into consideration any needed changes to the budget. In most cases, these changes were minimal.

A structured budget process automatically requires an informed connection to assessment results.

In addition, a master calendar coordinates assessment, strategic planning, budget and board meetings.

See previous column. See previous column. No further action needed at this time.

Step 8 – Implement improvements

Due to the size of the seminary and lack of bureaucracy, improvements are usually implemented quickly and effectively. Following are a few examples: 1) the Faculty will begin to meet immediately regarding revisions to the Standardized Bible & Theology Test; 2) A new required Research Methods course is in process for the DMin program; and 3) all DMin course papers will now be required to have a biblical/theological component.

The multi-year timeline, as noted earlier, will bring greater effectiveness to implementing improvements.

See previous column. Further comprehensiveness will be addressed by acting upon the concerns mentioned in the effectiveness column.


Step 9 – Evaluate assessment plan and process

Conducted each assessment cycle by the Assessment Committee.

This assessment chart is a logical method of approach.

All assessment plans and processes are evaluated by the Assessment Committee.


Overall structure of the comprehensive plan

The overall structure of the plan is sound and effective.

See previous column. The structure of the plan is comprehensive in that it utilizes the Assessment Committee, Faculty, and Assessment Director to accomplish the requirements of the plan. It also addresses all areas of the seminary including academics and operations.


2015

-17

Pho

enix

Sem

inar

y

Pag

e 13

8 C

ompr

ehen

sive

Ass

essm

ent P

lan

Program Outcome (The Student will be able to . . .)

Assessment Action Items Person Responsible

Date Due

MAM 1. Know and apply Scripture using proper hermeneutical principles.

2. Articulate sound doctrine according to historic Christian orthodoxy

Standardized Bible & Theology Test

The Assessment Committee will review the possibilities of improving the test so that it measures what it should.

Royer/AC 12/31/15



TH503 Doctrinal Statement

TH503 Research Paper should be considered to replace the TH503 doctrinal statements.

Royer/AC 11/30/15 COMPLETE



TH503 Doctrinal Statement

Add Juried Review of TH503 Paper to Criteria column should it be approved by the Assessment Committee.

Royer 12/31/15 COMPLETE

MAM 4. Demonstrate healthy relationships with God and neighbor. Faculty Graduate Review

Assessment Committee should review the use of the Faculty and Graduate review measure.

Royer/AC 5/1/16 COMPLETE

MDiv Core 1. Know and apply Scripture using proper hermeneutical principles.


Standardized Bible & Theology Test

Work on changes to the Bible & Theology test so it’s a learning tool for the students and therefore hopefully presents more meaningful assessment results. Budget implication – consider paying a stipend to Faculty who make these detailed revisions. Not accomplished. See item below in MAM assessment

Royer/AC Fall 2014 NOT COMPLETE

MDiv Concentratons

SLOs #5 and 6 per concentration 30‐Hour Review Remove 30‐hour review as direct measure Royer 5/1/15 COMPLETE

MDiv CF 5. The student should be able to counsel according to biblical principles.

New measure A direct measure needs to be developed for this SLO to be used in 2015‐16 assessment.

Smith 5/1/15 COMPLETE

MDiv CF 6. The student should be able to design and assess programs for biblical counsel and care.

N/A Review direct (and indirect) measures for MDiv CF in light of small enrollment. AC 9/1/15 COMPLETE

MDiv LD 6. The student should be able to demonstrate the ability to provide biblical leadership.

MS501 Paper Change MS501 Paper to MS509 Paper given the merging of courses.


DMin 1. Produce biblical and theological solutions to challenges in ministry culture

DMin Project Require of every DMin Project a biblical/ theological rationale for study – in the proposal or a separate chapter of the Project. 2) Require biblical/ theological rationales in practical ministry course papers as well as in biblical/theological papers proper.

Moody 5/31/16

DMin 1. Produce biblical and theological solutions to challenges in ministry culture

DM711 Paper Work with DM711 Professor to create a capstone assignment that will help to measure this outcome.

Moody 12/31/15

DMin 3. Exercise library and research skills appropriate for increasing ministerial competence.

N/A Create a Research Methods course that will be a required part of the curriculum ready for offering at least by 2017.

Moody 6/1/16

Comprehensive Assessment Plan

Step 2 – Identify and enrich assessment instruments In the 2015‐16 academic year, the Assessment Committee will review the Graduate and Faculty juried review instruments.


2013-15 Assessment Action ItemsDegree Program Assessment and Comprehensive Plan Assessment

2015

-17

Phoe

nix

Sem

inar

y

Pag

e 13

9 C

ompr

ehen

sive

Ass

essm

ent P

lan

prior year assessment history - ps.edu€¦ · prior year assessment history phoenix seminary...

Documents