2011/12 student learning assessment report · 2011/2012 mha student learning assessment report ......

1

2011/12 Student Learning Assessment Report

October 2012

2

2011/2012 MHA Student Learning Assessment Report

Introduction The Master of Health Care Administration (MHA) program updated its Student Learning and Assessment Plan (SLAP) for the 2011/2012 academic year (AY) taking into consideration feedback provided by the Student Learning and Assessment Committee (SLAC), the program’s faculty, as well as changes the program was undertaking through the new curriculum roll out. A copy of the program’s current plan is provided in the Appendix.

MHA Program & Learning Objectives The MHA faculty met and agreed on a combination of program and learning outcomes as a foundation for assessing student learning: Program Objectives—

1. The program will admit students that fit our target student (early to mid careerist) 2. Students will successfully complete the culminating experiences of the curriculum: Administrative Internship (MHA 640/642)

and Capstone (MHA 649). 3. MHA graduates will be satisfied with their academic investment at DMU/MHA

Learning Objectives—

4. Students will achieve the mid-careerist level on the 9 NCHL competencies the faculty identified as key learning outcome measures for the MHA curricula.

Measures and Data Collection

The MHA student Learning Portfolio (Direct), the annual Graduate Survey (Indirect), admissions data (Direct) and Capstone and Internship performance data (Direct) were identified as the primary measures of program objectives and student learning. The program director, program assistant, and special project coordinator for CHS identified Feedback Server as a tool to collect the portfolio data.

3

Learning Portfolio (Direct). The following instruments were developed within Feedback Server to allow for data collection:

• NCHL Competency Assessment – Initial and post-program (Student completes) –http://feedback.dmu.edu/fs.aspx?surveyid=9f850f0da494-f46997ee870cdd165df

• Preceptor Internship Evaluation (Preceptor and one other individual from host organization completes) - http://feedback.dmu.edu/fs.aspx?surveyid=027e30861d74198b7f81b1c2b3bc2bc

• Faculty Evaluation of Internship Portfolio (Faculty advisor completes) - http://feedback.dmu.edu/fs.aspx?surveyid=cf5bc365dd84bd0a536ead9967657b1

• Preceptor Capstone Evaluation (New data added to 2011/12 SLAP. Capstone preceptor completes) - http://feedback.dmu.edu/fs-Capstone.aspx

Links to the first three forms are provided on the program’s AdvisorConnection (MHA 600) faculty student advising Angel site; the Capstone preceptor evaluation is found within the Capstone Angel course. Directions on how to use each form are provided to the user. Returned forms are collected and the submitted data is maintained in the Feedback Server database. Annually, data is downloaded into Microsoft Excel for analysis and reporting.

Graduate Survey (Indirect). Each year, members of the graduating class are emailed a link to a survey in Feedback Server asking them to provide their summative views of the MHA program support, curriculum, faculty, faculty advisor, staff, etc. This year the survey was sent to graduates as soon as their diploma request was received versus waiting until the spring to survey the class. This change was made to increase response rates. Program faculty are most interested in the graduates view of the overall value of their academic experience. The four questions selected for review follow in the table below. A full copy of the survey results is available in the Program office. Admission Rubric and Decision Spreadsheet (Direct). The MHA program had two admission cycles in the 2011/12 AY. For each cycle, program faculty used an admission rubric they developed that can award each applicant up to 100 points. Once all the files are reviewed with the rubric, the data is combined into a decision support spreadsheet. The spreadsheet is used by the faculty to determine who the program will extend offers of admission to. Once admission decisions are finalized, program faculty run descriptive statistics on the accepted versus rejected groups and compare to see if their decisions fit the profiled student.

4

Assessment Results The following table summarizes the programs student learning goals, measures, and outcomes: Program Objective

Measure Tool Direct/Indirect

Benchmark

Actual Data & Date Collected

Who uses the data?

Narrative Summary

Action Planned/ Taken

1. The program will admit students that fit our target student (early to mid careerist).

Admission demographic data (decision spread-‐sheet and data analysis) pulled from the admission rubric evaluation work history, academic rigor, communication skills, and work ethic (D)

• Average: o 5 years of work experience

o 3.25 cumulative GPA

o Rating of 3 (75%+) or “Good” for admission essays

o Rating of 3 (75%+) or “Good” for references

• For prior 2 admission cycles (October and May):

• Average age= 29.4 yrs • Average years of experience=6.7 • Cumulative incoming GPA= 3.42 • Average essay score for admitted applicants:

o October ‘11 = 88% o May ‘12= 78%

• Average reference score for admitted applicants:

o October ‘11 = 93% o May ‘12= 78%

• Reviewed 65 applications/52 admission offers= 80% acceptance rate

MHA Faculty and Admiss-‐ ions staff

Goal Met

Continue using developed admission process. Work to further define each criteria within rubric to add more consistency/validity to our review/scoring process.

2. Students will successfully complete the culminating experiences of the curriculum: Administrative Internship (MHA 640/642) and Capstone (MHA 649).

Comparison of student enrollment versus successful completion (D)

• The program targets a completion rate of 90% for these two required courses.

Cumulative data from Summer 2011 – Fall 2012.

• Internship = 100% pass rate (35 total; 23 pass, 12 still in process).

• Capstone = 95% pass rate (62 total; 59 pass, 2 fail, 1 withdrawal)

MHA Faculty and Capstone course faculty

Goal Met

Phasing out the Internship & Capstone courses with new curriculum. Will update this objective/measure to the Field Based Learning course (MHA 749) for next years plan.

5

Calculating average GPA of student completing the Capstone course (D)

• The program target is set a an average GPA of 3.0 or higher

• 3.53 GPA MHA Faculty and Capstone course faculty

Goal Met

�

Learning Objective


Benchmark


Who uses the data?

Narrative Summary


3. Students will achieve mid-‐careerist level on the 9 key NCHL competencies that the faculty identified as key learning outcome measures for the MHA curricula.

MHA “9 Key” NCHL Competencies— (D)

Level “3” or Mid-‐careerist level identified by NCHL—

Cumulative data Fall 2011 – Summer 2012;

N = 64 Students. Bar Chart Scale: 1 = student pre program assessment on NCHL competencies

2= student post program assessment 3 = mentors assessment post program (capstone and internship preceptors plus academic advisor)

6

These 9 key competencies are listed (right) along with the established benchmark (Achievement Level 3) identified by NCHL for a mid-‐careerist.

1. Analytic Thinking Level 3: Recognizes multiple relationships, makes causal links.

Bar Chart Scale: 1 = student pre-‐ program assessment on NCHL competencies

2= student post-‐ program assessment

3 = mentors assessment post-‐ program (capstone and internship preceptors plus academic advisor)

MHA Faculty/ Course faculty supporting competency

Goal Met

Continue to monitor as new curriculum is rolled out. Implement new measures/data collection from revised curriculum map and new ePortfolio tool.

�

Learning Objective


Benchmark


Who uses the data?

Narrative Summary


2.7 3.6 3.4

0

1

2

3

4

1 2 3

Analytical Thinking

7

MHA students will achieve competency at the mid-‐careerist level by the end of their academic program. Student self-‐reported pre and post-‐achievement will be validated by triangulating student ratings with preceptor and faculty ratings from the Administrative

2. Financial Skills Level 3: Understands impact of reimbursement models

Bar Chart Scale: 1 = student pre program assessment on NCHL competencies

2= student post program assessment

3 = mentors assessment post program (capstone and internship preceptors plus academic advisor)


Goal Met

�

Added note: Course coordinator redesigned the two finance courses during 2010/11 AY. The goal was not met last year, but improved and was met this year.

�

Internship Learning Portfolio and the Capstone course evaluations.

3. Interpersonal Understanding

Level 3: Commits to Understanding Others


Goal Met

� � �

Learning Objective


Benchmark


Who uses the data?

Narrative

Action Planned/

1.8 3 3.6

0

1

2

3

4

1 2 3

Financial Skills

3.4 4.1 4.1

0

1

2

3

4

5

1 2 3

Interpersonal Understanding

8

Learning Objective


Benchmark


Who uses the data?

Narrative Summary


4. Professionalism Level 3: Maintains social accountability


Goal Met

� � �

5. Self Development Level 3: Considers the impact one has on others





Goal Met

� � �

2.6 3 3.4

0

1

2

3

4

1 2 3

Professionalism

2.7 3.2 3.4

0 1 2 3 4

1 2 3

Self Development

9

Learning Objective


Benchmark


Who uses the data?

Narrative Summary


6. Strategic Orientation Level 3: Aligns organization to address long-‐term health care environment


Goal not met

�

Added note: We were aware of this deficiency. A new strategy course is in development and will debut in January 2013.

7. Collaboration Level 3: Solicits input, values others’ expertise





Goal Met

� � �

1.9 2.7 2.9

0

1

2

3

4

1 2 3

Strategic Orientation

10

7. Collaboration Level 3: Solicits input, values others’ expertise





Goal Met

� � �

Learning Objective


Benchmark


Who uses the data?

Narrative Summary


8. Communication Skills

Level 3: Makes persuasive oral presentations





Goal Met

� � �

2.6 3.5 3.3

0

1

2

3

4

1 2 3

Communication

11

9. Accountability Level 3: Demands high performance from self and others





Goal Met

� � �

Program Objective


Benchmark


Who uses the data?

Narrative Summary


4. MHA graduates will be satisfied with their academic investment at DMU/MHA.

2011 Graduate Survey (ID)

80 % Agree or “Yes” Results of the 2011 Graduate Survey: Survey distributed to 70 graduates; 54 responded. Response rate = 77%.

Response rate Increased from 38% the previous year.

MHA Faculty Goal Met

Continue with survey dissemination process. Strive to increase participation rates each year.

o Did the program help you further develop competency in Health Care Administration?

This year (2011/12): 100%

Last year (2010/11 data):

100% (10) Yes

MHA Faculty 2011/12 Goal Met 2010/11 Goal Met

Program faculty formally introduce/address the competency model within the 3 new Exec. Residencies (specifically

o Did the courses offered in the program adequately

This year (2011/12 data):

87% Yes


93% (13) Yes

MHA Faculty 2011/12 Goal Met

Professional Development Seminar I, 2, and

2.8 3.9 3.9

0 1 2 3 4 5

1 2 3

Accountability

12

o Did the courses offered in the program adequately prepare you for your desired career?


87% Yes 13% No


93% (13) Yes 10% (1) No


Professional Development Seminar I, 2, and 3); are in the process of redesigning the Learning Portfolio assignment using

o Did your experience in the MHA program help achieve your goal?


91% Yes 9% No


100% (14) Yes


a new ePortfolio tool and are purposely tying the competencies to the curriculum and assessments;

o Did the program help you develop a career plan or provide career direction?


81% Yes 19% No


80% (8) Yes 20% (2) No


And, added a career dinner/panel discussion and exercise to Exec Residencies.

13

Discussion Program Objectives As the table above suggests, the MHA program faculty have met the first objective by successfully admitting individuals that fit our target of early to mid-careerist student profile. Though this objective is not a direct measure of student learning, the program’s admission rubric and new decision support spreadsheet have greatly aided the faculty in making tough admission decisions and assisted the program in achieving the program objective of admitting targeted early-to mid-careerists. Additionally, program faculty are reporting that the quality of students is improving. Second, student completion rates for the Administrative Internship and Capstone courses exceed the goal of 90% demonstrating that the work (i.e. limiting course to a single term, streamlining processes on AdvisorConnection, etc.) put into improving these core courses has resulted in the achievement of a second program objective measured by completion rates. In fact, historical data suggests that changes to the Capstone course (i.e. limiting enrollment to a single term, reducing section sizes, adding due dates throughout the term, changing course from pass/fail to a letter grade, etc.) has greatly improved the success rate of our students over the last four academic years and has stabilized above the target (table follows): Table. Capstone course completion rates 08/09—11/12

Status 2008/09 2009/10 2010/11 2011/12 Total registrations 53 50 48 62 Pass 41 (77%) 36 (72%) 47 (98%) 59 (95%) Not pass 12 14 1 2 fail, 1

withdrawal The Administrative Internship was phased out at the end of the Spring 2012 term out so no additional changes will be made. The Capstone course, however, continues to be offered to support students admitted prior to 2012. This course will eventually be replaced with the new Field Based Learning course and increase from a two to a three credit experience. Changes made to improve student learning in the Capstone course will be incorporated into Field Based Learning. The final program objective of overall graduate satisfaction with their academic investment was measured using the MHA Graduate Survey of the graduating class of 2011 (54/70 = 77% response rate). The change made to the administration of the survey (throughout

14

the year versus all at once in the spring) resulted in a dramatic improvement in the response rate (2010/11 results: 15/38=39% response rate). Survey results suggest that the strengths of program include:

o Student development on the NCHL competencies o Student preparation for their desired career o Student achievement of personal goals o Student development a career plan or career direction

Graduates completing the 2012 MHA Graduate Survey were overall satisfied with their investment in the program. Though the program did achieve its benchmark of 80% satisfaction, we believe there is room for improvement in the area of connecting student achievement to competency development and developing a career plan or goal. In response, we have developed several curricular interventions to improve this outcome. For example, there will be an increase in face-to-face time between students and advisors during the three required, on-campus, Executive Residencies for students entering the program Fall 2012 and beyond. Some of this time will be used to formally introduce the program’s competency model to students and connect the model to the curriculum (and assessments) and to the Learning Portfolio assignment. In addition, career planning activities will also be introduced in the residencies. The first residency was offered September 2012. In addition to career planning activities within the residency course Professional Development Seminar I (MHA 801), a career planning dinner was incorporated to allow students an opportunity to hear about career development and advancement from administrators working in the field. Preliminary feedback was very positive, and full results will be reported next year. Student Learning For the second year, electronic data was collected throughout the academic year from the pre/post competency surveys providing an opportunity to analyze learning outcomes. Dr. Ann York analyzed the data using Microsoft Excel and produced the bar charts in the summary table above. The first bar labeled “1” represents the mean score of student’s pre-program self assessment on NCHL competencies; the second bar labeled “2” represents the mean score of the student’s post-program assessment; and, the third bar labeled “3” represents the mean score of the mentors assessment post-program (capstone and internship preceptors plus academic advisor) The MHA curriculum met the goal of level 3 (mid-careerists) with all 9 key competencies with the exception of the following: Strategic Orientation (Mean=2.9).

15

This demonstrates improvement over the 2010/2011 AY report. Last year, we were below target on both Financial Skills and Strategic Orientation. As a result of redesign and increased rigor in Finance I and II courses under the direction of Dr. Richard Belloff, the goal has been met this year. Similar improvement in the Strategic Orientation scores are expected in the next few years as a new course entitled, Strategy Formulation & Implementation (MHA 742) will be launched and incorporated into the required experience into the second on-campus executive residency. Dr. Richard Belloff will lead the course designed specifically to grow student’s Strategic Orientation competence. In addition, the theme of Strategic Orientation will be reinforced within other courses across the new curriculum. Student Learning Assessment Process The 2011/12 Student Learning and Assessment Report for the MHA program provided a second look at direct measures of student learning related to the 9 Key Competencies identified by program faculty. Additional indirect measures of learning were also reviewed. Data were successfully collected from multiple sources and maintained throughout the year allowing for analysis and reported outcomes, and allowed a comparison to the inaugural year of data collection. This comparison revealed a gain or maintenance in meeting benchmarks, with no loss of progress on key indicators. The student Learning Portfolio and Feedback Server were identified as areas in need of improvement. Collecting learning outcomes data with these sources can be cumbersome and inconsistent. Therefore, beginning in Fall 2012, students entering the program will be required to purchase the e-portfolio tool Live Text™. Current students have the option to purchase it, but are not required. The transition to this tool will take time, but ultimately, the data will be more consistent regarding learning outcome data, artifacts, and surveys.

Recommendations Program Objectives/Student Learning In a review of the results provided above and comments provided on the survey, recommendations to improve program objectives include:

• Continue to formally integrate the NCHL competencies and the desired achievement levels within courses and residencies.

Communicate to students how the curriculum is developed to support their achievement. Include opportunities at multiple points throughout the student’s tenure with the program for a review of their achievement and a plan to reach the desired

16

achievement levels (using the newly designed Learning ePortfolio). Time will be allocated for this within each of the three Professional Development Seminars (part of the Executive Residencies) the students are required to attend.

• Executive Residencies will provide an opportunity for students to articulate their career goals, learn how to effectively “manage” a career, and develop a plan to achieve those career goals.

Finally, the program will introduce the new course on Strategic Orientation in Spring 2013 to improve student learning outcomes. Student Learning Assessment Process The MHA program will continue to evaluate the process and tools to assist us in identifying, collecting, maintaining and reporting measures of learning. Students will be informed of the effort, rationale given for why we are using certain tools and surveys, and invited to provide feedback on how we can make the process more meaningful and seamless to the learner. Outcomes data will be shared with students as well as other program stakeholders in a transparent manner, and improvements integrated in the curriculum and program delivery.

Status of Curriculum Mapping At the request of the DMU Provost, a summary of the Program’s curriculum mapping progress is provided below:

• In April 2010, MHA faculty participated in a competency-based mapping project with the NCHL. All online and classroom-based required course learning objectives were mapped using a series of measures (see below). The results of this mapping exercise were used to redesign the MHA curriculum.

o NCHL competencies, plus achievement level o Bloom’s Taxonomy Domain (Cognitive or Affective) o Bloom’s Taxonomy level (Knowledge, Comprehension, Analysis, Syntheses, Application) o Learning methods used o Assessment methods used

• The new curriculum was mapped to the NCHL competencies (with achievement level) again in the Spring of 2012 in preparation for the Program’s CAHME Candidacy Application. The program used a template provided by CAHME. A copy of “Figure 3: Competency Coverage Across the Curriculum” is provided in the appendix.

17

Once the new curriculum rollout is completed (Fall of 2014), the Program faculty intend to work with NCHL again to remap the curriculum using their mapping product.

Summary The MHA program faculty formally discussed the outcomes of this report, along with the process used to develop, at its annual curriculum retreat in November 2012. A summary of this report will be posted on the DMU webpage and Pulse for program stakeholder’s once reviewed by the dean, SLAC, and DMU provost.

2011/12 student learning assessment report · 2011/2012 mha student learning assessment report ......

Documents