2012/13 student learning assessment report · the target range for the mha program is the...

1

2012/13 Student Learning Assessment Report

October 2013

2

2012/2013 MHA Student Learning Assessment Report

Introduction The Master of Health Care Administration (MHA) program updated its Student Learning and Assessment Plan (SLAP) for the 2012/2013 academic year (AY) taking into consideration feedback provided by the Student Learning and Assessment Committee (SLAC), the program’s faculty, as well as changes the program was undertaking through the new curriculum roll out.

MHA Learning & Program Objectives Learning Objectives—

1. Students will achieve the intermediate level on the nine National Center for Healthcare Leadership (NCHL) competencies the faculty identified as key learning outcome measures for the MHA curriculum. These are measured by student self-assessment, preceptor and advisor assessment, and direct measures from coursework.

a. Financial Skills b. Strategic Orientation c. Communication d. Collaboration e. Accountability f. Interpersonal Understanding g. Analytical Thinking h. Professionalism i. Self Development

Program Objectives—

2. Students will successfully complete the culminating experiences of the curriculum: Field-Based Learning or Capstone course (depending on when student enrolled in the MHA program) as measured by completion rates and cumulative GPA. 3. MHA graduates will be satisfied with their academic investment at DMU/MHA as measured by the Graduate Survey.

3

Measures and Data Collection The NCHL model is a validated set of competencies defining key skills required at the beginning, intermediate and advanced career levels in managing health organizations across selected provider sectors, across disciplines of administration, nursing and medicine and across career levels. Research conducted to support the development of these competencies included both quantitative and qualitative methods. The target range for the MHA Program is the intermediate level appropriate for mid-career professionals. The NCHL competencies were measured through the MHA student Learning Portfolio (Direct), and Capstone or Field-Based Learning performance data (Direct). The student pre self-assessment occurs at the beginning of the MHA program; the student post self-assessment, as well as the preceptor and advisor assessments occur upon completion of the program. Per recommendations from SLAC last year, a rubric was added to more fully assess the level of achievement on the NCHL competencies. The full list of 26 competencies mapped to the curriculum is available in Appendix A. Each competency is rated on a scale of 3, 4, 5, or 6. The following rubric explains accomplishment levels. In the report, the level is reported for each learning outcome.

Competency Basic Intermediate Advanced

3 point scale 1 2 3

4 point scale 1 2 3 4

5 point scale 1 2 3 4 5

6 point scale 1 2 3 4 5 6

Basic skills encompassing knowledge and comprehension of subject matter.

Intermediate skills encompassing application of knowledge to analyze a problem.

Advanced skills encompassing ability to evaluate, judge and synthesize information..

As a result of our report last year, two areas were identified as needing more attention: Financial Skills and Strategic Orientation. Therefore, more resources were put into these areas over the past year. Additional direct measures of the two were added to this report, as well as several other direct measures. These additional direct outcomes also meet the recommendations from SLAC last year. Student satisfaction with MHA program was measured through the Graduate Survey (Indirect). Historical data is provided when available.

4

Assessment Results

Objective

Measure Tool Direct/Indirect

Bench-‐ mark

Actual Data and Date Collected

Who uses the data?

Narrative Summary Action(s)

Planned/Taken that Address Curricular Changes

1. Learning Objectives: 1.a. Financial Skills: Understands the impact of reimbursement models in healthcare; explains connections, assesses payment systems; develops incentives

NCHL Competency assessed by 1)Student Pre 2)Student Post 3) Preceptor and Advisor Post (Direct)

Levels 2-‐4 on 5 point scale (inter-‐mediate) Blooms level 2-‐4 (external)

AY 13

AY 12

AY 11

Pre 1.6 1.8 1.4 Post 2.8 3 2.9 Prec 3.5 3.6 NA

Program and Student

Met at the lower intermediate level for students, and the mid-‐intermediate level for preceptors and advisors. Both show a measurable gain over the student pre-‐survey

Increased rigor was added to the Finance I and II courses over the past 2 years. To assist the student in acquiring intermediate level skills, narrated slides, practice problem sets, and blended sessions (see below) were added. The full effect of this impact may not be felt for another 2-‐3 years.

Final examination MHA 625 Finance I (Direct)

90% pass rate at >70% (inter-‐mediate) Blooms level 1-‐4 (internal)

Pass rate: 84% Mean score: 82%

Program Not met. The professor has moved from a purely online format to blended format to increase student / faculty interaction.

To assist with student understanding, blended learning sessions using classroom and Adobe Connect for real time connection with the professor were added. More emphasis will continue to be placed on student professor interaction prior to testing. Will continue to monitor final exam scores.

Final Examination MHA 630 Finance II (Direct)

90% pass rate at >70% (inter-‐mediate)

Pass rate: 96% Mean score: 89%

Program Goal met. Indicates growth in finance knowledge as the student

Same methods as above.

5

Blooms level 2 -‐ 6 (internal)

progresses in program.

1.b. Strategic Orientation: Aligns organization to address long-‐term health care environment; performs market analysis; aligns strategy, structure and people; develops organizational plans.


Levels 2-‐3 on 4 point scale: (inter-‐mediate). Blooms level 3-‐5 (external)

AY 13

AY 12

AY 11

Pre 1.4 1.5 1.7 Post 2.4 2.7 2.5 Prec 2.9 2.9 NA

Program, advisor and student

Met at the lower range of the intermediate scale. Both student and preceptor / advisor post assessment shows measurable gain over student pre-‐assessment

Additional strategic orientation was added to the curriculum through two new courses added in 2013: MHA 626 and MHA 742. Further, a strategic focus is receiving increased emphasis across the curriculum. The full effect may not be seen for 2-‐3 years.

Final team case presentation in MHA 626 Organizational Behavior and Leadership Theory (Direct)

95% pass rate at >70% (inter-‐mediate / advanced) Blooms level 1-‐5 (internal)

Pass rate: 100% Mean Grade: 90%

Program and student

Met. Students worked in teams to create a strategic plan to address an organizational case. Other team members and the faculty evaluated each student’s contribution, then individual grades were assigned.

This course was added to set a firm foundation for strategy, leadership theory and organization behavior. It has only been offered once thus far, so the impact will be expected in future. Feedback on course was positive, and the course will continue, as will this measure. Also, team work is required for CAHME accreditation.

Final Marketing Plan for MHA 646 Strategic Marketing & Communication



Program and student

Met. Strategy is taught using a discussion of the principles of marketing. Students demonstrate learning by applying their knowledge to a health care service or product. A final marketing plan is developed and submitted at the end of the course.

This course has been a part of the MHA curriculum since 1999. It is a solid course that provides foundational knowledge and skill in the marketing and provides students a method to strategy development. However, students would benefit if faculty could help them connect the competence to the course objectives.

Final Case 95% pass Goal Met. This course was added to

6

presentation in MHA 742 Strategy Formulation (Direct)

rate at >70% (inter-‐mediate / advanced) Blooms level 2-‐6 (internal)


Students worked individually to create a strategic plan to address an organizational problem. This course occurs late in the program.

challenge the more advanced student for higher level analysis in strategic thinking. It has only been offered once thus far, so the impact will be expected in future. Feedback on course was positive, and the course will continue, as will this measure.

1.c. Communication: Prepares effective written and oral presentations; develops logical recommendations; prepares concise executive summaries.


Levels 2-‐3 on a 4 point scale (inter-‐mediate) Blooms level 2-‐5 (external)

AY 13

AY 12

AY 11

Pre 2.6 2.6 2.7 Post 3.2 3.5 3.2 Prec 3.2 3.3 NA

Program, advisor, and student

Met. Feedback from faculty and students indicate that written communication is improving, but need more opportunities for oral presentations.

To provide more opportunities for oral presentations, students in third executive residency present to a group of external reviewers and receive feedback on content and delivery. Also, Skype is being used for some presentations in MHA 633 and MHA 649

1.d. Collaboration: Develops effective working relationship with team; solicits input; willing to learn from others; encourages others

NCHL Competency by 1)Student Pre 2)Student Post 3) Preceptor and Advisor Post (Direct)


AY 13

AY 12

AY 11

Pre 3.5 3.5 3.6 Post 4.3 4.7 3.9 Prec 4.3 4 NA


Goal met. Students are scoring at the high intermediate to advanced level on this competency. This maps to DMU Competency #4: Display an ability to work collaboratively.

Team work will continue to be offered in several courses such as MHA 626, MHA 631, MHA 644, and others. Working in a virtual team online can be challenging, but students are gaining valuable experience using technology. The MHA program would like to work with ITS to provide even more robust tools for students and faculty to use.

Peer Reviews of Case Presentation in MHA 626 Organizational Behavior and Leadership Theory (Direct)



Program Goal met. Students are learning to use give and receive input from colleagues in an open and receptive manner.

Peer reviews will continue in MHA 626 and MHA 644. Instruction on how to provide high quality feedback is provided within the course. These activities and measures will continue. Although students ultimately perform well, they report it is challenging.

1.e. Accountability: Establishes high but achievable goals;

NCHL Competency by 1)Student Pre

Levels 2-‐4 on a 5 point scale

AY 13

AY 12

AY 11

Pre 3 2.8 2.7 Post 3.7 3.9 3.5

Program, advisor, and

Goal met. Our focus over the past 2 years has been

The faculty advising role has been enhanced with more frequent monitoring of student

7

demands high performance; confronts performance problems in self and others.

2)Student Post 3) Preceptor and Advisor Post (Direct)

(inter-‐mediate) Blooms level 3-‐6 (external)

Prec 3.9 3.9 NA student to catch students who are struggling earlier in program to provide advising and/or remedial work.

performance. A new tool allowing documentation is now in place to track advising interactions. Will continue to monitor.

1.f. Interpersonal Understanding: Commits to understanding others; sensitive to cultural, ethnic and social issues; interprets emotions and verbal content; seeks comprehensive and complex information.



AY 13

AY 12

AY 11

Pre 3.1 3.4 3.3 Post 4 4.1 3.7 Prec 4.2 4.1 NA


Goal met. This maps to the DMU competency #3: Value the human experience with sensitivity to the individual and cultural differences.

Diversity and inter-‐professional cases and examples have been integrated across the curriculum as appropriate to the course, and will continue.

1.g. Analytical Thinking: Identifies cause and effect relationships; analyzes relations between multiple parts of a problem; anticipates obstacles; thinks ahead.



AY 13

AY 12

AY 11

Pre 2.6 2.7 2.8 Post 3.2 3.6 3.3 Prec 3.4 3.4 NA


Goal met. This maps to DMU competency #1: Demonstrate knowledge of the science of human health and well being. Also to competency #5: Demonstrate an understanding of research methodology and its relationship to critical thinking.

Students are introduced to evidence based healthcare during the first Executive Residency. The focus on using high quality references and analytical thinking continues through the curriculum, culminating in Capstone / Field Based Learning course. This focus on evidence based healthcare has been increased over the past 2-‐3 years, and will continue.

1.h. Professionalism: Promotes organizational integrity; admits mistakes; serves others; maintains social accountability



AY 13

AY 12

AY 11

Pre 2.4 2.6 2.9 Post 3.1 3 3 Prec 3.2 3.3 NA


Goal met. This maps to the DMU competency #2: Manifest dedication to the highest professional standard.

The concept of professionalism is stressed across the curriculum, and modeled during the Executive Residencies by faculty and outside presenters.

1.i. Self Development: Reflects on and seeks to improve own performance; considers impact on others;

NCHL Competency assessed by 1)Student Pre 2)Student Post

Levels 2-‐ 3 on a 4 point scale (inter-‐mediate)

AY 13

AY 12

AY 11

Pre 2.6 2.7 3.4 Post 3.6 3.2 3.2 Prec 3.6 3.4 NA


Goal met at the high intermediate level.

8

modifies own behavior as needed.

3) Preceptor and Advisor Post (Direct)

Blooms level 2-‐4 (external)

2. Program Objective: Capstone / Field Based Learning: Students will successfully complete the culminating course in the MHA curriculum

Comparison of enrollment with completion. (Direct)

90% complete rate (internal)

Year Completion Rate 2013 95% 2012 95% 2011 98% 2010 72% 2009 77%

Program

Goal met. 55/58 complete; 3 filed for incomplete. Anticipate they will complete.

As “legacy” students graduate, the 2 credit hour Capstone will be phased out and replaced by the 3 credit hour Field Based Learning. It provides a more rigorous and structured experience to fully integrate the curriculum into a culminating experience.

GPA of graduating students (Indirect)

Median 3.0 or above (internal)

Year

GPA

2013 3.75 2012 3.53

Program Met. MHA students have no board exam, so this is included as an indicator of student success in the program.

3. Program Objective: Student Satisfaction with DMU MHA program

Graduate Survey (Indirect)

Response rate > 80% (internal)

Year Response Rate 2013 91% 2012 77% 2011 38%

Program Met. Response rate improved as the survey is sent at the same time as the request for diploma.

Continue to incorporate the graduate survey into the exiting process to achieve our desired response rate.

3.a Did the program help you further develop competency in Health Care Administration?


80% agree (internal)

Year Percent Agree 2013 94% 2012 100% 2011 100%

Program Met, although the percentage dropped a bit. May be due to higher response rate.

Continue to educate students on the competency model during the first Executive Residency. Create a closer connection between the course, assessments and competencies.

3.b. Did the courses offered in the program adequately prepare you for your desired career?


80% Agree (internal)


Program Met, consistent with previous years.

9

3.c. Did your experience in the MHA program help achieve your goal?


80% Agree (internal)


Program Met, although the percentage dropped a bit. May be due to higher response rate.

3.d. Did the program help you develop a career plan or provide career direction?


80% agree (internal)


Program Not met. This has been consistently low.

A career panel dinner has been added to the Executive Residencies, which has received very positive feedback. Advisors meet with students during each Executive Residency to review coursework and discuss career goals. Membership in the student health leaders has improved, and the club is offering quarterly networking dinners and career planning assistance such as resume review. The full impact of these efforts may not be seen for 1-‐3 years.

Please provide answers to the following questions:

1. Have you mapped your curriculum competencies? Yes, the curriculum has been mapped twice. The first time was April 2010 when the curriculum was mapped at the course level to the NCHL competencies and Bloom’s Taxonomy. Teaching methods and assessments were reported for each course. The second time was in 2012, at the course level, using a form within the CAHME candidacy application. A copy of this completed map is provided in Appendix A.

2. If so, at what level (e.g. course, objective, assessment, etc.)? It is mapped to the course level as shown in Appendix A. A rubric was developed (to fulfill CAHME candidacy requirements) that shows whether the competency is supported at the basic, intermediate, or advanced level.

3. What tool did you use to conduct the mapping? Do you have any feedback on the tool utilized? The primary tool used was the NCHL Competency-‐based Curriculum Mapping tool (www.nchl.org/universityresources). Strengths: 1) This tool is recommended and used by several CAHME accredited schools. 2) We have used it once so it would be could to use it again for consistency to assess achievements of new curriculum. 3) The process involved external review by NCHL curriculum experts. 4) The NCHL model focuses on career skills across the continuum of a career from early to advanced; career skills needed beyond the classroom into

10

the management suite. Drawbacks: 1) Several of the competencies are difficult to execute in an academic setting. 2) The tool / process is costly. 3) The tool only maps to the objective level. Assessments are collected by not linked to objectives.

Summary: The focus of this report is to identify and implement opportunities for curricular changes to improve student learning. These are noted within the table above. From the data, we can see opportunities for improvement. In those areas, specific curricular actions are outlined to address those areas of need. For example, under Learning Objectives, finance will remain an area of focus by using enhanced blended learning tactics and more opportunities for faculty / student contact. Another example, under Program Outcomes, the graduate survey indicated that career planning was not as robust as desired. Therefore, we will be enhancing this area with advising, the executive residencies, networking opportunities, the development of a learning portfolio and the career panel dinner. There are also several areas where the learning outcomes data indicate that the curriculum is working well. These curricular elements will stay in place, and will continued to be measured, at least through the CAHME self-‐study and accreditation process to demonstrate consistency and show data over time. Several new direct measures were added this year. Please note that students admitted after Fall 2012 are required to use Live Text, an e-‐portfolio tool, to collect artifacts from most courses. This will allow us to capture more direct measures of the competencies in future years as these students move forward in the curriculum. Overall, the new curriculum that was implemented in Fall 2012 appears to be improving student performance in key areas. However, the full extent of improvement will take several years to fully appreciate as our “legacy students” who enrolled prior to 2012 graduate and newer students have experienced the entire program. All of the faculty are actively engaged in the evaluation of learning outcomes and curricular improvement. We employ a continuous quality improvement process that involves all major stakeholders. Improvements are made on a continuous basis as feedback from students, faculty, and other stakeholders (such as the MHA advisory committee, employers, and DMU administration) indicate the need. Further, the impact of the rapidly changing health care system is integrated into course work in a timely fashion. The diagram below is a

11

snapshot of the student-‐centered approach we have adopted:

12

Appendix A.

2012/13 student learning assessment report · the target range for the mha program is the...

Documents