using nursing exam data effectively in preparing nursing accreditation
TRANSCRIPT
Using nursing exam data effectively in preparing nursing
accreditation
Ainslie T. Nibert, PhD, RN, FAANConsultant
Email – [email protected]
1
Objectives• Identify the steps needed to establish a systematic method
for documenting the reliability and validity of teacher-made tests used throughout the curriculum.
• Discuss analyses of student performance data collected from assessments used throughout the program that establish the success of the curriculum in meeting nationally-recognized standards and promoting achievement of program outcomes.
• Describe ways of incorporating assessment-related findings in the self-study prepared for the accreditation visit that illustrate how the testing program assists faculty in identifying student knowledge gaps, and evaluates the effectiveness of strategies employed to close these gaps.
•
2
Nursing Accreditation
3
Source: http://www.aacn.nche.edu/ccne-accreditation
Source: http://www.acenursing.org/mission-purpose-goals/
Source: http://www.nln.org/accreditation-services/the-nln-commission-for-nursing-education-accreditation-(cnea)
Teacher-made tests & Standardized Exams in Nursing provide essential data in our outcomes-oriented academic world• Nursing has moved from process-oriented standards to
program outcome-oriented standards• The increase in use of computerized, standardized exams that are
produced under rigorous psychometric standards provides reliable data and more opportunities for deeper analysis of results.
• As standardized testing resulted in ‘raising of the bar’ with nursing assessment, teacher-made testing has also moved increasingly into computerized applications that allow for psychometric analysis.
CCNE’s Accreditation Standards• Standard I: The mission, goals, and expected program outcomes
are congruent with those of the parent institution, reflect professional nursing standards and guidelines, and consider the needs and expectations of the community of interest. Policies of the parent institution and nursing program clearly support the program’s mission, goals, and expected outcomes. The faculty and students of the program are involved in the governance of the program and in the ongoing efforts to improve program quality.
• Standard IV c Licensure and certification pass rates demonstrate program effectiveness.• The NCLEX-RN® pass rate for each campus/site and track is 80% or higher for first
time takers for the most recent calendar year.
• http://www.aacn.nche.edu/ccne-accreditation/Standards-Amended-2013.pdf
5
ACEN’s Accreditation Standards• STANDARD 1 Mission and Administrative Capacity
• The mission of the nursing education unit reflects the governing organization’s core values and is congruent with its mission/goals. The governing organization and program have administrative capacity resulting in effective delivery of the nursing program and achievement of identified program outcomes.
• STANDARD 6 Outcomes • Program evaluation demonstrates that students and graduates
have achieved the student learning outcomes, program outcomes, and role-specific graduate competencies of the nursing education unit.
Source: http://www.acenursing.net/manuals/SC2013.pdf
6
NLN CNEA’s Accreditation Standards• Standard I: Culture of Excellence – Program Outcomes
• Establishing clear program outcomes is an essential first step in benchmarking and evaluating a nursing program’s success in achieving and sustaining a quality educational environment for faculty and students. Program outcomes can be defined as the results achieved in response to goals set by the program….
• 1a - Examples of faculty and staff engaging in regularly scheduled review of identified program outcomes with documentation of review outcomes and resulting decisions Examples of data-driven decisions based on the review analysis of achievement of identified program outcomes
• 1d -Three calendar years of trended data on licensure and certification pass rates. Documentation of implementation and evaluation plans to achieve set benchmarks related to licensure and certification examinations and a plan for intervention if needed.
• Source: http://www.nln.org/accreditation-services/the-nln-commission-for-nursing-education-accreditation-(cnea)
7
Documenting program achievement
Sources of outcome measuresInternal and external curriculum evaluation plans
Data collection and analysis Preparation of the self studyProduce evidence to support self study
Internal curriculum evaluation measures – teacher made testsExternal curriculum evaluation measures – standardized testsLicensure exam results
8
Relationship betweenTesting & the Curriculum
9
Internal and ExternalCurriculum Evaluation
Outcome Predictors
Where do I start?• What can I do now to start preparing for reaffirmation of
accreditation self study and visit• Specific recommendations for strengthening testing program
• Teacher made tests• Standardized tests• Item analysis reports• Exam analysis reports• Testing policies, procedures, and guidelines• Closing the loop – changes to curriculum based on these analyses
• Best Practices• Follow best practices for managing your program’s assessments• Strengthening the testing and evaluation component of your program
NOW will result in a sound dataset for analysis of the curriculum, and a stronger self-study
10
Best Practice #1: Establish Testing Committee• Design and revise the program’s testing policies
• Publishes Guidelines for Exam Development • Writing style protocol – apply to all tests• Critical thinking items – increase Application level
• Exam Administration Policies• Number of items/exam; length of time allotted/exam• Proctoring Guidelines• Dissemination of Grades (When? How?)• Test Review – to do or not to do; how to do it
…more Testing Committee Activities
• Systematic Item Analysis (see also #3 below)• Use statistical parameters for analyzing overall test and individual test items
• Adhere to published psychometric standards• Summarize analysis and include action plan• Review test blueprint as needed before next test• Review individual items earmarked for attention
• Compile/analyze standardized test results• Make recommendations to curriculum committee for changes based on findings
12
Relationship betweenTesting & the Curriculum
INTERNAL Curriculum Evaluation(Teacher-made Tests)
Recommended Item Analysis Standards
14
Parameter Level of Acceptance
Item Difficulty 30%-90%
Item Discrimination Ratio 25 % and above
PBCC (point biserial correlation coefficient)
0.20 and above
KR20 (Kuder-Richardson 20) 0.70 and above
Morrison, Nibert, Flick, J. (2006). Critical thinking and test item writing (2nd ed.).Houston, TX: Health Education Systems, Inc.
Establish the use of a Systematic Item Analysis used for all teacher-made tests
1. Review Difficulty Level
2. Review Discrimination Data
• Item Discrimination Ratio (IDR)
• Point Biserial Correlation Coefficient (PBCC)
3. Review Effectiveness of Alternatives
• Response Frequencies
• Non-distracters
Best Practice #2: Writing Style ProtocolEstablishing clear guidelines for faculty leaves little room for ambiguity; helps insure uniformity in the presentation of exams throughout the curriculum:
Will present or past tense be used for test items? Will options end in periods, whether or not there are
complete sentences? Will all options begin with a capital letter? When stressing a word in the stem, will it be highlighted,
boldfaced, italicized, underlined? Will the term patient or client be used?
Best Practice #3: Increase proportion of Application-and-above Items
17
Which intervention is most important?
Which intervention, plan, assessment data is/are most critical to developing a plan of care?
Which intervention should be done first?
What action should the nurse take first?
Which intervention, plan, nursing action has the highest priority?
What response is best?
Best Practice #4: Use Uniform Statistical Parameters for Item
AnalysisWhat is the acceptable number of mastery itemsto include?
What is the minimum acceptable difficulty levelfor a test item?
What is an acceptable discrimination level (PBCC) for a test item?
What is an acceptable reliability coefficientfor the exam?
18
Key Testing ‘Best Practices’
• Improve Test Blueprints • Adopt a uniform item style/format; publish the guidelines• Invert the proportion of Application-and-above test items
as compared to Knowledge/Comprehension items• Use a Systematic Item Analysis methodology• Include 10% alternative item formats – across all tests• Remove items that no longer align with NCLEX ® items
• Obsolete nursing diagnoses• Trade names for most medications (use generic only)
• Include 10% med calculations – across all tests• Revise length/time allotments – across all tests
19
Maintaining Consistent Testing and Grading Policies• Establish the Use of Best Practices with Testing• Update Policies and Re-educate annually• Discuss Testing Practices in a Faculty Forum
• Identify areas of inconsistency• Review test outcomes for impact of inconsistency
20
Relationship betweenTesting & the Curriculum
EXTERNAL Curriculum Evaluation(Standardized Tests)
Best Practices: Standardized Exams• Establish scoring benchmarks (evidence-based)• Determine weight of standardized exam in overall grade• Identify consequences for failure to achieve benchmarks
• Offer additional (parallel) versions of the exam – mix these up!• Allow enough time for remediation to occur between attempts.• Assign specific remediation activities (multiple strategies)
• Date/time for submission of these should be specified in the learning contract. Monitor remediation usage for student adherence to contract.
• Selections should be based on weak areas identified on scoring report.• Focus on building confidence – this is not a punitive activity. Model
effective study habits and test taking strategies.
• Use proctoring guidelines; take test security measures • Investigate suspected breaches and enforce published policies.
22
Teacher-made Tests and Standardized Tests: Vigilance with Test Security
23
1. Encourage moral behavior (Academic honesty program at your school with clear language placed in handbooks)
2. Discourage cheatinga. Before testing
1. Minimize access to exams and viewing of exam content2. Use highest levels of security available in learning mgmt.
systems for unit tests and all security features available in standardized testing platform - protect logins & access codes; proctors should initiate active dashboarding
3. Train proctors for all types of live proctoring activitiesb. During testing
1. Establish secure environment2. No deviations to test procedures or breakdown of
environmental security allowed. Ex: leaving room equates to the test being over for that student regardless of reason
3. Vigilant proctoring3. Detect cheating with Data Forensics and take actions as needed
Teacher-made Tests – Analyze/Document
24
Using an electronic test blueprint
25
Every test….Every test administrationConduct a Systematic Review of Teacher-made tests Each exam administered in the program should be evaluated and blueprinted using standard criteria agreed upon by the faculty; for example, level of item (knowledge/comprehension vs application/analysis) along with the percentage of alternative items. Look carefully at each test’s strengths & weaknesses•Consistent item analysis will be provided with all tests for better comparison and test improvement. •Faculty should complete the review on each test following administration, including item analysis data and any changes made based on the review that led to rescoring.
27
Teacher-made Test – Analysis results• Observations and Suggestions for NURS 101• Exam 2 and Final Exam were evaluated • Both exams lack predominance of questions written at the
application-and-above level on the cognitive taxonomy.
• Analysis of Leveling• K/C – Knowledge and Comprehension• A/A – Application and Analysis
30
Item Strength Evaluation: Nursing 101
31
Teacher-made Tests: Evaluation resultsIdentifying Testing Weaknesses•Review the item bank for consistency. If styles of items vary greatly from one exam to the next, edit items to conform to style guidelines adopted by the faculty.•Seeing the same problems with different tests used within a particular course? Authored by a particular faculty? •Consistently strive to increase the overall percentage of items written at the Application and Analysis levels of the cognitive taxonomy. Too often, the item bank predominantly reflects items written at the Knowledge and Comprehension levels.
32
Test Review Summary
33
Course Coordinator Report
34
Course Coordinator Report: Tests
35
Standardized Tests – Analyze/Document
36
Standardized Test Score Patterns
37
0
100
200
300
400
500
600
700
800
900
1000
1100
1 2 3 4 5 6 7 8 9
Sco
res
Student
Licensure Exam Results - Analyze
38
NCLEX® Test Plan Report: Client Needs
39
A closer look at Client NeedsSafe and Effective Care Environment•Coordinated/Management of Care•Safety and Infection ControlHealth Promotion and MaintenancePsychosocial IntegrityPhysiological Integrity•Basic Care and Comfort•Pharmacological (Parenteral) Therapies•Reduction of Risk Potential•Physiological Adaptation
40
NCLEX-RN® Client Needs – 2016 Changes
Percentage of Items from each category did not change from 2013 planSource: https://www.ncsbn.org/2013_NCLEX_RN_Test_Plan.pdf
NCLEX-RN® 2016 Test Plan Details
42
NCLEX® Test Plan Report: Performance
43
NCLEX® Test Plan Report: Percentages
44
References
Morrison, S., Nibert, A., & Flick, J. (2006). Critical thinking and test item writing (2nd ed.). Houston, TX: Health Education Systems, Inc.
National Council of State Boards of Nursing. (2016) 2016 NCLEX-RN test plan. Chicago, IL: National Council of State Boards of Nursing. https://www.ncsbn.org/RN_Test_Plan_2016_Final.pdf
Nibert, A. (2010) Benchmarking for student progression throughout a nursing program: Implications for students, faculty, and administrators. In Caputi, L. (Ed.), Teaching nursing: The art and science, 2nd ed. (Vol. 3). (pp.45-64). Chicago: College of DuPage Press.
Schroeder, J. (2013). Improving NCLEX-RN pass rates by implementing a testing policy. Journal of Professional Nursing, 29(2), S43-S47.
Sewell, J., Culpa-Bondal, F., & Colvin, M. (2008). Nursing program assessment and evaluation: Evidence based decision making improves outcomes. Nurse Educator, 33(3), 109-112.
45