nnaap (national nurse aide assessment program): job analysis, standard setting and next steps ada...
TRANSCRIPT
NNAAP™ (National Nurse Aide Assessment Program): Job Analysis, Standard Setting and Next Steps
Ada Woo, PhD
Senior Psychometrician, NCSBN
Outline
Overview of test development process 2009 nurse aides job analysis 2010 NNAAP standard setting
NNAAP Test Development Process
1. Job Analysis
2. Content Outline
3. Item Writing and Item Review
4. Subject Matter Expert (SME) Panel Review
5. Editorial Review
6. Building Test Forms
7. Setting the Passing Standard
8. Pretesting of Items
9. Administering Examinations
Job Analysis Process
“Evidence of validity based on test content requires a thorough and explicit definition of the content domain of interest… characterization of the domain should be based on job analysis.”
(AERA, APA, NCME, Standards for Educational and Psychological Testing, 1999, p.160)
Background on Job Analysis
Assist NCSBN in describing range of authorized duties and evaluating test plans.
Provide validity evidence to support the examination.
Conducted at least every five years. Large scale studies using multiple
methods.
Methodology
Document review Review of entry-level CNA daily logs Review of documents specific to CNA work
settings Large scale surveys to NAEs and entry-level
CNAs Non-responder study Methodology review by external
psychometricians
Survey Instruments
1. NA survey Work environment 115 NA activity
statements Last day of work Education and
training Demographic Comments
2. NAE survey Work environment 115 NA activity
statements Comments
Sample Selection
Randomly selected 6,500 nursing directors from hospitals, nursing homes and long term care facilities
Each director received three surveys to be distributed to one NAE and two entry-level CNAs
Return rates: 485 facilities opted out of the study 21 return surveys due to bad addresses CNA adjusted return rate – 10.7% (n=1,016) NAE adjusted return rate – 11.1% (n=666)
Job Analysis Survey Results
NA Ethnicity and Gender
17.6
.6 2.98.7
1.7 .5
65.4
0102030405060708090
100
AfricanAmerican
As ianIndian
As ianOther
Hispanic NativeAmerican
Pacifi cIs lander
White -Not of
HispanicOrigin
Race/Ethnicity
Perc
ent
Gender: 91.5% female; 8.5% male
NA Years of Experience
36.3
13.38.6 5.0 4.0 2.2 3.5 2.8 2.3 1.5 1.7 2.2 1.2 1.2 1.1 1.3
11.6
0102030405060708090
100
6 or le
ss mos
7 to 11
mos
1 year
2 years
3 years
4 years
5 years
6 years
7 years
8 years
9 years
10 yea
rs
11 yea
rs
12 yea
rs
13 yea
rs
14 yea
rs
15 yea
rs or m
ore
Years Experience
Perc
ent
NAE Years of Experience Supervising NAs
31.2
20.716.7
11.2 10.4 9.8
0
10
20
30
40
50
60
70
80
90
100
0 to 5 6 to 10 11 to 15 16 to 20 21 to 25 More than25
Years Experience Supervising
Perc
ent
Client Health Conditions
18.4%
1.1%
41.6%
24.4%
41.3%
39.0%
36.3%
5.0%
13.8%
1.7%
68.3%
52.7%
49.7%
62.6%
53.9%
3.8%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Well clients
OB (Maternity) clients
Stable chronic conditions
Unstable chronic conditions
Acute conditions
Clients at end of life
Behavioral/emotional conditions
Other
Clie
nt
Hea
lth
Co
nd
itio
ns
Percent
NA NAE
Client Age
0.0%
0.1%
0.3%
0.8%
7.7%
62.6%
28.4%
1.1%
2.4%
3.0%
8.6%
24.9%
81.1%
52.4%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
(less than one month)
(1 month–12 years)
(ages 13-18)
(ages 19–30)
(ages 31-64)
(ages 65-85)
(over the age of 85)
Age
of C
lient
s
Percent
NA NAE
Shift Worked
79.3%
15.1%
2.3% 0.8%0.3%
50.0%
4.4%0.5%2.3%0.2%
44.6%
0.2%0%
10%20%30%40%50%60%70%80%90%
Less than 1hour
1 hour – 4hours
5 – 8 hours 9 – 12hours
13 – 16hours
17 hoursor more
Hours Worked
Perc
ent
NA NAE
Most Performed Activities
Activity
1 Year or less NAActivity #
N
% Not Perform Activity
% Perform Activity
107 Ask for help when needed 551 0.00% 100.00%
98 Respect client's need for privacy/confidentiality
550 0.20% 99.80%
19 Follow Standard/Universal precautions (e.g., handwashing, personal protective equipment [PPE], isolation guidelines)
557 0.20% 99.80%
56 Allow client to do things at his/her own pace 553 0.40% 99.60%
93 Follow client's plan of care 552 0.50% 99.50%
Activity
Nurse Aide EvaluatorActivity #
N
% Not Perform Activity
% Perform Activity
98 Respect client's need for privacy/confidentiality
643 0.00% 100.00%
19 Follow Standard/Universal precautions (e.g., handwashing, personal protective equipment [PPE], isolation guidelines)
660 0.00% 100.00%
102 Follow code of ethics for nurse aides 642 0.00% 100.00%
107 Ask for help when needed 643 0.20% 99.80%
29 Use proper body mechanics 656 0.20% 99.80%
Least Performed Activities
Activity
1 Year or less NAActivity #
N
% Not Perform Activity
% Perform Activity
26 Perform cardiopulmonary resuscitation (CPR)
551 48.30% 51.70%
7 Assist client to fill out meal menu 554 46.90% 53.10%
23 Clear foreign-body airway obstruction (FBAO)
545 42.60% 57.40%
39 Prepare client for diagnostic test, procedure, or surgery (e.g., NPO)
554 41.50% 58.50%
46 Collect and label stool, urine or sputum specimens
552 37.30% 62.70%
Activity
Nurse Aide EvaluatorActivity #
N
% Not Perform Activity
% Perform Activity
31 Apply and monitor restraints 657 47.50% 52.50%
7 Assist client to fill out meal menu 652 40.60% 59.40%
51 Perform and record pulse oximetry 656 40.50% 59.50%
26 Perform cardiopulmonary resuscitation (CPR)
654 34.10% 65.90%
23 Clear foreign-body airway obstruction (FBAO)
648 30.10% 69.90%
Performance Ratings
0.0%
10.0%20.0%
30.0%
40.0%
50.0%60.0%
70.0%
80.0%90.0%
100.0%
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101
106
111
Activity Statements
Perc
ent
1 yr or less % Perf NAE % Perf
Most Important Activities
Activity
1 Year or less NAActivity #
NAvg.
ImportStd. err.
19 Follow Standard/Universal precautions (e.g., handwashing, personal protective equipment [PPE], isolation guidelines)
556 4.95 0.01
22 Apply and respond to client safety alarms 537 4.91 0.01
104 Report any suspected neglect, mistreatment or abuse
499 4.9 0.02
103 Report and record unusual incidents (e.g., errors, injuries, falls)
520 4.9 0.02
94 Follow Health Insurance Portability and Accountability Act (HIPAA)
519 4.87 0.02
Activity
Nurse Aide EvaluatorActivity #
N Avg. Import
Std. err.
19 Follow Standard/Universal precautions (e.g., handwashing, personal protective equipment [PPE], isolation guidelines)
660 4.93 0.01
104 Report any suspected neglect, mistreatment or abuse
635 4.91 0.01
103 Report and record unusual incidents (e.g., errors, injuries, falls)
632 4.85 0.02
22 Apply and respond to client safety alarms 628 4.84 0.02
94 Follow Health Insurance Portability and Accountability Act (HIPAA)
629 4.83 0.02
Least Important Activities
Activity
1 Year or less NAActivity #
N Avg. Import
Std. err.
5 Provide nail care 428 3.81 0.05
1 Provide foot care 418 3.83 0.05
7 Assist client to fill out meal menu 294 3.84 0.06
61 Assist or encourage family to help with client's care
418 4 0.05
18 Provide physical comfort measures (e.g., back rubs)
513 4.05 0.04
Activity
Nurse Aide EvaluatorActivity #
NAvg.
ImportStd. err.
34 Assist nurse with dressing change 507 3.45 0.05
7 Assist client to fill out meal menu 387 3.58 0.06
61 Assist or encourage family to help with client's care
559 3.78 0.05
5 Provide nail care 602 3.79 0.04
58 Assist client with recreational activities 599 3.9 0.04
Mean Importance Ratings
0.00
1.00
2.00
3.00
4.00
5.00
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101
106
111
Activity Statments
Impo
rtan
ce
New NAs NAEs
Conclusions and Next Steps
Respondents found activities listed in the surveys to be representative of CNA work.
Importance ratings given by CNAs and NAEs were similar.
Results obtained from job analysis surveys were used to determine test plan and weighting of test plan areas.
2011 NNAAP Test Plan Specifications
Reports and Technical Briefs
NNAAP 2009 Job Analysis Report (available on NCSBN website this summer)
2011 NNAAP Test Plan Report is available for download at https://www.ncsbn.org/1721.htm
NNAAP Test Development Process
1. Job Analysis
2. Content Outline
3. Item Writing and Item Review
4. Subject Matter Expert (SME) Panel Review
5. Editorial Review
6. Building Test Forms
7. Setting the Passing Standard
8. Pretesting of Items
9. Administering Examinations
Standard Setting
To determine a point on the ability continuum by which a candidate must exceed to be deemed competent
Specific to the NNAAP examination, the goal of standard setting is to determine the level of nursing ability necessary for certified entry-level nurse aides to practice safely and effectively.
Passing standard = passing score = cutscore
(con’t) Standard Setting
“Where the results of the standard-setting process have highly significant consequences . . . those responsible for establishing the cut scores should be concerned that the process by which cut scores are determined be clearly documented and defensible.”
(AERA/APA/NCME, Standards for Educational and Psychological Testing, 1999, p. 54)
Assumptions for Standard Setting
Certification and licensure is based on an assessment (usually an exam) that is content-valid, scalable and reliable.
We can create a defensible process that can quantify "how much" knowledge or skill a person must have to be considered minimally competent.
Scores on NNAAP Exam
Type of score is very important concept Candidates either "pass" or "fail” their exams.
The NNAAP uses a criterion-referenced standard setting process through the definition of minimally competent candidate.
We are not asking which candidates were in the top XX% on the exam. We are only concerned with the pass or fail decision.
Certification and licensure examinations are designed to: Protect the public from unsafe practitioners Ensure "best practices" of the profession Require entry-level professionals to achieve
a minimal level of knowledge or skill in their field to perform their job safely and effectively.
Importance of Standard Setting
(con’t) Importance of Standard Setting
If standard is set too low, we will certify some candidates who are unqualified.
If standard is set too high, we will deny a certification to some candidates who are qualified or competent.
Set appropriate standard while referencing the criterion.
Important Caveats
No “true” value exists Policy decision, related to mission of
organization (e.g., public protection) Standards should be revisited when test
specifications, scope of practice/range of authorized duties, education programs, and/or other factors change
NNAAP Standard Setting Process
1. Overview of standard setting process2. SMEs discussed qualifications for entry-level NAs3. SMEs reviewed NNAAP content outline4. Training on standard setting methodology5. SMEs rated practice items6. SMEs provided round 1 ratings of actual items7. SMEs discussed round 1 ratings8. SMEs provided round 2 ratings9. Gather validity evidence10. Determine cutscore
Subject Matter Expert (SME) Panel
Representative of practice settings and specialties, regions
Knowledgeable regarding practice analysis, test specifications
All panelists are RNs with minimum of two years of nursing experience
All panelists work with CNAs
Ten SMEs served on the panel of judges
SME Panel Characteristics
Characteristics # %
NCSBN Geographic Area
I 3 30%
II 2 20%
III 3 30%
IV 2 20%
Year of Post Licensure Experience1 to 15 years 2 20%
16 to 30 years 7 70%
31 to 45 years 1 10%
Work SettingNursing home 7 70%
Hospital 2 20%
Other long term care 1 10%
Certifications and Licenses Held (other than RN)*
C NA 8 80%
LPN/VN 4 40%
* Several panelists hold multiple licenses and certifications.
The Minimally Competent Candidate
To establish a passing standard, we focus on the examinees that possess “just enough” knowledge, skills and ability to practice entry-level nursing safely and effectively
We don't focus on high or low ability examinees
We focus on those who are "on the fence"
The Minimally Competent Candidate
What distinguishes candidates with "just enough" knowledge, skills and ability from the other two groups? Knowledge/skill of the fundamentals Little advanced knowledge Have learned enough to move on (no remedial
learning required)
“Should” vs. “would”
Modified-Angoff Method
Rate how minimally competent entry-level CNAs would perform on each NNAAP item.
Criterion-referenced Common in medical and health-related
credentialing Vast research base, defensible Used in NCLEX-RN, NCLEX-PN, MACE and
English proficiency standard setting workshops
Minimal Requirements for Entering the NA Profession
Training program is accepted by state boards of nursing and departments of health services
Nurse aide has minimum 75 hours of class room training
Nurse aide has minimum 16 hours of clinical training Nurse aide is at least 16 years old and in good
mental health Nurse aide works in the following settings: long term
care, acute care, assisted living or community health centers
Results from Standard Setting Workshop
Form and Round
% Cut Score Raw Cut Score (60 item test)
Standard Deviation
Rating Reliability
Form A Round 1
80.34% 48.20 (49) 0.004 0.81
Form B Round 1
81.34% 48.80 (49) 0.003 0.77
Form A Round 2
81.12% 48.67 (49) 0.003 0.79
Form B Round 2
81.87% 49.12 (50) 0.002 0.78
Note: Ratings from outlier were removed from the above results.
Panel Recommendations
SMEs reviewed results of standard setting workshop and their likely impact on NNAAP pass rate.
Upon reviewing the information including SMEs’ own knowledge about the industry, all 10 SMEs agreed to a cut score of 47 out of 60 items.
This new cut score will be implemented with the launch of 2011 NNAAP test forms.
Post-workshop SurveyQuestion Mean Rating Standard Deviation
Question 1: I understood the major goals of the standard setting workshop. 3.8 0.42
Question 2: I feel the training was adequate. 3.8 0.42
Question 3: I feel that the discussion of the minimally competent nurse aide candidate was comprehensive.
3.9 0.32
Question 4: I understood the group consensus on a minimally competent nurse aide candidate.
3.9 0.32
Question 5: I understood my task throughout the standard setting process. 3.8 0.42
Question 6: I understood the Modified Angoff rating process. 3.4 0.52
Question 7: I understood the Beuk rating process. * 2.9 0.57
Question 8: I had sufficient time to reflect on my first rating. 3.9 0.32
Question 9: I had sufficient time to reflect on my final rating. 3.9 0.32
Question 10: I feel the group discussions were productive. 4.0 0.00
Question 11: I feel my opinion was considered. 3.9 0.32
Question 12: I feel the passing standard recommended by the group was appropriate.
3.8 0.42
Rating Scale:
1=Strongly disagree, 2=Disagree, 3=Agree, 4=Strongly agree
SME Feedback
“I enjoyed the opportunity to meet other NA Educators from different states and be able to understand the variety of challenges each of us face. I feel honored to be a part of this process.”
“Supportive, well informed group leaders, appreciative of our input, made the process understandable. I would gladly come back anytime.”
“I learned a great deal about the process. I was very impressed with the professionalism and knowledge of the presenters. This is a very comprehensive process and I am honored to be a part of it.”
Next Steps
2011 NNAAP forms will be launched in January 1, 2011.
These test forms were constructed according to test specifications generated from the 2009 job analysis of nurse aides.
All test forms were equated to be comparable in difficulty. Passing score for all 2011 test forms is 47 out of 60.
A larger number of items will be pretested in 2011 to build NNAAP item pool.
Next NNAAP item writing/review workshop will be hosted in March 2011
Reports and Technical Briefs
Test Plan Report – https://www.ncsbn.org/2011_NNAAP_Te
st_Plan-Test_Specifications_Report_links.pdf
Standard Setting Report – complete report will be mailed to clients; proprietary information.
Questions?