Among the advantages of acomputer-based test over apaper-and-pencil test is its

capability to administer innovativetypes of test questions (items). Innova-tive items refer to test items that incor-porate functionality and elementsbeyond those available in a traditionaltext-based, multiple-choice format.These innovative elements may berealized in the structure of the items ortasks or may refer to the addition ofmultimedia such as graphics, audio,animation, or video.1 The allure of suchitems to testing programs is their poten-tial for measuring constructs that aredifficult to measure using the traditionalmultiple-choice item format. To theextent that such constructs comprise aparticular measurement domain of in-terest, the addition of innovative items

may ensure better coverage of thedomain, thereby increasing the contentvalidity of the inferences that can bemade from the test scores.

The National Council of StateBoards of Nursing (NCSBN) recentlyconducted preliminary research on thefeasibility of including various types ofinnovative items in its National CouncilLicensure Examinations for nurses(NCLEX-RN and NCLEX-PN). Previousresearch by NCSBN yielded several alter-nate item types that have been incorpo-rated into the NCLEX.2-4 This articledescribes the most recent research thatinvolved developing a trial set of inno-vative items designed to advance theassessment of nursing practice. Thespecific project goals were to (a) identifya set of nursing skills and processeswhose measurement is likely to im-prove through the use of innovativeitem types or elements (eg, those thatreflect higher order thinking, aspectsof physical assessment, or communica-tion skills), (b) develop innovative itemtemplates and prototype items for as-sessing a subset of the identified skillsand processes, (c) conduct a usabilitystudy for a set of items based on thesetemplates, and (d) perform a pilot

study using innovative items. The pur-pose of this article was to describethe development process and examineresults from evaluative components(usability study and pilot study of aset of trial innovative items) of the re-search study.

Test DevelopmentThe initial phase of test developmentinvolved defining a set of specific skillsand processes that would be particu-larly suitable for new types of itemsand relevant to the practice of newlylicensed registered nurses.

To define the target skills andprocesses, a group of subject matterexperts was convened. Two sourceswere especially influential in focusingthis process. The first source documentwas the Boards of Nursing’s ranking of5 nursing competency areas.5 Theseareas, according to their importanceranking, were (a) application of knowl-edge to practice, (b) critical thinking, (c)ethical/moral standards of practice, (d)competence in performing clinicalskills, and (e) effective communicationskills. Second was a set of competen-cies identified in a report by the Insti-tute of Medicine,6 which included (a)provide patient-centered care, (b) workin interdisciplinary teams, (c) employevidence-based practice, (d) apply qual-ity improvement, and (e) use infor-matics. Based on the recommendationsfrom these reports, the expert panelidentified 4 of the 8 sections from theNCLEX-RN test plan that may benefitmost from innovative item development:(a) management of care, (b) safety andinfection control, (c) psychosocial integ-rity, and (d) basic care and comfort.Within each of these areas of the testplan, specific objectives were selected.

Once the target skills, processes,and objectives were identified, the sec-ond phase focused on item templatedevelopment. The item templates andresulting items had to meet the follow-ing criteria:

& be able to assess a problem-solvingor higher order skill needed by RNs

56 Volume 34 & Number 2 & March/Apr i l 2009 Nurse Educator

Nurse EducatorNurse Educator

Vol. 34, No. 2, pp. 56-59

Copyright ! 2009 Wolters Kluwer Health |

Lippincott Williams & Wilkins

Evaluating Innovative Items forthe NCLEX, Part IUsability and Pilot Testing

Anne Wendt, PhD, RN, CAEJ. Christine Harmes, PhD

National Council of State Boards of Nursing (NCSBN) has recently conductedpreliminary research on the feasibility of including various types of innovative testquestions (items) on the NCLEX. This article focuses on the participants’ reactions toand their strategies for interacting with various types of innovative items. Part 2 in theMay/June issue will focus on the innovative item templates and evaluation of thestatistical characteristics and the level of cognitive processing required to answer theexamination items.

Authors’ Affiliations: Director (Dr Wendt),NCLEX Examinations Department, NationalCouncil of State Boards of Nursing, Chicago,Illinois; Director (Dr Harmes), PhD Program inAssessment and Measurement, James MadisonUniversity, Harrisonburg, Virginia.Corresponding Author: Dr Wendt, NationalCouncil of StateBoards ofNursing, 111EastWackerDr, Chicago, IL 60601 (awendt@ncsbn.org).

9Copyright @ 200 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

but that is difficult to effectivelyassess using traditional multiple-choice items;

& be targeted to a construct that maybe better assessed via an innova-tive item;

& be suitable for specification in anitem template, from which manysimilar items could be readilydeveloped;

& be suitable for computer-adaptiveadministration;

& have a clearly correct answer; and& be immediately scorable by the

computer.

Templates were initially developedfor the following item types: (a) graph-ics inclusion, (b) graphics interaction,(c) audio inclusion, (d) video inclusion,(e) video interaction, ( f ) animation in-clusion, and (g) decision task item sets.Figure 1 presents a computer screenprint of video interaction addressingcommunication skills. Figure 2 presentsa computer screen print of a graphicsinteraction item addressing safety. AFlash-based application was developedto administer these item types and tocollect detailed examinee action andresponse data. After discussion regard-ing the match of item types to targetskills and processes, item writers createditems within each of the templates. Itemcontent and specifications were revisedin subsequent sessions.

Usability TestingIn the third project phase, 2 evaluativestudies were conducted using the pi-lot innovative items. The first was a

usability study, using the methods ofexpert review and user testing. Asrecommended by Harmes and Parshall,7

expert review was conducted as partof a user-centered, iterative design strat-egy. At various points in the develop-ment of item templates and prototypeitems, experts from measurement, nurs-ing, and interface design reviewed theprototypes. After each review, appro-priate modifications were made. Usertesting was then conducted, in whichthe set of prototype items was adminis-tered to a small sample of examineesrepresentative of the target examineepopulation.

User testing was conducted in 3iterative stages. The first iteration in-cluded 2 students from a bachelor’sdegree program, the second included 1student from a bachelor’s program, andthe third included 3 students from anassociate’s degree program. Four of the

6 students were women, and 2 weremen. One student was a non-nativespeaker of English.

The think-aloud protocol was usedas examinees proceeded through thetest items.8 These individual sessionswere digitally recorded for transcriptionand analysis. After each participant’scompletion of the examination, theinterviewer posed a set of structuredinterview questions related to impres-sions of the test items and suggestionsfor revision. Results from both datacollection methods (observation andinterview) were highly informative andwere used to make substantial modifi-cations to the items before pilot test-ing. Examples of changes included (a)changing the colors of buttons and tabsto allow for easier recognition, (b) re-ducing the amount of scrolling required,(c) increasing the window size forviewing videos, and (d) modifying iteminstructions. Based on the detailedresults from the usability testing, thefinal set of 23 items was selected andrefined for use in the pilot test.

Pilot TestExamination Results

The pilot test was administered to 224students from 4 nursing programs. Two ofthe programs offer associate’s degrees,and 2 offer bachelors’ degrees. Most par-ticipants (84%) were women, and 79%were native English speakers. At the con-clusion of each student’s pilot test, a post-test survey was administered to gatherdemographic information and assess thestudents’ perceptions of the pilot test.

The prototype items yielded itemdifficulty values ranging from veryFigure 1. Video interaction addressing communication skills.

Figure 2. Graphics interaction item addressing safety.

Nurse Educator Volume 34 & Number 2 & March/Apr i l 2009 57

9Copyright @ 200 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

difficult (2 items with p-values ofapproximately .2) to very easy (1 itemwith a p-value of approximately .9).However, item difficulty values formost items were in the middle rangefrom .4 to .7. The students used theinnovative components, playing andreplaying the video and audio clips, aswell as referencing the ‘‘exhibit’’ mate-rial. The participants did not seem tohave problems using any of the inno-vative features for accessing item con-tent or responding to items.

Survey Results

Data analysis for each of the percep-tions questions was broken down bydegree type and by whether Englishwas the student’s first language. A #2

test (at the 0.01 level of statisticalsignificance) was used to determinewhether either of these variables wasassociated with the responses to eachperception question. In most cases,there was no significant associationwith degree type or whether Englishwas the student’s first language.

Computer Experience

Students tended to be experiencedwith using the computer in general,with 68% stating that they were experi-enced or very experienced. A similarpattern was found for students’ experi-ence using computers for testing. Sixty-seven percent said that they wereexperienced or very experienced.

Perception of Item Quality

Students strongly felt that the itemsassessed higher order thinking skills,with 93% responding either somewhator very effectively. They also thoughtthat the items were fairly difficult, with89% responding either somewhat orvery difficult. Compared with text-based multiple-choice items, the stu-dents generally viewed the items asmore realistic (63% rated them as some-what or much more realistic). Similarresults were observed for the relativechallenge of the items (53% ratedthem as somewhat or much morechallenging than text-based multiple-choice items). Finally, 60% respondedthat the test items, relative to text-basedmultiple-choice items, allowed them to

demonstrate their nursing competencesomewhat better or much better. Overall,these responses indicated that the itemswere viewed very favorably by thestudents.

Impressions of TestDelivery Software

Generally, students felt that the itemsclearly indicated what they were to doto answer the items (64% respondingvery clear) and that the testing soft-ware was easy to use (66% respondingvery easy).

Inclusion on the NCLEX

Students were asked if they felt that thetypes of items on the pilot test shouldbe part of the NCLEX. Although manystudents responded yes, definitely(47%), compared with those whoresponded no, definitely not (13%), asizeable percentage responded unsure.It was unclear whether the frequentunsure ratings were due to generallyuncertain feeling about the items orwhether they were due to examineeswanting some (but not all) of the pilottest items to be on the NCLEX.

Quality of SpecificItem Types

There were 3 survey questions thatassessed realism, likeability, and diffi-culty of the items. Eighty-six percent ofstudents felt that the video items weresomewhat or very realistic, and 77%responded that they liked them or likedthem very much. Ninety percent ofstudents responded that the videoswere very clear, 72% felt that thegraphics were very clear, and 94% ofstudents rated the quality of audio clipsas good or excellent. Finally, regardingthe supplemental item information in‘‘exhibits,’’ 88% reported looking at theexhibits and finding them somewhator very useful. The final questionaddressed the groups of items allpertaining to the same nursing scenario(decision task item sets). A high per-centage (94%) rated these items assomewhat or very effective.

The overall conclusion that can bedrawn from these survey responses isthat the students generally had positive

or very positive reactions to the pilottest items. They tended to feel that theitems were (a) clear, (b) of appropriatedifficulty, (c) realistic, and (d) able tomeasure their higher order levels ofnursing competence. The survey ques-tion with the least positive responsesconcerned whether students felt thatthe types of items on the pilot testshould be part of the NCLEX. It isunclear how to interpret the findingthat 40% of the students respondedunsure, particularly in light of thepositive responses to the other surveyquestions.

Educational ImportanceIt is essential that the licensure exami-nation for nurses should assess theentry-level nurse’s ability to practicesafely and effectively to protect thepublic from unsafe practitioners. Theintroduction of innovative items, suchas those developed in this study, mayallow for more authentic assessment ofsome important nursing skills that, here-tofore, have not been as directlyassessed on the licensure examination.In addition, some higher order cognitiveprocessing skills, such as evaluate andcreate, may be assessable using innova-tive items. Furthermore, these itemtypes may allow some candidates moreand perhaps better opportunities todemonstrate their competence thanwith the exclusive use of text-basedmultiple-choice items. Based on theresults of this study, the nursing stu-dents who participated believed theitems to be more representative of thework actually performed by nurses onthe job. Additional research shouldfocus on these issues as they areessential for most professional licensureexaminations and will ultimately helpprotect the public.

REFERENCES

1. Parshall CG, Harmes JC. Design of inno-vative item types: targeting constructs,selecting innovations, and refining proto-types. CLEAR Exam Rev. 2008;19(2):18-25.

2. Wendt A. Beyond multiple choice: thequest for authentic testing. Paper pre-sented at the annual meeting of theAmerican Educational Research Associa-tion; April 12-14, 2004; San Diego, CA.

3. Wendt A, Kenny L, Marks C. Assessing cri-tical thinking using a talk-aloud protocol.CLEAR Exam Rev. 2007;18(1):18-27.

58 Volume 34 & Number 2 & March/Apr i l 2009 Nurse Educator

9Copyright @ 200 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.

4. Wendt A. Investigation of the item char-acteristics of innovative item formats.CLEAR Exam Rev. 2008;18(1):22-28.

5. NCSBN. 2006 NCSBN Annual BusinessBook. Chicago, IL: NCSBN; 2006.

6. Institute of Medicine. Health Professions

Education: A Bridge to Quality. Washington,DC: The National Academies Press; 2003.

7. Harmes JC, Parshall CG. An iterative pro-cess for computerized test development:Integrating usability methods. Paper pre-sented at the annual meeting of the

Florida Educational Research Association;November 8-10, 2000; Tallahassee, FL.

8. Van Someren MW, Barnard YF, SandbergJAC. The Think Aloud Method: A PracticalGuide to Modeling Cognitive Processes.London, England: Academic Press; 1994.

MARTTI: Wireless Interpretation System Improves Communication

Teaching our students culturally appropriate communication skills can be a challenge. Many people seekinghealth care have limited proficiency with the English language. Safe, timely and effective diagnosis and treatmentof presenting conditions depends upon adequate interpretation of a patient’s symptoms, concerns and needs.Neither health care facilities nor nursing programs have the resources to hire qualified medical interpreters to fitevery communication challenge that arises. The Language Access Network (LAN) has developed a HIPPAapproved wireless, two-way video and audio wireless connection to a skilled medical interpreter known asMARTTI (My Accessible Real-Time Trusted Interpreter). This device is designed to assist with interpretation andlanguage barriers to health care. MARTTI provides 24 hour a day, 365 days per year access to interpreters forover 150 languages including American Sign Language (ASL). MARTTI allows healthcare providers to use asmall screen for face-to-face communication or a wide 19 inch screen for emergency situations. A documenttranslation service allows patients to discuss forms in their native language and in a transparent environment.Interpreters employed by LAN take education programs that focus on various healthcare related topics includinghospice, medical terminology and sexual health. LAN staff interpreters have degrees in ASL interpretation andnational certification through the Registry of Interpreters for the Deaf.

Nurses at both Aultman Hospital in Canton, Ohio and Texas Children’s Hospital in Houston report successfuluse of MARTTI, Ann Bykes RN, MSN a clinical specialist at Texas Children’s Hospital says that using MARTTIis more accurate than trying to ‘‘wing it’’ on her own, and an improvement over standard phone interpretiveservices. Debby Taylor, RN, MSN, MBA of Aultman Hospital notes that the best thing about MARTTI is thatthe device can go anywhere that assistance with interpretation is needed. Go to the LAN website (http://www.languageaccessnetwork.com/) to evaluate MARTTI and also to determine if your students should be made awareof this technology.

Source: Wood D. (November 2008). Wireless interpretation system improves communication. NurseZone.com:Devices and Technology. Available at http://www.nursezone.com/Nursing-News-Events/devices-and-technology.aspx?ID=18461&Tab=1. Accessed November 13, 2008.

Nurse Educator Volume 34 & Number 2 & March/Apr i l 2009 59

9Copyright @ 200 Lippincott Williams & Wilkins. Unauthorized reproduction of this article is prohibited.