![Page 1: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/1.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
A Ten-Year Review of CBT
ByCasey Marks, Ph.D., NCSBN
Geraldine Piskorski, MBA, ASCP Board of RegistryAnthony R. Zara, Ph.D., PV
Presented at the 2004 CLEAR Annual Conference:
21st Century Regulation: “Show Me” What Works
September 30, 2004
![Page 2: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/2.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Session Agenda
1. Speaker backgrounds
2. Before transition to CBT
3. Transition
4. Operations
5. Refinement
6. CBT lessons learned
7. Conclusions
![Page 3: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/3.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Speaker Background
Casey Marks, NCSBN
Geraldine Piskorski, ASCP Board of Registry
Anthony Zara, Pearson VUE
![Page 4: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/4.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
What We’re Trying to Accomplish
1. Share experiences
2. Compare and contrast large vs. small programs’ issues
3. Share solutions
4. Outline successful practices
![Page 5: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/5.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Some Industry Trends
1. Exploding numbers of programs
2. More complicated item types
3. Security concerns
4. Globalization
5. Maturing technologies, e.g., Internet
6. Increased choices for program sponsors
![Page 6: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/6.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Issues Before Transition to CBT
Small program• ASCP/BOR background
• Size and depth of item bank • Variable vs. fixed length test• Opportunity to review• Size of Examinee
population• Availability of computer
testing facilities• Cost of delivery
Large program• NCSBN background
• NCLEX facts
• Pre-CAT research program
• Barriers?
![Page 7: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/7.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Foundational Pre-CBT Research
• Dimensionality
• IRT calibration
• Scale stability
• Comparability of results
• Early on, much psychometric research was necessary; now a bit less, but still need to validate findings with your program.
• Mode of administration– What flavor of CBT? (CAT not right for everyone)
![Page 8: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/8.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Operational Research
• Examinee interface• Stopping rules• Test length• Item pre-testing plan• Test timing• Candidate impact• Beta testing
Here is where your program’s specifics get analyzed
and determined
![Page 9: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/9.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Transition Issues
Small program• Transition from testing
two days each year to year-round testing
• Staffing• Stakeholder
communications
Large program• CBT benefits• Create roadmap to
success• Organizational
processes• Stakeholder
communications
![Page 10: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/10.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Benefits of CBT
• More frequent administrations – (daily testing)
• Measurement accuracy and efficiency• Advanced item types can be used• Scheduling on demand• Increased security• Instant candidate results• Professional testing environment• Reduced time between retakes
![Page 11: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/11.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Roadmap for CBT
• Needs analysis• Stakeholder analysis• Building consensus• Working decision processes• Communications• Leverage the experience of others when
analyzing your program.
![Page 12: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/12.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Organizational Processes
• Organizational Decision Process– Understanding needs– Options for solutions
• Contracting Process– RFP or directed proposals– Provider options– Focus on outcomes
![Page 13: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/13.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Operations Issues
Small program• Psychometrics• Administration• Stakeholder support
Large program• Batch to continuous
model– Test development– Psychometrics– Administration– Stakeholder support– Staffing
![Page 14: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/14.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Program Operations
• Batch to Continuous: Different Mindset
– Test Development• Constant pool rotation -- ability to introduce
content more quickly• More items needed• Continuous item development/reviewing• Item writer recruitment
![Page 15: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/15.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Program Operations
• Batch to Continuous:– Psychometrics
• Constant item maintenance• Different item exposure concerns• Candidate volume spikes/dips due to content changes
and passing standard changes• Ability to remove underperforming
problematic/underperforming items during administration
![Page 16: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/16.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Program Operations
• Batch to Continuous:– Test Administration
• Individual issues/incidents• Ongoing test center monitoring• Ratio of investigations to candidates increases• Time from exam registration to result• Time-dependent registration policies
– Application process
– Fee changes
– Rescheduling issues
![Page 17: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/17.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Program Operations
• Interface with stakeholders– Test accommodations– Communicating program changes– Procedure/documentation changes
• Stakeholder Support– Direct support– Information/communication
• Accessible • Timely• Constant• Rumors/urban legends
![Page 18: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/18.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Program Operations
• Batch to Continuous:– Program Staffing
• Dedicated• Specialized• Outsourcing
– Customer Service• The on-demand mentality• Expectations
![Page 19: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/19.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
CBT Program Refinement
Small program• Immediate pass/fail
results at test center• Providers - more
options• Quality of graphics
Large program• Ongoing research• Item types• Providers• Service
![Page 20: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/20.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Program Refinement - Research
• Ongoing Research
– Methodology and content– Staff and/or test service– External
• Implementation dates seemingly “arbitrary”
![Page 21: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/21.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Program Refinement – Item Types
• Advanced item types
– Media– Stimuli– Response formats
![Page 22: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/22.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Program Refinement - Providers
• Service Provider Change– Between 1994 and 2002 same CBT provider– New test development and administration provider
• (2 for 1) in 2002
– New center network– New test driver/candidate interface– Candidate support– Data reconciliation between test services– New application/administration/reporting/results
processing systems– Seamless…
![Page 23: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/23.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Program Refinement - Service
• Innovative item types (beyond MCQ)– Utilizing same exam administration parameters
• More frequent administrations – 2 to 4 to 8 per year
• Additional testing time– Opportunity to demonstrate competence under
optimal exam termination rules
• International administration– Outside U.S. and territories starting in 2005
![Page 24: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/24.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
CBT Lessons Learned
Small program• Importance of good
communication with all stakeholders
• Size of examinee population
• Improved benefits to examinees
Large program• Benefits obtained worth
it?• Program complexities• Stakeholder
satisfaction?• Service?• Etc.
![Page 25: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/25.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Challenges
• Large program, yet perfection is needed/expected daily…– More moving pieces, more expertise
needed– Change is harder to integrate– The more complex the program, the more
difficult to educate stakeholders– Timing of program changes never
convenient for all stakeholders
![Page 26: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/26.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
General Conclusions
• Any large vs. small program things?
• Any “gotchas”?
• Need to consider all program stakeholders when considering procedure changes
![Page 27: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/27.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Conclusions
• CAT is a good thing for the NCLEX…– Boards of Nursing enjoy:
• Convenience and speed of system• The ability to focus on exam content not psychometric and/or exam
administration
– Candidates like:• Quick registration, scheduling and results receipt • Scheduling options• Personalized service
– NCSBN appreciates:• Security benefits• The ability to test content in new and novel ways
![Page 28: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/28.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Discussion
![Page 29: ASCP Board of Registry______________ ASCP Certification – The Standard of Excellence September 30, 2004 2004 CLEAR Conference A Ten-Year Review of CBT](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4a5503460f94b3debb/html5/thumbnails/29.jpg)
September 30, 2004 2004 CLEAR Conference
ASCP Board of Registry______________ASCP Certification – The Standard of Excellence
Contact Information
Casey Marks, Ph.D.Director of Testing ServicesNCSBNemail: [email protected]
Gerry Piskorski, MBADirector, Applicant, Certificant and IT ActivitiesASCPemail: [email protected]
Anthony R. Zara, Ph.D.Vice President Testing ServicesPearson VUEemail: [email protected]