ascp board of registry______________ ascp certification – the standard of excellence september 30,...

September 30, 2004 2004 CLEAR Conference

ASCP Board of Registry______________ASCP Certification – The Standard of Excellence

A Ten-Year Review of CBT

ByCasey Marks, Ph.D., NCSBN

Geraldine Piskorski, MBA, ASCP Board of RegistryAnthony R. Zara, Ph.D., PV

Presented at the 2004 CLEAR Annual Conference:

21st Century Regulation: “Show Me” What Works

September 30, 2004



Session Agenda

1. Speaker backgrounds

2. Before transition to CBT

3. Transition

4. Operations

5. Refinement

6. CBT lessons learned

7. Conclusions



Speaker Background

Casey Marks, NCSBN

Geraldine Piskorski, ASCP Board of Registry

Anthony Zara, Pearson VUE



What We’re Trying to Accomplish

1. Share experiences

2. Compare and contrast large vs. small programs’ issues

3. Share solutions

4. Outline successful practices



Some Industry Trends

1. Exploding numbers of programs

2. More complicated item types

3. Security concerns

4. Globalization

5. Maturing technologies, e.g., Internet

6. Increased choices for program sponsors



Issues Before Transition to CBT

Small program• ASCP/BOR background

• Size and depth of item bank • Variable vs. fixed length test• Opportunity to review• Size of Examinee

population• Availability of computer

testing facilities• Cost of delivery

Large program• NCSBN background

• NCLEX facts

• Pre-CAT research program

• Barriers?



Foundational Pre-CBT Research

• Dimensionality

• IRT calibration

• Scale stability

• Comparability of results

• Early on, much psychometric research was necessary; now a bit less, but still need to validate findings with your program.

• Mode of administration– What flavor of CBT? (CAT not right for everyone)



Operational Research

• Examinee interface• Stopping rules• Test length• Item pre-testing plan• Test timing• Candidate impact• Beta testing

Here is where your program’s specifics get analyzed

and determined



Transition Issues

Small program• Transition from testing

two days each year to year-round testing

• Staffing• Stakeholder

communications

Large program• CBT benefits• Create roadmap to

success• Organizational

processes• Stakeholder

communications



Benefits of CBT

• More frequent administrations – (daily testing)

• Measurement accuracy and efficiency• Advanced item types can be used• Scheduling on demand• Increased security• Instant candidate results• Professional testing environment• Reduced time between retakes



Roadmap for CBT

• Needs analysis• Stakeholder analysis• Building consensus• Working decision processes• Communications• Leverage the experience of others when

analyzing your program.



Organizational Processes

• Organizational Decision Process– Understanding needs– Options for solutions

• Contracting Process– RFP or directed proposals– Provider options– Focus on outcomes



Operations Issues

Small program• Psychometrics• Administration• Stakeholder support

Large program• Batch to continuous

model– Test development– Psychometrics– Administration– Stakeholder support– Staffing



Program Operations

• Batch to Continuous: Different Mindset

– Test Development• Constant pool rotation -- ability to introduce

content more quickly• More items needed• Continuous item development/reviewing• Item writer recruitment



Program Operations

• Batch to Continuous:– Psychometrics

• Constant item maintenance• Different item exposure concerns• Candidate volume spikes/dips due to content changes

and passing standard changes• Ability to remove underperforming

problematic/underperforming items during administration



Program Operations

• Batch to Continuous:– Test Administration

• Individual issues/incidents• Ongoing test center monitoring• Ratio of investigations to candidates increases• Time from exam registration to result• Time-dependent registration policies

– Application process

– Fee changes

– Rescheduling issues



Program Operations

• Interface with stakeholders– Test accommodations– Communicating program changes– Procedure/documentation changes

• Stakeholder Support– Direct support– Information/communication

• Accessible • Timely• Constant• Rumors/urban legends



Program Operations

• Batch to Continuous:– Program Staffing

• Dedicated• Specialized• Outsourcing

– Customer Service• The on-demand mentality• Expectations



CBT Program Refinement

Small program• Immediate pass/fail

results at test center• Providers - more

options• Quality of graphics

Large program• Ongoing research• Item types• Providers• Service



Program Refinement - Research

• Ongoing Research

– Methodology and content– Staff and/or test service– External

• Implementation dates seemingly “arbitrary”



Program Refinement – Item Types

• Advanced item types

– Media– Stimuli– Response formats



Program Refinement - Providers

• Service Provider Change– Between 1994 and 2002 same CBT provider– New test development and administration provider

• (2 for 1) in 2002

– New center network– New test driver/candidate interface– Candidate support– Data reconciliation between test services– New application/administration/reporting/results

processing systems– Seamless…



Program Refinement - Service

• Innovative item types (beyond MCQ)– Utilizing same exam administration parameters

• More frequent administrations – 2 to 4 to 8 per year

• Additional testing time– Opportunity to demonstrate competence under

optimal exam termination rules

• International administration– Outside U.S. and territories starting in 2005



CBT Lessons Learned

Small program• Importance of good

communication with all stakeholders

• Size of examinee population

• Improved benefits to examinees

Large program• Benefits obtained worth

it?• Program complexities• Stakeholder

satisfaction?• Service?• Etc.



Challenges

• Large program, yet perfection is needed/expected daily…– More moving pieces, more expertise

needed– Change is harder to integrate– The more complex the program, the more

difficult to educate stakeholders– Timing of program changes never

convenient for all stakeholders



General Conclusions

• Any large vs. small program things?

• Any “gotchas”?

• Need to consider all program stakeholders when considering procedure changes



Conclusions

• CAT is a good thing for the NCLEX…– Boards of Nursing enjoy:

• Convenience and speed of system• The ability to focus on exam content not psychometric and/or exam

administration

– Candidates like:• Quick registration, scheduling and results receipt • Scheduling options• Personalized service

– NCSBN appreciates:• Security benefits• The ability to test content in new and novel ways



Discussion



Contact Information

Casey Marks, Ph.D.Director of Testing ServicesNCSBNemail: [email protected]

Gerry Piskorski, MBADirector, Applicant, Certificant and IT ActivitiesASCPemail: [email protected]

Anthony R. Zara, Ph.D.Vice President Testing ServicesPearson VUEemail: [email protected]

ascp board of registry______________ ascp certification – the standard of excellence september 30,...

Documents