from paper and pencil to computer based testing - a case study

www.KryterionOnline.comwww.KryterionOnline.com

Irrigation Association Case Study

From Paper and Pencil Testing to Computer Based Testing Co-authored by the Irrigation Association and Kryterion, Inc. October 2010

Background

Every year the Irrigation

Association™ (IA) delivers

thousands certification exams across

the country to landscape contractors,

irrigation designers, water managers and agricultural engineers. Until recently, the

exams were delivered via the traditional method of paper and pencil. Exam sites

were scheduled across the country on a “on demand” basis. Packages of exam

papers with instruction booklets were sent via secure delivery to proctors at the

facilities and returned afterwards. However the number of candidates was growing

and the administration of exams was becoming more and more challenging.

Problems

The administration of the exams was becoming overwhelming and it was difficult to

scale the program. In some cases candidates had to wait months to be able to take

the exams and then they had to wait weeks for their results. Candidates looking to

of 5

www.KryterionOnline.comwww.KryterionOnline.com Page 2 of 5

certification to improve their employment situation were being put off by long wait

times. An unknown number of candidates were dropping out of the process.

The IA is looking to become NCCA accredited. Consistency of delivery and test

security were issues that needed to be addressed. Computer based testing (CBT)

seemed like a good solution but the Board had a preconception that it would be

difficult to do. Some on the Board were not ready to go to computer based testing.

Some members of the board were not as technology based, and had, as well, a

general fear of the unknown.

Method/Solution

We (the management of IA) did a gap analysis

to start building a picture of what the differences

were between paper and pencil and CBT. Our

concerns included our current standing running

a program that was both NCCA accredited and

legally defensible. We looked at our current

vendor to see if they could meet our needs.

SMT, AMP and Kryterion also responded to an

RFP we sent out. Initially we wanted to run paper and pencil and CBT

simultaneously. However, after a review of our needs Kryterion advised against it.

They recommended against running a paper pencil program. They explained that

paper and pencil testing would probably increase the workload rather than

simplifying everything.

After lengthy discussions in December 2009 the board decided to bring together

the three vendors for a face-to-face meeting in January 2010. Kryterion presented

the only complete end-to-end CBT solution – online item banks, ability to manage

candidates, high levels of security, a network of over 300 testing centers in the US

“Initially we wanted to run paper and pencil and CBT simultaneously. However, after a review of our needs Kryterion advised against it.”

of 5


and live access to information (we had none prior to this). Once the board was

able to see the demonstration they were able to visualize how this would be a

relatively straightforward transition. The board decided to move to CBT.

Results Achieved

We had originally planned for a one and one half

year transition to CBT. However, once we

understood how simple the transition could be

we did it over just a few months, during the

summer when things were quieter. The first

thing we had to do was upload our item banks

into Webassessor. While doing the transfer, we

discovered several mistakes in questions,

corrections and mis-keyed items that we thought

had been fixed. We now have the ability to fix

mistakes and be assured that they were fixed properly including an automatic audit

trail. This gives us great confidence moving forward. During the transition we went

through every item, painstakingly reviewing every question. We updated where

necessary to allow for changes in technology and the industry. Because of our

tight time schedule it was completely exhausting, yet the process itself was

relatively simple, fast and quite satisfying. Our certification director built the test

forms then the board reviewed and approved them.

Now registration is much faster. Candidates can register to test within a few weeks.

Candidates still must apply to the certification program but they can then register

for an exam location. While previously candidates had to wait 30-45 days for

results, they now get results right away. That quick turnaround time could make the

difference in career changes. This is an industry with high-turnover, getting trained

and certified quickly is important.

“We are more confident that we are giving good exams. We have the ability to write new sections quickly and efficiently.”

Sherrie Schulte, Certification & Education Director

of 5


While it is too early to tell if there will be much in the way of cost savings, if there

are they will probably not be significant. However we have gained competitive

advantage and are able to make the exams more readily available, which

increases people’s willingness to test. For example, one program requires three

separate tests. These could now be taken in a matter of weeks versus several

years with the previous testing sytem. The perceived benefit to our candidates is

huge.

With regards to Kryterion we have had very positive interaction. If we had issues

they were handled in a timely manner.

Sherrie Schulte, Certification & Education Director, was intimately involved in the

transition.

“I was new and they (Kryterion) were very helpful with my many questions. In

the past our other vendor did all the testing and item banking so now it was

being thrown at me which was a totally different thing – now I wonder why we

didn’t switch a long time ago”

“We are more confident that we are giving good exams. We have the ability to

write new sections quickly and efficiently. In the old process we had there were

too many gaps and room for error and everything took weeks and months.

Now everything is all in one place, so it is much faster, and I can track where

we are. It’s a weight off my shoulders. We are no longer risking security of our

program on emails and mailing exams all over the country. It doesn’t take

much to send an email to the wrong person by mistake. Overall we have

greatly improved our efficiencies and are able to keep the exams relevant,

accurate and accessible in a timely manner”

of 5


Irrigation Association Lessons Learned

CBT vendors are not all the same. Some have very similar solutions to PNP

vendors and don’t deliver the same benefits. Do your homework when seeking a

new vendor and talk to references. Don’t be afraid to switch, as long as you have a

good plan, it can be done quite quickly and with relatively few headaches. Avoid

trying to do both PNP and CBT at the same time as a transition plan as this is not

really necessary and might create an overwhelming amount of work for your

administrators.

About Kryterion

Kryterion is a full-service test development and delivery company that serves the

Association, Technology and Education markets with world-class secure testing

solutions.

Our solutions include the five key elements of certification and testing:

Psychometric services, exam management, security, real time access and test

delivery, all managed through our proprietary Webassessor test development and

deployment system. Our clients test with confidence and convenience.

For more information on how Kryterion can streamline your testing program

contact:

Kryterion Certification and Licensure Sales

[email protected]

from paper and pencil to computer based testing - a case study

Documents