pdf testing

P2

IAMPS 2011

PDF® Paperless Testingt

PDF Paperless Testing01Outline

‣ Computer-based testing

‣ YACBTS - Yet another CBT “system”

‣ Characteristics of a P2T document

‣ Anatomy of a PDF

‣ The process of creating a P2T document

‣ Usability, security and other amenities

‣ The Adobe in-house smack-down (Adobe Acrobat vs Adobe Reader vs LiveCycle RE)

‣ Let’s start our business (i.e., let’s talk about money)

‣ A “CBT” in your pocket

‣ P2T and adaptive testing: preliminary reflections

‣ Three final questions

PDF Paperless Testing

A Computer-Based Testing (CBT) is a method of administering tests by means of electronic devices. Advantages of this method over traditional paper&pencil testing (PPT) have been exhaustively demonstrated in the literature. We will not illustrate them here, as we believe that you know them well.

Over the years, the Italian Air Force Selection Center has acquired (and used) several CBT systems, accumulating a fairly reasonable experience on the matter. Generally speaking, we are pretty satisfied with them, but have also found some undesirable drawbacks:

‣ installation (or update) of CBTs isn’t always a breeze;

‣ license plans may be pricey;

‣ while possible, it isn’t easy to incorporate proprietary material (e.g., old paper&pencil tests) into CBTs.

02Computer-based testing

http://en.wikipedia.org/wiki/Test_(student_assessment)

http://en.wikipedia.org/wiki/Test_(student_assessment)


With the intention of adding a new tool to the ones already in use, our team strived to develop a CBT system that could address these drawbacks.

We were not interested in creating a full-fledged system, rather finding a technique for rapidly authoring and delivering psychological tests/surveys.

The basic assumptions were that such CBT system should be:

‣ cross-platform and not dependent from complex infrastructures;

‣ free from cumbersome installation/update procedures;

‣ easy to manage/set up;

‣ cheap.

We focused our attention on the PDF format and its capabilities of interacting “intelligently” with the user.

Following a Aha Erlebnis, we realized that we could make a PDF behave as if it was a tiny little CBT system. This was the beginning of the PDF Paperless Testing project, a.k.a. P2T.

03YACBTS - Yet another CBT “system”

PDF Paperless Testing04YACBTS - Yet another CBT “system”

How can a PDF document (P2T) be a tiny little CBT system? If properly authored, a PDF document can:

‣ behave in specific ways as events happen (e.g., document is opened/closed, page xx is selected or not and so on);

‣ accept user inputs through the use of form elements (e.g., text-fields, check-boxes, list-boxes, push-buttons and so on);

‣ perform a wide array of calculations.

QA

How can a PDF document (P2T) do these fancy things? By means of some programming instructions that resides in the PDF itself. In other words, everything is inside the PDF document. There is no need of installing softwares.

QA

PDF Paperless Testing05Characteristics of a P2T document

A super-duper P2T document can:

‣ perform a login procedure by checking test-taker personal information;

‣ track time and show to the test-taker a countdown watch;

‣ code test-taker’s answers;

‣ calculate raw and standardized scores;

‣ present a chart of the results;

‣ store test-taker’s data.

We have started our P2T project with fixed length testsand surveys, but we think that it could be possible to create adaptive tests (more on this later).

PDF Paperless Testing06Anatomy of a PDF

PDFs may be conceived as a stratification of layers, each of which serves different purposes. In developing our P2T documents, we utilized some of these layers in order to implement the basic functionalities of a CBT system.

Form layer; contains form elements (e.g., check-boxes, push-buttons, etc.) for accepting user inputs

Annotations layer; used to display (in combination with T) graphical representations of data

Typographic layer; contains graphics and texts (i.e., items texts)

...and behind the scenes...

XML layer; used to store data (e.g., test specifications, test-takers information)

JavaScript layer; contains the programming code that makes the PDF a P2T document

T

FA

J

X

This is where the magic happens!


What is Javascript?JavaScript is an object-oriented scripting language used to make documents (i.e., web-pages, pdf files) more interactive.

JavaScript is a language that lets you make your pages interact with the user and respond to what he does. Acrobat Standard and Acrobat Reader are capable of understanding Javascript code without the need of installing additional softwares.

QA

Is Javascript some obscure and exoteric language?Absolutely not. It is one of the scripting languages most used nowadays (e.g., interactive web pages), relatively easy yet powerful enough to accomplish complex tasks.

We are sure that in your unit there are smart IT guys who know it pretty well.

QA


Do I have to change the Javascript code each time I make a P2T?The answer is no. But it really depends on the ability of your computer experts to write reusable code.

The code we implemented can be adapted to different test scenarios, with a minimum amount of changes (or no changes at all).

QA

Do P2T intelligent functions require a lot of code writing?The answer is no. But it really depends on the ability of your computer experts to write optimized code.

It took us approximately 1.000 lines of code to implement the basic functionalities of a CBT system (Examinee Registration and Scheduling, Test delivery, Post-Examination Processing).

Such a figure, in terms of programming effort, is low. And..hey! we are not good programmers.

QA


What is XML?Extensible Markup Language (XML) is a set of rules for encoding chunks of data in machine-readable form. An XML document is just a text file that stores data in a way that information is made inherently meaningful. The information is organized in a tree-like structure and can be easily retrieved by accessing its nodes (or meaningful tags).

QA

Do I have to change the XML code each time I make a P2T?Yes, but it’s an easy task. We used XML to deal with test-takers information (e.g., name, surname, gender, etc.) and test specifications (i.e., test duration, number of items, keys, etc.).

XML code doesn’t entail advanced programming skills.

QA

PDF Paperless Testing10The process of creating a P2T document

The shrinkUsing programs such as Microsoft Word, the psychologist assembles a test (fixed length, multiple choice items in the scenario illustrated today) by putting items, graphics, multimedia and whatever he thinks the test/survey may deserve.

He then saves the document as a PDF file.

The geekUsing Adobe Acrobat Pro, the computer expert opens the PDF file, places the interactive elements (e.g., check-boxes), puts the Javascript code, attaches two XML files: one relates to test-takers, the other to test specifications.

He then hands back the vitaminized PDF (or P2T) file to the psychologist who can deploy it.

PDF Paperless Testing11The process of creating a P2T Test

Now that I have my vitaminized PDF what am I supposed to to?Well, just put it in a computer and run it!

QA

What kind of software will run my vitaminized PDF?There are two choices. Test-takers may use Acrobat Standard (full capabilities, but is not free) or Acrobat Reader (some limitations, but it is free).

Technical Note: If the test-takers use Acrobat Reader, our vitaminized PDF has to be given “extended capabilities” and can be run from no more than 500 workstations (but without limits on the number of times the document is filled in).

QA

PDF Paperless Testing12Usability, security and other amenities

Test creation

Test session setup

Usability

PDF Paperless Testing Traditional CBT

Data security

Data integrity

Data Handling

AppleLinuxWindows


Our P2T document relies on special functions that are fully operational if the test taker uses Acrobat Standard and somewhat limited with the free version of Acrobat Reader. The Reader’s limitations can be circumvented. Let’s see three different scenarios.

13The Adobe in-house smack-down

Test-takers use Acrobat Reader, and the P2T document has been “Reader Enabled” using Acrobat Pro

Test-takers use Acrobat Reader, and the P2T document has been “Reader Enabled” using LiveCycle RE (server-based)

Test-takers use Acrobat Standard (there is no need of “Reader enabling” the P2T document)

affordability affordability affordability

1 2

functionalities functionalities functionalities

3


We can hypothesize two options based on a medium-sized PC Lab (40 computers). The first one stems from the necessity of investing the minimum amount of money. Everything will work (with few limitations). The second one is the best compromise (in our opinion) between costs and functionalities.

14Let’s start our business

Test Development CostsTest Administration

Nr. 1 License Adobe Acrobat Pro

Nr. 40 Acrobat Reader (free) ~500 euro+

Check prices with your local Adobe reseller;inquire also for volume licensing plans

Check prices with your local Adobe resellerUnlimited number of deployable testson a maximum of 500 workstations

Unlimited number of deployable tests

1

Nr. 1 License Adobe Acrobat Pro

Nr. 40 Licenses Acrobat Standard ~10.000 euro+2

PDF Paperless Testing15A “CBT” in your pocket

Since the P2T documents don’t require to be installed, they can be carried (and deployed) anywhere.

Should you be worried about security, the P2T documents may be stored on an IronKey drive (encrypted with high-speed military-grade hardware encryption).

When an IronKey drive is plugged into a laptop or desktop computer, the user must authenticate with a password before data and applications are accessible.

If a thief tries to break into an IronKey and enters 10 incorrect passwords, the Cryptochip will securely erase all of the stored information. This ensures no trace of your data can be recovered from the device.

All encryption and password verification are performed in hardware, and cannot be disabled by worms, viruses or other malware.

75 mm

19 mm

4Gb99$

Check prices with yournational Iron Key reseller

PDF Paperless Testing16P2T and adaptive testing

So far, we have talked about fixed length tests. Our P2T documents are characterized by a mixture of static and dynamic elements. The static part is represented by the fixed set of items the test-takers are asked to deal with. Entities such as check-boxes, list-boxes and push-buttons are the dynamic side of the experience. Finally, the javascript and the XML code make our P2T documents self-administering and self-correcting.

At this stage of the project, the Italian Air force didn’t explore the possibility of making a step ahead with adaptive testing, but tried to collect some preliminary information on the issue, finding that it is possible for a PDF document to [i] connect to an external datasource (i.e., item bank) and [ii] present items dynamically (i.e., with no static part at all).

What we don’t know is whether it is feasible to implement computationally-intensive tasks (e.g., ability estimate) using Adobe’s products.

Anyhow, the piece of software that creates completely dynamic PDF documents is called LiveCycle Designer (included in the Acrobat Pro Suite) and is based upon XFA technology (not covered here).

PDF Paperless Testing17P2T and adaptive testing

Authoring software

Technology

Fixed length Tests/surveys Adaptive testing

Self-contained test

Dynamic content

Adobe Acrobat Pro LiveCycle Designer

AcroForm, Javascript XFA, Javascript

Cross platform dep.

Coding experience

PDF Paperless Testing18Three final questions

Are you an Adobe or Iron-Key reseller?Nope.

QA

Can you disclose your code to us?We will be happy to do so (at no costs). If you are interested in P2T, contact us before leaving.Maj Pierpaolo Calanna ([email protected])Maj Andrea Mastrorilli ([email protected]) Italian Air Force Selection Centre

QA

Can you show us a real example?Yes, sure.

QA

pdf testing

Documents