transforming the future of examination marking and services · 2015. 11. 2. · transforming the...

Reducing turnaround times | Improving quality | Simplifying logistics Improving security | Improving management information

Increasing marker consistency | Delivering candidate anonymity Reducing costs | Real-time monitoring | Exception handling

Transforming the future of examination marking and services

Contents03 Transforming the future of examination marking05 Benefits of e-Marker®09 e-Marker® CMI+ features13 e-Marker® development14 Focus on quality15 Using segmentation

Transforming the future of examination marking

Transforming the future of examination markinge-Marker® is designed to meet the diverse needs of all types of assessment and testing bodies from a few thousand to millions of scripts; e-Marker® provides the tools to increase the efficiency and effectiveness of the marking process.

Powerful marking solutione-Marker® harnesses the power of the Internet, along with dedicated processes, specialist services, sophisticated software functionality and high specification technology to deliver a flexible and powerful marking solution, delivering direct benefits in terms of cost, quality and control.

Managing script logisticsDRS has four decades of experience in managing examination data. We have been working with awarding bodies for much of that time and have extensive experience of the complexities associated with the lifecycle of a script.

Candidate scripts are securely scanned and imaged using DRS’ award winning PhotoScribe® image scanners, capturing every single answer. The images are then securely managed within e-Marker®, the integral electronic marking element in our fully inclusive examination marking system.

We will take care of the logistics including secure receipt of the scripts and scanning and imaging of answers in preparation for markers to mark electronically.

Removing errorse-Marker® uses tried and tested quality control procedures to ensure that once marking is completed, no further checking is necessary - clerical errors are removed by the process and regular automated sampling checks marking standards every step of the way.

Real-time management informationWith a comprehensive administration module, developed in conjunction with leading awarding bodies, e-Marker® enables the management and tracking of all stages of examination marking providing real-time information and vital reporting functions.

“ e-Marker® is a

powerful marking solution that provides real-time management information

“

Benefits of e-Marker®


When moving from traditional paper-based marking to electronic marking methods, e-Marker® is proven to bring improvements in both marking quality and real-time marker monitoring.

Paper-based systems can only be improved until the limitations of working with one copy of a candidate’s answer become apparent. Limitations include: • a second marker may be influenced by the first marks written on the scripts• reliance on scripts being sent through the post • markers being in the same place for a period of time.

e-Marker® removes all these limitations while ensuring that scripts are managed safely and securely with all candidates answers marked with complete anonymity.

“Behind all the electronic marking technological innovation, we have never lost sight of the candidate. Every student has their own hopes, dreams and ambitions and it is that simple, yet critical vision that dictates and drives the passion in which we deliver e-Marker®”.

Cost savings

Significant savings can be made by simplifying the complex marking process in areas such as:• standardisation – comprehensive on-line training and support• reduced script movement• automated checking of addition, eradication of transposition errors and missing scripts.

Many of the exception processes which currently require labour intensive manual intervention are handled through the e-Marker® system, including the matching of additional and unexpected scripts to the data held on record.

Segmentation marking of essay style answers

e-Marker® is able to bring unique benefits in the ability to mark essay style booklets. Long form answers can be marked at item level, no matter in what order the student completed the questions, or the length of the answers.

Benefits are:• ability to handle direct text imported from on-screen testing tools• ability to mark question by question• escalation to senior marker or administration options• exception handling processes via a single user interface.

“Markers using e-Marker® for essay style answers are enjoying the experience of marking long form answers electronically”.

• maximised space on-screen for candidate answer• ability to navigate through script via thumbnails• ability to display marking guidance• direct entry marking or marking via annotation• standardised, drag-and-drop comments and annotations • marker privileges can be added or removed by an administrator in real-time• marker has ability to review their own past marking• marks can accrue by annotation or direct from examiner input.

Simplifying processes

Processes that could only be supported by marking from central locations can now be carried out remotely in markers’ homes without needing to process paper. Images of candidates answers are electronically and securely distributed anywhere around the world allowing markers to mark wherever they may be and at whatever time suits them.

The removal of paper-based processes not only results in reduced costs but also provides an increase in quality control such as regular automated sampling of marking, rather than depending upon reviewing paper scripts selected from a marker’s allocation.


Enhanced Quality Control

• marker performance can be automatically monitored via moderated answers• marking can be sampled automatically or by an examiner and checked by a senior marker• quality control (QC) sampling rate can be adjusted throughout marking and on a marker-by-marker basis• high degree of configurability in applying marking standards, including sampling interval, tolerance and escalation methodology• audit reporting per examiner• senior markers can review work of other markers• automated quality of marking reporting• marking progress reports• senior markers can intervene in the marking process for QC purposes• administrators can be set up for differing levels of access• individual marks can be set to be checked or deleted then re-marked as part of QC.

Streamlined re-marking techniques

The awarding appeals process module within e-Marker® has proved instrumental in enabling awarding bodies to meet key Government targets. Paper scripts do not need to be physically recirculated for review and re-marking. Instead, the image of the script already exists within e-Marker® and by simply selecting the candidates to be reviewed, the electronic image of the script can be re-marked.

A process that in the past could take up to two weeks to complete is now being processed in less than 24 hours - literally at the click of a button.

Reducing turnaround times

Turnaround times for marking are reduced with e-Marker®.

Auto-marking provides the mechanism for a small number of examiners to mark a very large number of items more reliably.

Scripts are scanned immediately at a secure centre with no paper scripts sent to examiners.

Awarding bodies using the system have reduced their overall processing target by at least a week.

Increased security

Security has been at the core of the design objectives for e-Marker®. The management of confidential information is vital in the processing of examinations and with four decades of experience in managing worldwide examination data, DRS is fully aware of the associated issues.

We have built-in sophisticated procedures and checks for handling question papers and marking schemes within e-Marker®. Processing systems are isolated from any other systems with access control and CCTV monitoring in place.

DRS’ hosting centre is located in a state-of-the-art data centre with multi-layers of security and back-up services. The DRS infrastructure has been designed with fail-over and redundancy incorporated to ensure the maintenance of a continuous service. Service delivery is deployed across a 128-bit encrypted Internet link and can support thousands of simultaneous markers.

Candidate identity is treated with the highest security throughout and the way in which e-Marker® handles candidates’ script images means that no marker, no matter what level, can identify a pupil from the application in normal marking circumstances.

Analysis

A wide variety of standard reports, and post-hoc analysis are offered as part of the e-Marker® service.

Real-time reports can be produced to measure how well an examination performed, what kind of answers candidates gave to questions, how long it took for examiners to mark each item and how closely each examiner met the marking standards as judged by the Quality Control processes.

This real-time marking progress and quality, together with regular feeds of marks to enable the assessment of the examination’s performance, can be made more quickly than if using the traditional paper-based marking approach.

“ e-Marker® reduces

costs, improves quality, reduces turnaround times, improves management information, improves security and delivers candidate anonymity

“


e-Marker® CMI+ features

e-Marker® CMI+

Cos

t Red

uctio

n

Impr

oved

Qua

lity

Turn

arou

nd T

ime

Sim

plifi

ed L

ogis

tics

Sec

urity

Features Benefits

Real-time marking managementThe on-line process means that you will always know the progress of the programme, each paper and each examiner and an administrator can intervene at any stage if required

v v v v vAnomaly and missing scripts identificationThe reporting suite means you can identify problems and intervene immediately v v v v vContinuous performance monitoringThe seeding function enables you to continuously ensure that markers are meeting quality standards v v v vEssay marking abilityThe ability to mark an unconstrained long form answer whilst still retaining the benefits of segmentation, a robust quality methodology and a pleasant marker experience

v v v vStandard commentsA menu of recommended awarding body pre-defined comments is available to markers to ensure consistency for students feedback and ease of future analysis

v vCustomisable annotation toolsMarkers can create their own annotations on the scripts vParagraph toolMarkers can select different parts of the candidate’s script and e-Marker® will draw the separate parts together in order that they can be reviewed as one by the marker

v v vMultiple-part markingThe marker can mark a number of associated parts together vException handling toolsIf a marker comes across a candidate’s response that does not belong to the question that is being marked, it can be highlighted by the marker and referred to administration for action

vReduced script journeyScripts scanned immediately at a secure centre. No paper scripts sent to examiners v v v vFaster transfer of marksAll marking is real-time v v vProblem escalationAll markers can electronically escalate problems to a senior examiner v v vAuto-totalling of marksTransposition errors are eradicated as e-Marker® will total the item level marks v v v vNo answer can be missedIt is not possible to miss an answer response by mistake as e-Marker® will ensure that every item is presented to a marker

v v v v

e-Marker® CMI+ features

e-Marker® CMI+

Cos

t Red

uctio

n

Impr

oved

Qua

lity

Turn

arou

nd T

ime

Sim

plifi

ed L

ogis

tics

Sec

urity

Features Benefits

Electronic mark guidanceMarking guidance is tagged electronically to questions, ensuring that markers are given clear instructions on how to allocate marks

v vCentralised mark guidanceAny updates or amendments to the marking guidance can be rapidly published to markers v v v vFull image of script availableA marker can refer to more than just the image of the individual question in order to allocate a mark, should they need to

vSeeding capabilityTo keep markers on-standard v v v vElectronic re-allocation of scriptsIn the event that questions need to be re-allocated between markers, the process is immediate and dynamic v v vItem specialisationThree different marker types (auto, general, expert) allow allocation of individual questions to the most appropriate marker

v v

Candidate answer magnificationThe e-Marker® marker interface provides a range of tools to allow an improved image either through magnification or through adjusting the contrast

v v

Effective utilisation of resourcesAppropriate use of expert markers through auto-marking and general (clerical) marking v vAuto markingSingle numbers or words/short phrase answers can be individually key captured but marked in bulk by a single examiner, potentially saving large amounts of time and providing consistent and standard marking

v v

Measurement toolkit suite for geometric markingElectronic overlays enable fast and accurate marking of geometry questions, eg: the Graphical Measurement Tool allows an answer template to be overlayed to within 0.5mm accuracy on top of the candidates’ response

v v v v

Examiner marker quotasMarkers can optionally be sent maximum quotas by the awarding body to aid the process of marking v vCandidate anonymityThe candidate remains anonymous throughout the marking process v v

e-Marker® development

We believe that e-Marker® is the most effective tool for carrying out the electronic marking of paper scripts. Electronic marking enables a step-change in examination marking that could not be achieved through marking from paper.

e-Marker® development

Throughout all the electronic marking technological innovation, it is important to never lose sight of the candidate. At DRS, we understand that every student has their own hopes, dreams and ambitions and it is that simple, yet critical vision that dictates and drives the passion in which we develop and deliver e-Marker®.

DRS continually seeks to ensure that e-Marker® provides the functionality that awarding bodies require and as such there is an on-going development programme that provides regular new releases.

Local scanning and hosting We are developing partnerships around the world to enable awarding and assessment bodies to scan scripts in-country whilst harnessing the benefits e-Marker® services can bring.

e-SubmissionAs well as catering for paper scripts DRS is also developing the facility to receive electronic documents from sources such as e-Assessment applications and coursework.

What do I do next?

Step 1 - Visit from DRS to discuss requirementsStep 2 - DRS receive sample papers and analyse suitabilityStep 3 - Non-live proof concept using e-Marker® on scripts that have already been assessedStep 4 - Progress to trialling and piloting using live scripts, initially with parallel marking

Sample (Non-live)

Proof of Concept (Non-live)

Trial (Live)

Pilot(Live)

Review and Scope

Review and Scope

Review and Scope

Focus on qualitye-Marker® enables enhanced quality control as the restrictions of paper-based marking are removed. The use of images enables individual questions to be segmented and marked separately, bringing the following benefits: • improved marking quality - markers

are able to concentrate on an individual question at a time• marking bias is reduced and reliability increased – more than one marker marks an individual candidate’s script• anonymity is maintained as markers do not know whose answer they are marking • reviewed by suite of quality control products• quality control interventions can be included post-hoc• improved consistency from markers as a result of focusing on one question at a time• quicker familarisation with mark scheme for a particular question.

Research conducted by DRS and the NFER¹ on five GCSE subjects² examinations showed that there were:• exact agreement between markers and the seeded questions of between 79% and 98.7%• agreement to within one mark between 91% and 99.4%

Once all marking is complete, no further quality control measures are needed. The marks awarded are accurate, robust and reliable.

Focus on quality

¹ - NFER - National Foundation for Education Research, Slough, UK ² - 2006 GCSE Geography, French, ICT, Mathematics and Science

Using segmentationSegmentation improves marking quality and provides a more cost-effective use of marking resources by enabling each segmented question to be routed to a specific marking approach.

DRS can advise you on how to allocate each question to the most suitable marking approach.

Auto-marking: Simple one letter or one-word answers are imaged, sent for data keying and then placed into e-Marker®. All the answers given by candidates are listed in groups, inspected by a senior examiner and correct ones given marks. This enables very large numbers of answers to be marked by one person, bringing additional marking consistency. In addition, e-Marker® can provide the details of every answer for subsequent analysis, enabling question papers to be improved.

General marking: Simple answers of a few words, or diagrams and charts can be marked by well-trained markers who are not subject experts.

Expert marking: More complex answers, including essays, that require specialist knowledge, are reserved for expert markers – enabling the assessment and testing body to focus often scarce resources on the most important areas.

“ e-Marker® has

the ability to handle essay style answer booklets, mark question by question and handle processess via a single user interface

“

Worldwide specialists in examination servicesFor four decades, DRS has been supporting global awarding and assessment bodies, schools, colleges and universities with its secure examination processing and assessment data capture solutions including:

• exam entry and registration forms including photo capture• fully optimised examiner answer and mark sheets for electronic capture• multiple choice questionnaires and booklet production and processing• bespoke examination scripts production and processing• electronic examination marking solutions• sophisticated technologies designed specifically for scanning exam papers• key from image services.

If you are looking to reduce the administrative pressures on your establishment whilst improving the accuracy and reliability of your examination and assessment data, DRS can offer you proven solutions designed to deliver.

DRS is an associate member of:

DRS Data Services Limited | 1 Danbury Court | Linford Wood | Milton Keynes | MK14 6LR | EnglandTel: +44 (0)1908 666088 | Fax: +44 (0)1908 607668 | [email protected] | www.drs.co.uk

transforming the future of examination marking and services · 2015. 11. 2. · transforming the...

Documents