acec2010-titleacec2014.acce.edu.au/sites/2014/files/attachments/eexa… · web viewfigure 1:...

eExams transforming curriculum. Andrew Fluck and Mathew Hillier

eEXAMS TRANSFORMING CURRICULUM

Andrew Fluck1 and Mathew Hillier2

1University of Tasmania2University of Queensland

Abstract

The eExams project in Australia has sought to prove a sustainable, scalable and secure pathway for computers to be used in examinations. Initially developed at the University of Tasmania, it has been used in various degree courses and in the pre-tertiary sector across the state.

Over a thousand students have used the eExam System since its launch in 2009. Candidates bring their own devices to the examination, and boot up from a specially crafted USB stick. This provides the same modified Ubuntu software environment (based on Linux) and a full office suite to every candidate; access to the exam materials and a secure partition for the candidate’s answers.

As with any long-term software development project, the eExam System continues to evolve. Work on the version 5 began at the University of Queensland in 2013. This will add a copy of the learning management system Moodle on the USB stick to include a range of computer marked question types, such as short answer and multiple-choice. A restricted network connection to an institutional LMS is also possible, using a secure gateway.

This presentation illustrates the way users have made the transition from paper-replacement to post-paper examinations. The style of emerging post-paper exams illuminates the way in which curriculum change may be facilitated through this new assessment format.

Introduction

The fundamental role of computers in education at all levels is evolving. In many cases they are used to support the existing curriculum. As students become more able to bring their own devices, this situation may change to re-define curricula (Puentedura, 2013), perhaps even schooling itself (Downes, Fluck, et al., 2001, p.23). In a cautious way, the Exams project seeks to accelerate understanding and capability where it is prudent to do so.

In 2014 the inception of the new Digital Technologies subject is foremost in many Australian computer educators’ minds. They may have some difficulty assessing student achievement in the subject without using computers. It would seem anachronistic this subject should be wholly assessed on paper. This makes a case for a computer-based assessment environment that can be implemented on a class-by-class basis. A good solution will scale from individual classrooms to nation-wide assessments. The eExam System provides this range using free and open-source software whilst re-using USB sticks for each test.

In other subject areas, teaching topics and skills using school computers which will be assessed without them may be discordant to many teachers. One of the authors has personally witnessed the distress for both teacher and Year 5 student when the latter requested the use of a computer ‘like I normally do’ at the commencement of a NAPLAN literacy test. Our speculation is that the introduction of a method for conducting assessments on student computers in a fair and secure fashion will provide a useful lever for transforming curricula in the future. Our project and its future extension should change high stakes testing and by implication give students a wider range of contexts for assessing new digitally-based skills. These skills have been shown in the

of 8


parallel ‘Calculus for Kids’ project as outpacing chronological age by many years (Fluck, Ranmuthugala, Chin & Penesis, 2011). This is important because of the tension between the high rate of innovation in the information industries, and the slow rate of technology adoption in education. Computers are rapidly changing our world. But school curricula change slowly, at the mercy of politics, restricted resources for teacher professional development, and well-intentioned social inertia. An individual teacher desiring to innovate by presenting newly discovered knowledge in the classroom has to provide evidence of student achievement at the end of each year according to pre-ordained accountability frameworks, mostly using pen-on-paper techniques. There are tensions between information technology and school use as expounded by the former prime minister of Australia:

While ICT has fundamentally reshaped whole industries, revolutionized production processes and generated massive improvements in productivity in our workplaces, our education systems have been slower in adapting. (Gillard 2008)

Previous work

Paper-based exams in the UK are a major barrier to curriculum change (Ripley 2007, p.10). This helps to explain why the recent revision of the English National Curriculum, which schools have to implement from September 2014, makes almost no reference to digital technology. To tackle this deficiency, three ministers (Education, Skills and Enterprise, and Higher Education) have set up the Educational Technology Action Group (ETAG) in order to advise government on how to overcome barriers to transformation of the education system through the use of digital technology (Hancock 2014). There is extensive work on educational transformation, and widespread agreement that assessment is a major barrier to, or enabler of, change in education. We need to develop new ways of assessing people, to better reflect the knowledge, skills and personal attributes that are needed in the Information Age (Heppell 1994, p.154; Twining et al. 2006). There is a significant mis-match between current assessment practices and new learnings like computational thinking that digital technology facilitates (e.g. Ridgway & McCusker 2004, p.38; Venezky & Davis 2002, pp. 11-12). Computational Thinking (Wing 2006) is at the heart of the new subject released by Australian Curriculum, Assessment and Reporting Authority (ACARA) in February 2014. This project works to make this kind of thinking evident in many other subjects. There is strong support for the need to change how we assess learning in order to rectify this mis-match (for example Lemke and Coughlin 1998, p.18; Lewin et al 2000; McFarlane et al 2000; Barton 2001 pp.27-28; ICTRN 2001; Trilling and Hood 2001). Newhouse (2013, p.15) writing on computer-based transformational assessment identified the “lack of experience for students and teachers [as] a constraint in using ICT to support summative assessment, particularly where the stakes are high”.

As indicated above, accountability and assessment are strong levers determining the nature of skills and content in educational curricula at all levels. The qualities of e-assessment (if and how computers are used in the assessment process) and the extent of e-assessment are important determinants of the relationship between rapid technological innovation and socially conservative education. Ripley (2009) presents two ‘drivers’ of e-assessment: business efficiency and educational transformation. Drivers emanating from the business efficiency have focussed upon computer-based assessment. This is often implemented as multiple choice questions (MCQ), as is common within learning content management systems, and have the advantage of automated marking. There is extensive literature on the business efficiency aspects, but the eExam System is a new approach that aligns well with the need for educational transformation. Candidates start up their own personal computers from a specially crafted USB drive in an eExam.

The USB drive controls the computer throughout the exam, providing a secure operating system which prohibits access to any other drive or communication function. By putting the computer into the hands

of 8


of every examination candidate, assessors can leverage educational transformation and include the use of professional software tools into the curriculum. Knowing candidates can be asked to undertake highly complex investigations or engineering design work in the exam hall can boost expectations in the classroom. The technology for implementing examinations on computers is developing. Some systems rely upon institutional equipment and ‘thin client’ (dumb computers) with networking connections to a central server. These are reliant upon all elements in the communication link remaining operational throughout the critical period of the exam; if any one element, such as a wireless access point, should fail, the examination has been jeopardised. Also, each candidate is restricted to questions of a very simple nature within the confines of a web page window. Typically question material relies upon pre-set text stimuli, or diagrams, video or interactive elements at successively rising cost making them viable only for very large cohorts. Multiple choice question types are popular within this paradigm.Fluck, Pullen and Harper (2009) in explaining the eExam system, described how candidates boot their own personal computers from a live operating system USB and complete short and essay style questions while preventing access to unauthorised networking or other data sources. They are also able to run specialist software (beyond Office tools) within the secure eExam system. The specialist software can include Windows programs running under the Wine compatibility layer or any other program chosen by the assessor which runs in a native Linux/Ubuntu environment. Examples include educational software (which candidates have critiqued in the examination, or digital multi-media products from school pupils which candidates have assessed). This goes beyond the limited MCQ aspects of previous systems.

The eExam System is therefore scalable (not reliant on institutional equipment limitations), resilient (communications infrastructure is not essential), fair and unbounded (every candidate gets the same full operating system environment; no-one has personal access to software unavailable to other candidates). The eExam System provides a unique opportunity to transition to paper-replacement exams on computer, and then beyond to post-paper exams. This simple pathway for adoption is possible because existing exams translate easily to the new computer-based environment (in the paper-replacement stage), whilst laying the foundation for running complex software applications as part of the exam in post-paper tests. The technological advantages with this implementation pathway are a winning combination. The icing on the cake is the free and open source nature of the system, eliminating licencing costs and extra fees for students.

Figure 1: Example security image on desktop of student personal computer booted from eExam USB.

of 8


Deciding upon an eExam Platform

The idea of using word processors for text production appears un-controversial; but improving academic performance through the use of advanced software is far rarer in other fields such as mathematics or science. Yet important discoveries are being made in the latter, for instance the award of Nobel prizes for computational chemistry in 1998 and 2013. This symbiosis of human endeavour with computational techniques generates important new knowledge to which students need access. By allowing examination candidates to demonstrate proficiency using software tools, they will be able to perform more complex tasks which will also be more authentic in relationship to the real world and professional practice.We argue the use of computers in final exams (as a component of balanced assessment methods) will remove hurdles to curriculum transformation. There is fierce competition from proposers of various exam-on-computer vendors. At the heart of this contention are key philosophical debates relating to costs, the ideal format, and technological delivery platforms. Who should pay – should the candidate, teaching institution or qualification authority pick up the cost? Would a free, open source (FOSS) method be better (compare Microsoft Windows – a commercially available operating system, with Ubuntu, a FOSS equivalent). Beyond this debate is that of reticulation. Is the current internet infrastructure reliable, resilient and sufficiently capacious to be trusted with people’s lives as determined by the outcome of a high stakes assessment? Finally, in this tense area, can all assessment questions be sufficiently well posed within the confines of a web-window as part of a locked-down test oriented software application; or should we be looking to assess student ability within an unrestricted computer operating system where specialist software of any complexion can be run? Is there an implicit cultural bias embedded in a particular system? Would Australia be better to adopt a system devised locally, or to import a commercial system from the USA, or a FOSS system from Finland?

This area of intellectual enquiry is new ground. It is particularly important we gather impartial, unbiased knowledge in this area because adoption of particular strategies can have a knock-on effect. Thus, when a state university adopts an eExam strategy, there is huge incentive for feeding school systems to adopt the same eExam approach with its associated benefits or restrictions (Fluck & Mogey, 2013).

Current status of eExams

eExams have been used at the University of Tasmania since 2009, and with official sanction in the formal exam halls since 2011 following their acceptance by academic senate in March of that year. A parallel adoption was made by the Tasmanian Qualifications Authority in 2011, when a paper-replacement eExam was used for Information Technologies and Systems at Year 11/12 pre-tertiary level statewide. This was followed in 2012 by a post-paper eExam, where candidates were required to view a web-site on screen – patently not possible on paper. In 2013 the Authority went further and held an open internet-accessible exam, which pushed the boundaries yet again.

Figure 2: An eExam USB.

of 8


Since that time, eExams have been held in a wide range of disciplines, ranging from educational technology, constitutional law and history. They have also been held in a wide range of places – in the conventional exam hall alongside candidates using pens; in formal computer laboratories; in public libraries in Amsterdam (for students enrolled by distance). Three key security features make eExams fair. All communication ports are blocked or monitored by the modified implementation of Ubuntu, as are all accesses to data storage devices other than the USB boot device. The last feature is purely visual. Assessors are asked to supply a unique image to be placed on the desktop at bootup; one which candidates could not acquire elsewhere. For non-technical exam supervisors, this image assures them each candidate has booted up into the exam environment rather than using the operating system on their own laptop hard drive. This guarantees every candidate has access to precisely those software applications permitted for the exam.

In most cases, assessors begin by offering their students the choice of pen or keyboard for the examination. This makes a low-impact change which individuals control their rate of adoption. We refer to this stage as paper-replacement exams, because to all intents and purposes the nature of the assessment is unchanged. Over various cycles, more and more students select the keyboard, and when a sufficiently large proportion do so, the next stage can begin.

This is the post-paper exam. This kind of exam contains elements which are difficult, if not impossible to present on paper. It may be something as simple as a full colour photograph; or perhaps include a video file to be viewed as a stimulus. For instance, a mathematics education exam contained a video of a teacher introducing a fractions topic – candidates were invited to comment on the pedagogical techniques used in the lesson. These two examples could still have been delivered via a web-page, but using current wireless network access point standards there would be great difficulty simultaneously downloading a movie to several hundred candidates.

Table 1: Changing aspects of eExams

From 2007 To 2013120 candidates Over 1000 candidatesPaper replacement .. to full multimedia and questions requiring software operation.Institutional equipment ..to personally owned computersEducational computing UTAS Law, Mathematics pedagogy, TQA Information Technology & SystemsLaunceston campus Statewide, interstate and overseas

One of the authors trains teachers to assess digital products created by schoolchildren. These can include narrated powerpoints with animation. This is an example it would be difficult to embed into a web-page. The digital product could be a file from almost any program – not just from an Office application. Therefore the full operating system available to each candidate becomes more important. Another skill taught is that of evaluating digital educational resources. Using the eExam System, cadidates can be asked to review a novel piece of software against pre-set criteria. Another post-paper examination could ask candidates to use a particular software tool to accomplish a professional task and submit the output file (a bridge design, an evaluation of a new molecule, a statistical analysis of two very large datasets) as part of their response.

In this way the eExam System allows a transition to a testing environment which is far more sophisticated than pen-on-paper. This facet provides strong support for curriculum transformation, making it possible to assess new skills far beyond those which can be demonstrated using pen-on-paper. Figure 3 illustrates the kinds of question we anticipate:

of 8


Watch the video Complex DNA [95 seconds] and use the enzyme replication simulation software to construct a molecular junction inhibiting the binding process.

Figure 3: Example of a possible future post-paper examination question

As at 2014, eExams are extending into several subjects at the University of Tasmania, whilst trials are commencing at the University of Queensland.

Conclusion and future

The adoption of eExams by different bodies, even in the same state, has not led to uniform standards or procedures. This is not intrinsically a bad thing. It surely indicates vibrancy and a range of experimentation with an innovation. However, this diversity has illuminated some areas for discussion and resolution of important new social issues.

For instance, university lecturers have accepted the idea of a graduated progression into eExams with the initial foray being a paper-replacement stage. This means each candidate can be given the option of using keyboard or pen, making individual choices up to the last minute (and sometimes beyond). In this environment, ‘equity’ has been interpreted as ‘an equal chance to choose a suitable text-production tool’. However, in the Year 11/12 context, with high-stakes testing leading to major life-forming educational opportunities, the definition of equity became ‘one in, all in’. All candidates in this sector were required to use a computer for the whole examination. No student could argue they were disadvantaged through the use of a different assessment environment to any other.

Working through these divergent understandings of ‘equity’ will take some social adjustment and open discussion. It’s especially important this happen, otherwise the paper-replacement stage will become a hurdle preventing the more strategically vital post-paper stage emerging.

A similar project in Finland, the DIGABI project, has gone one step further. The focus of a hacking competition to identify shortcomings of the original open-source core code, the Matriculation Examination Board of Finland has published a schedule for the conversion of all examinations (presumably for Years 11/12) to eExams by the year 2020. This international adoption of the technique is instructive for other societies wishing to find some consistency in high-stakes assessment ‘going digital’.

The future of the eExams project will build upon the current funding provided by the Australian Government Office for Learning and Teaching. It will incorporate Moodle, an open-source learning management system which facilitates automatically marked questions of several complexions. Trials for this are expected in late 2014.

There is also discussion around the concept of establishing an ‘eExams Foundation’ analogous to the Moodle Foundation, to promote further development of the open-source code base and underpin related commercial services. This will provide a governance model for eExam development, a forum for inter-institutional sharing of ideas and a launchpad for supportive relationships with operational services commercial entities. The eExams project will continue to develop complementary research capabilities across the social

of 8


and economic sciences that can connect data-driven and analytical models to promote evidence based policy development. This project has the potential to generate student assessment material in digital format in vast quantities, opening up the possibility of applying big data techniques. Providing ethical clearance is obtained, we will use such data to answer questions such as ‘how much more text do students type compared to handwriting?’ and ‘do candidates achieve at higher levels when using computers in a paper-replacement examination?’ Answers to such questions will provide the evidence to drive curriculum transformation policy adoption.

References

Barton, P. E. (2001). Facing the hard facts in education reform. Princeton, NJ: Educational Testing Service. http://www.ets.org/Media/Research/pdf/PICFACINGFACTS.pdf

Downs, T., Fluck, A., Gibbons, P., Leonard, R., Matthews, C., Oliver, R., Vickers, M., Williams, M. (2001). Making better connections: Models of teacher professional development for the integration of information and communication technology into classroom practice. Canberra: Australian Curriculum Studies Association, Australian Council for Computers in Education, Technology Education Federation of Australia.

Ferrell, G. (2012). A view of the Assessment and Feedback Landscape: baseline analysis of policy and practice from the JISC Assessment & Feedback programme. JISC, UK. http://www.jisc.ac.uk/media/documents/programmes/elearning/Assessment/JISCAFBaselineReportMay2012.pdf

Fluck, A and Mogey, N. (2013). Comparison of institutional innovation: Two universities' nurturing of computer-based examinations. Learning while we are connected: proceedings of the 10th IFIP World conference on computers in education, 1-7 July, Poland, pp. 11-20. http://edu.rsei.umk.pl/wcce2013/publications/v1/V1.2_010-Fluck-fullR-FPR.pdf.

Fluck, A. and Pullen, D.L. and Harper, C. (2009) Case study of a computer based examination system. Australasian Journal of Educational Technology, 25 (4) 509-523.

Fluck, A and Ranmuthugala, SD and Chin, CKH and Penesis, I, (2011). Calculus in elementary school: an example of ICT-based curriculum transformation, SITE 2011, Nashville, USA, http://www.editlib.org/j/SITE/v/2011/n/1 , pp. 3203 - 3210. ISBN 1880094843.

Fluck, A. (2013). Implementation of on-campus digital examination practices. Transforming Assessment in a Digital Era conference, RACV City Club, Melbourne, 31 July.

Gillard, J. (2008). Address to the Australian Computers in Education Conference, Canberra, 1 October 2008. http://www.deewr.gov.au/Schooling/DigitalEducationRevolution/Documents/ACECMinistersSpeech_011008.pdf.

Hancock, M. (2014). Speech on education technology at BETT show. https://www.gov.uk/government/speeches/matthew-hancock-speech-on-education-technology-at-bett-show

Heppell, S. (1994) Multimedia and Learning: Normal children, normal lives and real change, in J. Underwood (Ed) Computer based learning: Potential into practice. London: David Fulton Publishers.

ICTRN (2001). Building an ICT research network, Helping to create schools of the future , June 2001, Coventry: Becta

Lemke C. and Coughlin E.C. (1998). Technology in American Schools: Seven Dimensions for Gauging Progress. A Policymaker's Guide. Milken Exchange on Education Technology, Milken Family Foundation. http://www.mff.org/assets/Uploads/newsroom_archive/publications/ME158.pdf

Lewin, C, Scrimshaw, P, Mercer, N and Wegerif, R (2000). Linking Schools and Home with Low-cost Word- processors. European Conference on Educational Research, Edinburgh.

of 8


McFarlane, A.E., Bonnett, M.R., Williams, J. (2000). Assessment and Multimedia Authoring – a technology for externalising understanding and recognising achievement. Journal of Computer Assisted Learning, 16, 201–212.DOI: 10.1046/j.1365-2729.2000.00133.x

Newhouse, P. (2013). Literature Review and Conceptual Framework. In P. J. Williams and C. P. Newhouse (Eds.) Digital Representations of Student Performance for Assessment, 9-28. Sense Publishers: The Netherlands.

Puentedura, R. R. (2013, May 29). SAMR: Moving from enhancement to transformation [Web log post]. Retrieved from http://www.hippasus.com/rrpweblog/archives/000095.html

Ridgway, J. and McCusker, S. and Pead, D. (2004). Literature review of e-assessment. Futurelab, Bristol. http://dro.dur.ac.uk/1929/1/Ridgway_Literature.pdf?DDD29+ded0kmt

Ripley, M. (2007). E-assessment – an update on research, policy and practice. FutureLab: Bristol. www.futurelab.org.uk/litreviews

Ripley, M. (2009). Transformational Computer-based Testing in The Transition to Computer-Based Assessment: New Approaches to Skills Assessment and Implications for Large-scale Testing. F. Scheuermann & J. Björnsson (Eds.) pp. 92-98.

Trilling, B and Hood, P (2001). ‘Learning, technology and educational reform in the knowledge age or “We’re wired, webbed, and windowed, now what?”’, in Paechter, C, Edwards, R, Harrison, R and Twining, P (Eds) Learning, space and identity, pp7–30, London: Paul Chapman Publishing Ltd

Twining, P; Broadie, R; Cook, D; Ford, K; Morris, D; Twiner, A. & Underwood, J. (2006). Educational Change and ICT: an exploration of Priorities 2 and 3 of the DfES e-strategy in schools and colleges. Coventry: Becta. (ISBN 1853794651) http://is.gd/niDK3f

Venezky, RL & Davis, C. (2002). Quo Vademus? The Transformation of Schooling in a Networked World. Paris: OECD. http://www1.oecd.org/innovation/research/2073054.pdf

Wing, J. (2006). Computational Thinking. Communications of the ACM, 49(3)33-35.

of 8