using games based technology in formal assessment of learning

Upload: mhairi-mcalpine

Post on 07-Apr-2018

219 views

Category:

Documents


0 download

TRANSCRIPT

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    1/12

    Using Games Based Technology in Formal Assessment of LearningMhairi McAlpine, Linn van der Zanden and Veronica Harris,

    Scottish Qualifications Authority, Glasgow, [email protected]@[email protected]

    AbstractWhile scoring is a routine part of most computer games, and games based learning continues to grow,the formal assessment of learning within a gaming context has been somewhat neglected. Theopportunities afforded within games to control the environment have advantages for both learning and theassessment of learning in that context. Allowing a serious game to produce output which can then beused for the assessment of prior learning may overcome practical and validity issues where simulated.

    In 2008, the Scottish Qualifications Authority with the support of the European Social Fund, starteddeveloping Games Based assessments for the hurdle tests which form part of their pre-existingvocational qualifications. By embedding the assessment into a simulated work environment it was aimed

    to provide a more authentic and valid assessment than traditional pencil and paper. To ensure parity withexisting assessments and the maintenance of standards, a report was produced at the end of the gamewith the responses given in the game used to complete the answers to questions. This report wasidentical to the outcome of a completed pencil and paper test. Initial trials based on a retailingenvironment suggested that this approach would be popular with vocational candidates and the teacherswho assessed them. There were however a number of practical and organisational difficultiesencountered which need to be overcome before we could mainstream this innovative assessmentmethodology.

    This paper examines whether Games Based Assessment can provide a practical, cost effective andacceptable method of assessing candidates in vocational subjects, whether it can provide a moreauthentic assessment experience and as the impact that the change in mode has on the validity andreliability of the qualification. It also goes on to look at the natural progression of games based learninginto games based assessment once the medium becomes more mainstream

    Keywords: Assessment, SQA, vocational education, outcome based learning, games basedassessment, formal assessment, hurdle test, vocational assessment.

    mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]
  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    2/12

    Introduction

    SpaceWardeveloped in 1961 at MIT is widely accredited as being the first computer game and by just thefollowing year, the Chicago Tribune was reporting on the use of computer games for learning. In the 40years since, computer game technology has advanced enormously and its potential for use in Teachingand Learning has an increasing profile. Within Scotland we have the Consolarium at Learning andTeaching Scotland (Independent, 2010) a dedicated Games Based Learning Thinktank which aims toexplore how computer games can be used to best effect to support the new Scottish Curriculum forExcellence, while in the wider UK both FutureLab (FutureLab, 2004) and the now defunct BECTA(BECTA, 2006) have commissioned large scale research into the potential for the use of games tosupport and enhance learning in classroom, informal and domestic situations.

    One area which has not been so widely explored is the potential for computer games to assess learning.On the face of it, it looks like an obvious use for the technology especially as the concept of assessmentthrough scoring has been built in from the very earliest computer games. However, in practice there areconsiderable challenges to be overcome at the assessment design and practical implementation stagesas well as ensuring that the conditions required for formal certified assessment of learning are upheld.

    Development of Skills for Work Qualifications

    In 2008 the Scottish Qualifications Authority successfully applied to the European Social Fund foradditional monies to support the development of the new Skills agenda in particular the new Skills forWork qualifications.

    It had been recognised for some time that there was a need for vocational courses for young people inthe 14-16 age group, which will give them the opportunity to develop knowledge and skills which will helpto prepare them for the world of work. To meet this need, in 2006 SQA started to develop Skills for Workcourses aimed at pupils in the 14-18 age range with a firm focus on vocational rather than academicskills. These courses differed from other vocational provision because they focused on the genericemployability skills needed for success in the workplace, offering opportunities for learners to acquirethese critical generic employability skills through a variety of practical experiences that are linked to aparticular vocational area such as construction, hairdressing, hospitality etc.

    The first qualifications were offered as school/college partnerships and intially aimed at students whowere not engaging with school. These students were likely to become poor attenders and disengagefrom learning. In many cases they would re-engage with learning after their formal school leaving date, ina college or a work based learning setting frequently entering with no qualifications. The hope was thatthese new qualifications could engage their interest and meet their needs as they embarked on thetransitition from school to college or employment while offering them a qualification en route. Thesequalifications became very successful not only for the disengaged learners they were aimed at, but alsofor more motivated learners, who had a clear career focus and wanted an insight into the industry theywished to enter.

    The qualifications conformed to the Scottish National Qualifications design rules, which ensure a commonstructure across different subject areas. These design rules specify that National Qualifications should be

    comprised of three hurdle tests ungraded internally marked tests which candidates are required to passto continue with their course, as well as an externally marked final component.

    In discussing with the qualification developers how they felt that computer assisted assessment couldenhance the Skills for Work experience the primary issues that were raised was the difficulty ofmaintaining a physical portfolio where educational provision and assessment could be undertaken in anumber of different locations, including school, college and workplace as well as the barriers that somestudents found in the requirement for hurdle tests. SQA requires these tests to be passed to ensure theappropriate standards are being applied and that candidates entering the final assessment are suitably

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    3/12

    prepared. Many students, however, felt that the writing that was expected from them in these tests wasout of kilter with the type of learning that they were doing in the workplace based setting and also theprimary form of assessment which was folio based.

    Rationale for the Use of Games Based Assessment

    Vgotsky (1962) focused on the role of the interpersonal in learning and development and the importanceof the socio-cultural setting of learning. This is further developed in situated learning theory as developedby Lave (1988) which suggests that learning is primarily unintentional and occurs only when it is situatedwithin authentic activity, context and cultural setting. He suggests that knowledge and skills are onlyacquired through the inclusion in a community of practice which embodies the beliefs and behavioursthat the student is to acquire. Brown, Collins & Duguid (1989) have further developed social learningtheory through the concept of a cognitive apprenticeship which supports learning in a domain byenabling students to acquire, develop and use cognitive tools in authentic domain activity. Learning, bothoutside and inside school, advances through collaborative social interaction and the social construction ofknowledge.

    The difficulties of a physical portfolio were easily solved by the provision of an e-portfolio system, howeverthe difficulties encountered by the hurdle tests were more challenging. The attractiveness of the

    qualification in the first place was premised on their practical nature and written assessments underminedthat attractiveness. To overcome this, some of the assessments of the hurdle test involved role play andsimulated situations, however it was difficult to make that standardised with limited control over the non-assessed character in the role play. It was thus desirable to reinforce the situational backdrop of Skillsfor Work courses in the assessment design.

    SQA decided to develop Games Based Assessment as it could provide the look and feel of being in aworkplace environment with the opportunity to interact with another character within it in a standardisedmanner. It was hoped that this could increase the validity and reliability of the assessment, while alsomaking it more authentic for the candidates. However, while games have been used extensively inlearning activities for many years, there is little literature on the use of Games for formal assessmentpurposes. Indeed a major review of games based learning conducted for JISC (de Freitas, 2006)mentioned assessment and accreditation as one of the pitfalls for the use of games based learning and

    a potential stumbling block for its widespread adoption.

    Description of the Games Based Assessments Developed

    Classified by the Hertz System (Hertz, 1997), the games produced for Skills for Work qualifications are amix of role play and simulation. In the games you adopt the character of a worker in the environment the exact role is undefined - however it would probably be most akin to a work placement student beingassessed. At the same time you are asked to do specific tasks within the environment although not atrue simulation such as a flight simulator - there is still the immersive approach. Candidates wereassessed on exactly the same performance criteria as per the traditional assessment and a report wasproduced (see Appendix 1) detailing the result. This report was designed to be as close as possible tothe test script the candidate would have produced, had they been assessed in the traditional manner.Two games were developed for launch in Autumn 2010.

    The first, in Retailing, was designed for candidates at SCQF1Level 5 (EQF Level 3). In this assessment,the candidate found themselves in a shopping mall, which contained an electrical store, a fashion storeand a food retailer. Candidates could go into the stores in any order where they would be met by asupervisor who would explain the organisational guidelines specific to that store. In the electrical store,they were asked to approach and assist customers and recommend them a product based on the resultsof their questioning; in the food store they were asked questions about key pieces of consumer legislationand in the fashion store they were asked to identify the features and benefits of products as well as select

    1 Scottish Credit and Qualifications Framework

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    4/12

    complimentary or alternative products. Once the candidate had completed all of the required tasks in thethird store that they visited, they were approached by a customer wishing to complain. The candidatewas then required to deal with the complaint by following the organisational guidelines for the store thatthey were in.

    The second, in Health Sector, was designed for candidates at SCQF Level 4 (EQF Level 2). In thisassessment the candidate found themselves in a hospital. They were directed to a supervisor, who gavethem general instructions on navigation and behaviour within a hospital environment before beingdirected to the radiologist. On the way they were intercepted by a nurse and asked questions about theirpersonal responsibility for infection control before finding the radiologist who then questioned them abouthow biomedical devices, cross contamination and droplet spread can spread infection in a health sectorenvironment. They were then directed to a nurse who asked them to perform a risk assessment of thehospital waiting area by walking round the environment, identifying hazards, the risk associated with eachhazard and the corrective action that could be undertaken to minimise the risk. On completion they weredirected back to the supervisor, who simulated a sneeze following which the candidates were asked whataction they should now take.

    Developing the Framework of the Assessment

    The Conditions of Learning(Gagne, 1965) identifies a nine-step process for instructional design usingevents of instruction. This 9 step model has been much utilised in game based learning (Beker, 2005).Some studies have found that the connection between specified learning outcomes and the learning thathas been undertaken in games can be less than linear (Egeneldt-Nielsen, 2005) and there are moves toconsider other ways in which the learning undertaken in games may be documented. However, giventhat learning outcomes tend to be at the heart of current competence based qualifications the dominantform within the vocational sphere, an outcome based approach provides a starting point for exploration.Although not specifically designed for assessment, Gagnes framework allows us to consider theassessment event as a process and consider how a student can move through the event in a meaningfulway.

    The figure below shows these instructional events in the left column and the associated event in the gamebased learning in the right column. The first six instructional events follow the Gagne sequence, howeverstage 7 providing feedback, was deliberately missed. The software allows for instant feedback to begenerated as candidates respond to an activity, but a primary objective was to ensure comparability witha paper examination where candidates would not receive feedback until after the whole assessment wascompleted. Thus we removed the feedback option from the game.

    Table 1: Gagnes 9 events of instruction.

    Instructional Event

    Gain attention The learner is introduced to the environment and given general directionsabout where they find themselves and what they will be doing

    Inform of objective At each stage the objective and performance criteria that would beassessed were clearly signaled to the learner

    Stimulating recall The environment itself is designed to be a recall trigger, as they moveabout into different areas, they should encounter situations which arefamiliar to them from their learning

    Present stimulus The candidate is guided to a virtual area where a scenario has been setup with a character ready to interact with the candidate

    Provide guidance Guidence on how to response to the character is provided on screenthrough instructions

    Elicit performance Following the guidance provided

    Provide feedback This stage, although possible, is explicitly not done to replicate theconditions of a traditional assessment.

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    5/12

    Assess performance Performance is assessed by the tutor who receives a report of thecandidates

    Enhance retention Again this stage was not considered to be suitable for a formal summativeassessment

    The games development engine, Thinking Worlds - which SQA used as a platform, had been selected

    partly on the low cost of development but also the potential for rich interactivity through a first personperspective. SQA felt this would have the desired effect of placing the candidate within the virtualenvironment. While the software had the possibility of automated marking, SQA chose not to implementthat module.

    A critical test of the acceptability of the assessment was ensuring that the teacher remained in control ofthe assessment. The assessment is not graded, but assessed on a pass/fail basis reflecting the openended nature of the original assessment. Many of the responses that were available for the candidates tochoose were not strictly right or wrong, but closer to a correct response or further way. By providing areport and allowing the teacher to judge whether they have sufficiently demonstrated all of the requiredcompetencies the module can be assessed more holistically as well as retaining the principle of internalassessment for the hurdle tests.

    Development of the assessment

    It was critical to us that the assessment developed through the games based assessment wascomparable to the hurdle tests generated on paper. As per a paper development, subject specialistswere recruited to generate the assessment content while artwork the artwork was developed. Thedevelopment of artwork prior to the content development proved to be erroneous, as unnecessarymaterial had been developed on an ad hoc basis., while further development of essential artwork had tobe undertaken once the content was written.

    Relating tasks in the game environment to the outcomes and performance criteria the fundamentalbuilding blocks of assessment units within SQA proved much more difficult in the game environment.Balancing the desire to utilize the environment fully, the need to adhere to the criteria and the limitationsof the software, proved a difficult task. Subject specialists who develop assessment material for SQA tend

    to be experienced classroom practitioners, although as a mature demographic, less familiar withcomputer games (Social Gaming Monitor, 2010). It quickly became apparent that there was a tensionbetween developing a good game which fully utilised the possibilities of the environment and ensuring avalid and reliable assessment.

    On a number of occasions the subject specialists deviated from the assessment specifications in order toenhance the playability of the game as they discovered new possibilities and explored different avenues.The subject specialists developments were overseen by the Qualifications Managers who take ultimateresponsibility for assessment materials which are published by SQA. Qualification Managers usually takea hands-off approach to development; given the experience of the subject specialists, work is usually ofhigh quality and any errors can be righted relatively quickly. At checking stage however it becameapparent that the unfamiliarity of the medium had led the subject specialists to deviate from thespecification. As the development had already been done, an expensive process of re-engineering the

    materials to fit with the learning outcomes and performance criteria had to be undertaken under a muchhigher level of supervision than normal as well as further revision.

    In moving to an operational development, it was decided that additional checkpoints were required overand above what would normally be required for a hurdle test development, given the potential fordeviation from the specification, the unfamiliarity of the subject specialists with the medium for which theywere developing content and the high cost of rework. The development of the artwork specification stagewas moved to after content signoff to ensure that the artwork developed supported the assessmentmaterial written, rather than the material being written around the environments which had beendeveloped.

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    6/12

    Initial Piloting and Feedback

    In June 2009 a prototype of the game was taken to schools and evaluated in context with Skills for Workcandidates. Candidates and tutors had the opportunity to play the game and then participated inindividual semi-structured interviews with an external educational consultant.

    All of the candidates on which we tested the game played games recreationally, however none had priorexperience in using a game as a learning or assessment tool. All of them found controlling the charactereasy and managed both movement and interactions with no difficulty, reporting them as easy to use.They did note that the game was slow and the graphics not as smooth as they were used to a productof a lower cost development engine than that which would be used in commercial games although theydid consider them realistic.

    Pupils reported that they would be happy to have their knowledge, skills and abilities tested through thegames based medium and they felt that it made the assessment more fair. They considered that thegames based environment was helpful for them in creating a context for them to demonstrate theirabilities adding that they were no longer required to imagine a context which was given to them through awritten description. They also considered that this type of hurdle test would be less stressful than a

    traditional pencil and paper version.

    Teachers identified a number of barriers to introducing a game based assessment primarily practical.ICT provision in schools tends to be available only through specialist ICT suites. Implementing gamesbased assessment requires obtaining a booking slot and ensuring that all required software is availableon the machine prior to the assessment session. As this is controlled at a local authority level, this ispotentially a time consuming process. They foresaw few barriers for candidates attempting anassessment in this manner, instead suggesting that the visual presentation may overcome some of thedifficulties that candidates had with written material.

    A copy of the questions used to support the semi-structured interview are included as Appendix 2.

    Report Generation, Access Management and e-Portfolio Integration

    Once the game was developed and functional, it became apparent that there were a number ofsurrounding conditions associated with summative assessment through a game, which would not be socritical within learning activities.

    As it was decided not to automate the marking but instead to produce a report, SCORM compliant datawas extracted from the game and a report generated from the output. In order to verify candidate detailscould be verified through a central access management system rather than locally, the game required anonline rather than standalone environment. Furthermore, in a traditional hurdle test, candidates would begiven as much time as they required and could change their answers. However the game was linear, soonce candidates had undertaken a virtual task, they could not return and there was also a requirementthat once the report was submitted by a candidate it was not viewable, just in case they showed it toanother candidates who had not yet undertaken the assessment. Thus maintaining the associated

    assessment conditions of a hurdle test became a major focus of the latter part of the development.Ensuring accessibility, security and the ability of candidates to review and revise their script prior tosubmission required technical solutions.

    In order to replicate the conditions, it became clear that the game had to be housed within a largerenvironment which could ensure the authentication and script management services essential tooperational use. As mentioned, the folio of evidence that candidates were required to produce as part ofthe assessment could be held in an e-portfolio, provided as part of the qualification. However while thegame platform was SCORM compliant, the e-portfolio which housed the evidence generated by Skills forWork candidates was not. Consequently a Moodle installation hosting the game was integrated into the

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    7/12

    e-portfolio, so that access to the game could be restricted and the final output would only be given totutors once the candidate had approved it.

    Candidates were allowed to retake the game as many times as they wished within the one session andview the report produced at the end. Once they were happy with the report, they submitted it forassessment. Following this, the candidate had no further access to the report, it was only available to thetutor for online marking, ensuring the security of a completed script.

    Conclusion and Next Steps

    The formal assessment of learning through a games based approach has posed considerable challenges.The main issues and steps that we have taken to address them have been outlined below.

    1. Aligning the game based activities to the learning outcomes and performance criteria particularly where possibilities are broad and errors may be expensive to correct. For futuredevelopments we are ensuring that the development of assessment content which is carefullyaligned to outcomes and performance criteria is prioritised and that transparency overdevelopment is achieved through multiple signoffs.

    2. There are challenges with subject specialists working in a new and unfamiliar medium. For new

    qualifications in the Skills for Work suite, we are working closely with the qualificationsdevelopment staff to make them aware of the possibilities of using games based assessment, sothat outcomes and performance criteria are written in a manner which does not precludeinteractive virtual assessments and awareness is raised through demonstrating existingdevelopments.

    3. The conditions required for a formal assessment can be onerous once translated to an onlineenvironment, balancing the need for security with the need for candidates to have an accessibleroute to assessment. The integration of the e-portfolio system to manage the access to theassessment, report the outcome and ensure the security of the completed script should overcomethese issues.

    The games piloted in 2009 have now been redeveloped and integrated into the e-portfolio product via anintermediate Moodle installation. The first developments in Retailing (SCQF Level 5/EQF Level 3) andHealth Sector (SCQF Level 4/EQF Level 3) will be available for live candidate use from August 2010, withfurther developments in Energy (SCQF Level 5/EQF Level 3), Uniformed and Emergency Services(SCQF Level 4/EQF Level 2) and Health Sector (SCQF Level 5/EQF Level 3) following in early 2011.Additional games based assessments are planned to be made available for August 2011.

    Initially they are launched as a supplement to the existing provision and we are encouraging the centresto provide them as a revision test for the first year rather than as an acceptable form of assessmentevidence, while we gather comparative data for the games based and traditional paper versions. Weintend to compare the item level marks obtained by candidates on the paper based and on the gamebased version and use that data to check the validity and reliability of the games based assessments, aswell as monitor the impact that they may have on any sub-group of candidates prior to a full launch as an

    assessment methodology.

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    8/12

    References

    Becker, K. (2005). How are games educational? Learning theories embodied in games. Proceedings ofDiGRA 2005 Conference. Retrieved February 15, 2007 fromhttp://www.digra.org/dl/db/06278.23299.pdf

    Becta (2006) Computer games in Education Project report available athttp://research.becta.org.uk/index.php?section=rh&&catcode=&rid=13595

    Brown, J.S., Collins, A. & Duguid, S. (1989) Situated cognition and the culture of learningEducationalResearcher, 18(1), 32-42.

    de Freitas, S. (2006) Learning in immersive worlds. A review of game-based learning. available at:http://www.jisc.ac.uk/media/documents/programmes/elearninginnovation/gamingreport_v3.pdf

    Egeneldt-Nielsen (2005) Can Education and Psychology join forces available athttp://www.nordicom.gu.se/common/publ_pdf/222_egenfeldt_nielsen.pdf

    FutureLab (2004) Games and Learningavailable athttp://www.futurelab.org.uk/resources/documents/lit_reviews/Games_Review.pdf

    Gagn, R. M. (1965) The conditions of learning and theory of instruction New York, NY: Holt, Rinehart &Winston

    Independent (2010) How computer games discovered virtuous realityavailable athttp://www.independent.co.uk/life-style/gadgets-and-tech/gaming/how-computer-games-discovered-virtuous-reality-1871927.html

    Lave, J. (1988) Cognition in Practice: Mind, mathematics, and culture in everyday life. Cambridge, UK:Cambridge University Press.

    Social Gaming Monitor (2010) available athttp://4.bp.blogspot.com/_RHJCq8Jes1s/S6zp7o8TtmI/AAAAAAAABMA/m_hOgR0T4TM/s1600/Social_Gaming_Monitor_International_tn51.png

    Vygotsky, L. S.(1962)Thought and language. Cambridge, MA: The MIT Press.

    http://www.digra.org/dl/db/06278.23299.pdfhttp://research.becta.org.uk/index.php?section=rh&&catcode=&rid=13595http://research.becta.org.uk/index.php?section=rh&&catcode=&rid=13595http://www.jisc.ac.uk/media/documents/programmes/elearninginnovation/gamingreport_v3.pdfhttp://www.nordicom.gu.se/common/publ_pdf/222_egenfeldt_nielsen.pdfhttp://www.futurelab.org.uk/resources/documents/lit_reviews/Games_Review.pdfhttp://www.independent.co.uk/life-style/gadgets-and-tech/gaming/how-computer-games-discovered-virtuous-reality-1871927.htmlhttp://www.independent.co.uk/life-style/gadgets-and-tech/gaming/how-computer-games-discovered-virtuous-reality-1871927.htmlhttp://4.bp.blogspot.com/_RHJCq8Jes1s/S6zp7o8TtmI/AAAAAAAABMA/m_hOgR0T4TM/s1600/Social_Gaming_Monitor_International_tn51.pnghttp://4.bp.blogspot.com/_RHJCq8Jes1s/S6zp7o8TtmI/AAAAAAAABMA/m_hOgR0T4TM/s1600/Social_Gaming_Monitor_International_tn51.pnghttp://www.digra.org/dl/db/06278.23299.pdfhttp://research.becta.org.uk/index.php?section=rh&&catcode=&rid=13595http://www.jisc.ac.uk/media/documents/programmes/elearninginnovation/gamingreport_v3.pdfhttp://www.nordicom.gu.se/common/publ_pdf/222_egenfeldt_nielsen.pdfhttp://www.futurelab.org.uk/resources/documents/lit_reviews/Games_Review.pdfhttp://www.independent.co.uk/life-style/gadgets-and-tech/gaming/how-computer-games-discovered-virtuous-reality-1871927.htmlhttp://www.independent.co.uk/life-style/gadgets-and-tech/gaming/how-computer-games-discovered-virtuous-reality-1871927.htmlhttp://4.bp.blogspot.com/_RHJCq8Jes1s/S6zp7o8TtmI/AAAAAAAABMA/m_hOgR0T4TM/s1600/Social_Gaming_Monitor_International_tn51.pnghttp://4.bp.blogspot.com/_RHJCq8Jes1s/S6zp7o8TtmI/AAAAAAAABMA/m_hOgR0T4TM/s1600/Social_Gaming_Monitor_International_tn51.png
  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    9/12

    Appendix 1: Excerpts from Report

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    10/12

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    11/12

  • 8/6/2019 Using Games Based Technology in Formal Assessment of Learning

    12/12

    Appendix 2: Evaluative instrument for semi-structured interview

    It should be noted that the questions below were used as the basis for a semi-structured interviewconducted individually with each of the teachers and pupils following the trial

    Questions for pupils to consider1. What is the extent of your experience of playing digital games (for recreation or fun)?2. What is the extent of your experience of playing digital games with a view to learning about

    something?3. What was your overall experience of using the Thinking Worlds gaming environment (ease of

    use, error-free, speed of response)?4. Ease of navigating through the game environment (both shopping mall, health centre)?5. Ease of initiating / completing interactions with game characters?6. Which of the following list of behaviours (interactions) did you experience?(Explain what they are

    to pupils): MCQ / Classify / Comprehend / Label /Contrast / Replace True/False; and Locateitems

    7. How easy was it to understand and respond to these behaviours?8. Did you find consistency of experience within each behaviour type (e.g. within Multiple Choice

    Question)?9. Did the environment represent a realistic interpretation of the sector you were studying?

    10. Would you be confident that this games-based approach provides a fair assessment of yourknowledge and skills in the subject?

    11. Did the game environment help you understand the context of the questions you were beingasked? Or was it a distraction? Or did it not matter?

    Questions for teachers to consider1. To what extent have you been using online experiences (game-based or otherwise)to elp pupils

    learning in the subject?2. Identify the key factors that have to be managed in order to carry out a gamesbased assessment

    in your school (e.g. booking computer suites, getting approval to load software)?3. In the pilot did you observe any barriers to participation on the part of pupils (e.g. lack of basic IT

    skills, levels of comprehension of game behaviours, lack of other skills)4. In your view are there specific pupil groups for whom this approach would not be suitable or

    appropriate?5. In your view are the assessment outcomes generated through the game-play valid?6. How well did the game environment reflect the scenarios you have been developing as part of

    your teaching the subject?7. To what extent did the game environment reinforce the concepts being tested in the assessment?

    Or form a distraction from the concepts being tested?8. What impact did the use of games-based assessment have on pupil motivation (if any)?9. Did you notice any improvement in pupil performance when using the software versus a

    conventional approach?