designing electronic performance support systems to facilitate learning

INTRODUCTION

Virtually all cognate human behaviour is motivated by goal-seeking activity (van Merriënboer, 1997). Suchactivity is invariably mapped onto a sequence of tasksthat allows an individual (or group of people) to movefrom some initial state of existence to some othersought after (and hopefully, more desirable) state ofbeing.

Within an educational context, the ‘more desirable’state referred to above is likely to involve the acqui-sition of ‘richer’ knowledge states and/or an improvedskill set (within a given problem domain). Naturally, animportant aspect of instructional design is therefore,either (a) the explicit formulation of task sets that can achieve these richer states (and which students can use directly), or (b) the identi� cation of ‘learningscenarios’ that allow students to discover these tasksets for themselves.

Despite the skills and knowledge that they have, mostindividuals (or groups) often find themselves insituations where a given goal is to be achieved, but yet they are not equipped, either intellectually or skill-wise, to achieve its realization. It is, therefore,important to consider how the absent skills and/orknowledge can be acquired. Table 1 illustrates someof the possibilities and their potential limitations. This

paper is concerned with the eighth of the ten methodsthat are listed – in particular, the use of an electronicperformance support system (EPSS).

The use of electronic performance support systems to facilitate skill and knowledge acquisition (andperformance improvement) was pioneered by Gery inthe late 1980s and early 1990s (Gery, 1991). Work byMcGraw (1994), Stevens and Stevens (1995) and Wild(2000) has also contributed to the growing base oftechniques for employing this approach. Reeves andRaven (2002) have presented a useful review of someof the recent developments that have taken place in thisarea. Our own work in this � eld began in 1992 and hascontinued throughout the last decade (Banerji, 1995;Barker, 1995; Beacham, 1998; Flinders, 2000). Themost recent of our EPSS projects now forms the basisfor this present paper (Pearson, 2001).

In a previous paper (Barker and Hudson, 1998), wesuggested that a fundamental requirement of an EPSSfacility is that it should increase a user’s on-the-jobperformance within a given task domain. This can beachieved in two basic ways. First, through the pro-vision of ‘automated aids’; and, second, by providingvarious mechanisms to support ‘on-the-job’, ‘just-in-time’ training – which will enable users of any given

Designing Electronic Performance Support Systems toFacilitate LearningPaul van Schaik, Robert Pearson and Philip Barker, University of Teesside, UK

SUMMARY

An electronic performance support system (EPSS) is a computer-based environment that facilitatesskill and knowledge acquisition within a particular domain of study. As well as having pedagogicutility, an EPSS can also be used to augment and enhance an individual ’s performance beyond thoselimits that are set by innate natural ability. The authors’ on-going research programme in this area hasbeen investigating the potential utility of EPSS tools in a range of domains. This current paper discussesan application of this technique within the domain of ‘quantitative research methods’ as taught withina psychology degree course. As well as describing the design and implementation of the system, thispaper also outlines the results of an evaluative study of the prototype EPSS tool that has been created.

Innovations in Education and Teaching Internationa lISSN 1470-3297 print ISSN 1470-3300 online © 2002 Taylor & Francis Ltd

http://www.tandf.co.uk/journalsDOI: 10.1080/1355800021016104 3

289 IETI 39,4

system to ‘learn as they do’. Naturally, within aneducational context, the emphasis on teaching andlearning activities will change the underlying designemphasis that is employed. Therefore, we believe thatan EPSS facility that is designed to ful� l an educationalneed should accommodate the following three basicrequirements. First, it should act as a pedagogic agentto assess skill and knowledge requirements for a giventask sequence and ‘� ll in’ the gaps relating to a user’scapability. Second, it must function as a transfer agentto develop skills and knowledge using ‘on-demand’and/or just-in-time mechanisms. Third, it should act asan augmentation aid to improve human performance –over and above an individual’s (or a group’s) innatenatural ability.

Depending upon their scope and capability, supportaids that are used to improve human performance canbe classi� ed according to a six-tier taxonomy similarto that which is presented in Table 2.

The work described in this paper constitutes anexample of an integrated electronic performancesupport system for the facilitation of task execution

within the domain of ‘quantitative research methods’.Our motivation for wanting to develop an EPSS for this domain stems from the current lack of avail-ability of integrated systems for use in this area. Some approaches to the use of ‘stand-alone’ electronicteaching packages for speci� c techniques have beendescribed in the literature – see, for example, Morris(2001). However, there is a growing need for morepowerful integrated environments (Branford, 2001).Although online packages for supporting statisticaltasks do exist (such as MINITAB and SPSS), these arenot really regarded as integrated EPSS systems.

Although our system has been designed primarily foruse by students following psychology courses, thesystem can be used within other domains – bothundergraduate and postgraduate. For example, studentsfollowing courses in human–computer interaction(HCI) often have to design experiments to observehuman behaviour (such as reading rates, typing speeds,reaction times, etc.). Such students therefore need toknow how to design and conduct experiments. Theyalso need to understand which statistical tests that theyshould use in order to analyse their data and makeinferences based upon their observations. The EPSSenvironment described in this paper is designed to meetthis need.

As has been suggested earlier, the EPSS facility hasbeen designed to support task-driven scenarios similarto that depicted schematically in Figure 1. Becausecertain aspects of the system allow technology-mediated communication between colleagues withinand without a given organization (via the Internet andtelephone), the EPSS facility therefore also exhibitssome aspects of an organizational electronic perfor-mance support tool (see Table 2).

290 IETI 39,4

Table 1 Some sources of skills and knowledge

1 General Educational Experiences– school, college, lifelong learning

2 Books, Lectures, Courses, Videos– ‘slow-track’ techniques; can be become out-of-date

3 Coaching– both manual and automated (usually small groups)

4 Apprenticeship– effective but costly; can be time consuming

5 Trial and Error Investigations– slow and costly; often dangerous

6 Research– both ad hoc and structured approaches

7 Conversations with Others– both face-to-face and technology mediated

8 Computers and Electronic Performance Support Systems– ‘fast-track’ techniques; easily updated and delivered

9 Virtual Learning Environments– easy access to shared resources; facilitate groupwork

10 Virtual University Systems– global access to courses and training

Table 2 Types of performance support aid

Simple Performance Support Tools– telephone, OHP, notepad, pencil

Electronic Tools– PowerPoint Slide Projection System, voice notepad

Integrated Performance Support Tools– Microsoft Of� ce, packages such as My Yahoo!

Personal Support Tools– mobile phone, personal digital assistant, laptop computer

Organizational Support Tools– departmental intranets, databases, decision support

Global Performance Support Tools– the Internet and private global networks

Within Figure 1, reference is made to the use of ‘mentalmodels’ as a mechanism that individuals use for storingthe knowledge that they have of a particular domain.The importance of mental models within the generalcontext of problem solving has been discussed in moredetail elsewhere (Barker et al., 1998; Barker and vanSchaik, 1999; van Merriënboer, 1997).

In the remainder of this paper, we describe the imple-mentation of the prototype performance supportsystem that we have built. Details are also given of the evaluative study that we have conducted withrespect to undergraduate usage and opinions of thesystem.

SYSTEM DESIGN AND DEVELOPMENT

This section of the paper brie� y discusses the variousphases involved in the implementation of the EPSSfacility. The following topics are described and dis-cussed: the needs analysis, design considerations andthe development phase. Short descriptions are alsogiven of the various ‘building blocks’ that make up theprototype EPSS facility.

The needs analysis

In order to identify relevant content for the EPSS, aneeds analysis was conducted. This took the form of a survey of psychology students at the University of Teesside. The needs analysis was undertaken inorder to identify the most important topics (relevant to quantitative research methods and statistics) that

the EPSS should contain. It also served to determinewhether students would be likely to use an EPSS tosupport their studies. The results of the needs analysiswere also used to identify the nature of the materialsand learning aids that the students currently employ. It was found that the main statistical software used bythe students was the ‘Statistical Package for SocialScientists’ (SPSS). The EPSS facility therefore had tobe designed ‘around’ the use of SPSS.

Before the prototype EPSS could be designed and built, the hardware and software requirements of theproposed system had to be identi� ed. Based upon ourprevious studies of both stand-alone and distrib-uted EPSS tools, it was decided that in order to meetboth Banerji’s and Beacham’s design principles, anInternet-based system would be the most advantageous(Banerji, 1995; Beacham, 1998). The prototype systemwas therefore based upon the use of HTML and Java-Script technologies. In addition, because the EPSS hadto be easily updateable and be capable of incorporating‘personal’ performance support aids, it was decidedthat a ‘database-driven’ approach had to be used. Thiswas achieved through the use of a software technologyknown as Active Server Pages (ASP).

In order to address the individual needs of potentialusers, the information that was gathered during the needs analysis was used to identify a number of different functions that had to be supported by the tools that were to be embedded in the prototypeEPSS facility. These included: an ‘Advisor’ system, a‘Help’ facility and a ‘Personal Area’ – the latter beingused for each individual user’s ‘Notes’ and person-alized ‘Performance Aids’. The generic performancesupport facilities that were produced for the proto-type EPSS environment were based upon tools forperforming ‘statistical power analysis’.

Design and development

The database

In order that the EPSS could be database-driven (using ASP techniques in conjunction with MicrosoftAccess), an appropriate database had to be designedand created. This had to contain information relating to the Help and the Advisor systems. It also had to storedetails of the users themselves, their personal notes and linkages (to various items in the EPSS and othersources on the Internet) and usage records. An entityrelationship diagram (ERD) showing the requireddatabase tables and their relationships to each other ispresented in Figure 2.

Electronic Performance Support Systems 291

Figure 1 A student-centred EPSS based on task-drivenscenarios

An important aspect of the database system was its use of the ‘Structured Query Language’ (SQL) in orderto search for and retrieve information. As is discussedlater, SQL provides many useful facilities for theimplementation of data retrieval operations.

The help system

The aim of the Help facility was to allow users to obtaineasy access to any available information relating toquantitative research methods and statistical conceptsthat happened to be stored within the Help table of thedatabase. This could be achieved in either of two basicways. First, by allowing a user to submit a word or term(describing his/her help requirement) which could thenbe used as a search criterion to locate and return anyrelevant information contained in the Help table.Second, by displaying the whole contents of the Helptable (on the computer screen) and allowing users tosearch through it manually.

In order to improve the effectiveness of the � rst searchmethod listed above, the SQL ‘Like’ operator wasemployed (in conjunction with the ‘%’ parameter)within the SQL statements that were used. The ‘%’parameter was a useful search facility as it could beused to represent any possible string of characters.Using this method, searches were more likely to � nd (ifany is available) and return some relevant informationto the user.

When constructing the help system, the needs analysiswas used to highlight potential areas in which usersmight need support. The analysis results, therefore,were used to identify the content areas that the helpsystem should attempt to cover.

The advisor system

The aim of this tool was to guide users towardsinformation about procedures that they wanted to carryout or towards concepts that they wished to learn about.When the system guides users towards appropriateinformation, they are given a number of optionalchoices at each level of the hierarchical decision treethat is used to navigate through the EPSS contentpages. As users proceed through the navigation tree forany given topic, each of these options is explained;users can then choose either the option that representsthe data that is being examined or an option thatcorresponds to the procedure that they wish to carryout on their data. An example of a procedure is shownin Figure 3.

The major part of the content of the basic web pagesthat make up the Advisor system is ‘hard coded’.However, each of these pages is partially database-driven because any content that is used on a number of different pages (for example, ‘nominal’, ‘ordinal’and ‘interval’ descriptions) and the de� nitions of keystatistical terms (such as ‘related t-test’ and ‘correla-tion’) is always retrieved from the Help table in thedatabase.

Some of the pages within the system also offer links to other pages that contain further information to support the concepts and ideas being explained in the Advisor system. All of the bottom-level pageswithin this facility have links to further pages thatexplain the steps that have to be performed in order to carry out the relevant statistical procedure usingSPSS.

The idea of teaching users how to carry out tasks inSPSS was also explored. For example, in the case of the‘related t-test’ a simulated SPSS environment wasused. This environment enabled users to learn the stepsrequired in order to carry out this procedure by actuallyexecuting them within the EPSS environment. Thisfacility was implemented by displaying screen shotsof SPSS along with menu options and buttons thatcould be used as ‘hot-spots’ – which when clickedwould allow a user to proceed to the next step in thetask sequence being executed.

292 IETI 39,4

Figure 2 Entity-relationship diagram for the database

As was mentioned earlier, the needs analysis high-lighted areas where potential users might need support– and therefore suggested the content areas that the Advisor system should attempt to cover. Bearingthis in mind, the domain knowledge of the EPSS was organized into two broad areas that dealt withdescriptive statistics and inferential statistics, respec-tively. These areas could be displayed in the form oftree diagrams representing how users would be guidedtowards relevant information. Figure 4 shows the treediagram for the Descriptive Statistics unit and Figure5 presents the diagram for the Inferential Statisticssection. In each diagram, the rectangles represent webpages within the Advisor system and the arrows denotehypertext links. A link on each web page also allowsaccess to a map of the descriptive or the inferentialstatistics Advisor – depending upon which part of thesystem a user happens to be in. Each map includes linksto the relevant pages in the Advisor system.

The personal area and notes

The aim of the personal area and personal notes was toallow the incorporation of personal performance aidsand tools. In order to access their personal area andtheir personal ‘sticky notes’, users must log into thesystem. Once logged in, a user is able to add, edit andremove their notes from the host pages that they relateto. When within their individual personal areas, usersare also able to add links to personal performance aidsand tools on the Internet as well as edit and remove allof the ‘sticky notes’ that they have added to pages. Anexample of a personal area is shown in Figure 6.

In addition to the user’s notes and personal area, a‘usage record’ is also kept of all the EPSS pages thatindividual users view. A usage record starts as soon asa user has logged onto the system. The page-id of eachweb page that a user visits is then concatenated to theroot entry for the usage record. When a user leaves the


Figure 3 Advisor System – related t-test

294 IETI 39,4

Figure 4 Tree diagram for descriptive statistics

Figure 5 Tree diagram for inferential statistics

system (by logging off), the usage record is stored inthe Usage Record table of the database (see Figure 2).Some of the other data that is also stored in the usagerecord include – the user’s user-id, the date and thetime at which the user logs off. The usage recordsenable the system administrator to monitor and analyseEPSS usage. In order to provide the sticky notes facilityand the audit trail that is held within a usage record,every page of information in the EPSS system has tohave its own unique identi� er associated with it. Forthis purpose, within the current implementation of thesystem, we use the actual � le name in which a web pageis stored.

The performance aids

The performance aids that were developed wereintended to provide special purpose tools to executetasks that students would normally carry out ‘by hand’.Within the prototype EPSS, one of the generic skillareas that have been implemented involves ‘power

analysis table look-up’. The EPSS is able to cater fortwo possible situations. First, it could be used for‘within subjects tests’ (using the related t andWilcoxon’s T standard statistical tests). Second, it couldbe applied to ‘between subjects tests’ (based on the useof the unrelated t and the Mann-Whitney U statisticalprocedures). Each of these could be applied in both a‘prospective’ and a ‘retrospective’ way.

As different variables are required to carry outprospective and retrospective power analysis, twoseparate tools were designed. For each of these tools aweb page was constructed that allowed a user to chooseand submit the research design for a given experiment.This could be based on either an ‘independentmeasures’ approach or a ‘repeated measures’technique. In addition, the following information wasneeded: the directionality of the test (either one- or two-tailed), the statistical test to be used, (the related t,Wilcoxon’s T, unrelated t, or Mann-Whitney U) and theeffect size (d). For the prospective power analysis, thepower was also required and for the retrospective


Figure 6 Personal area

power analysis the sample size also had to be speci� ed.An example of specifying the parameters of aprospective power analysis using the performance aidsis presented in Figure 7.

In order to implement the performance aids for the power analysis tools, four ‘look-up’ tables were required. These were created dynamically (inJavaScript) using two-dimensional arrays. AppropriateJavaScript code was then developed in order to pop-ulate the arrays, display them on the user’s computerscreen and search through them in order to locate the required entries – carrying out interpolation if thiswas deemed necessary. Once relevant entries in theJavaScript tables have been located the correspondingvalues in the displayed tables can be highlighted on theuser’s screen. The results of a prospective poweranalysis are illustrated in Figure 8.

EVALUATION

Method

An evaluation of the prototype EPSS for quantitativeresearch methods was conducted in order to establishstudents’ performance with the system and their overallacceptance of it as a teaching and learning aid. Inaddition, students’ level of knowledge of concepts inquantitative research methods was also analysed.

Design and participants

The EPSS was evaluated as part of revision lab classes(class sizes of 15–20) at the start of the secondacademic year of an undergraduate psychologyprogramme. The evaluation employed a single sampleof 89 psychology students (75 females, 13 males andone not stated) who had taken two research modules in

296 IETI 39,4

Figure 7 Performance aids – prospective power analysis (1)

their first year of study; the quantitative researchmethods content of these modules was included in theEPSS. The students’ mean age was 24.6 with astandard deviation of 8.4. Students rated themselves aspossessing relatively little experience with regard toquantitative research methods (median = 2.5 with semi-interquartile range (siqr) = 0.9 on a 5-point Likert scale)and SPSS (median = 2 with siqr = 0.9 on a 5-pointLikert scale). They rated themselves as moderatelyexperienced in the use of computers (median = 3 withsiqr = 0.5 on a 5-point Likert scale).

Materials and apparatus

The EPSS was mounted on an intranet server. Aquestionnaire was used as an evaluation instrument and was divided into four sections (see Appendix). Thefirst section measured students’ prior knowledge of concepts in quantitative research methods; itemployed 20 multiple-choice questions. The answersheet required respondents to write down as their

answer: (a) one of four answer alternatives presentedfor each question and (b) their level of confidence(percentage) in the answer given. Section 2 measuredthe demographic details of respondents. The thirdsection of the questionnaire measured the students’ task performance when using the EPSS – through the answers that they gave in response to tasks theywere asked to conduct. Finally, section 4 was used to measure the respondents’ acceptance of the system,including the key concepts of perceived usefulness andintention to use – as identified in technology accep-tance research (Davis and Venkatesh, 1996). Five-point Likert scale items were used in sections 2 and 4of the questionnaire.

Procedure

Students � rst � lled in section 1 of the Questionnaire.Next, they were instructed to use and explore the EPSSfor 10 minutes. Following this they were asked toanswer the questions in sections 2, 3 and 4 and use theEPSS as needed when completing section 3.


Figure 8 Performance aids – prospective power analysis (2)

Results

Knowledge of concepts in quantitative researchmethods

Over the 20 knowledge questions, the mean percentageof correct answers was 53.1 with a standard deviationof 12.2. The mean con� dence in answers was 48.2%with a standard deviation of 18.3. The percentages ofcorrect answers for individual questions are presentedin Table 3.

Questions relating to the following concepts had fewerthan 40% of correct answers: variability , bi-variatecorrelation , errors in statistical testing, degrees offreedom, linear regression and binomial distribution;questions related to other concepts had more than 50% correct answers. Over all the 20 questions, thepercentage of correct answers had a significantcorrelation with con� dence, r = 0.25, p< 0.05.

Use of the EPSS

The percentages of correct answers for the four ques-tions requiring the use of the Advisor system showedgood performance on three questions (see Table 4).Advisor Question 4 suffered from poorer performance

because many respondents answered the question interms of difference rather than relationship as stated inthe question. Percentages of correct answers for twoquestions requiring the use of Tools showed goodperformance (see Table 4). Furthermore, performancewas good on the questions requiring the use of the Helpfacility – showing high percentages for successfully� nding concepts within Help (see Table 4). An overallscore for EPSS performance was calculated as the totalnumber of correctly answered questions and itemsfound in the Help system. The mean percentage scorewas 84.6 with a standard deviation of 13.9. Pearson’scorrelation between EPSS success and percentage ofcorrectly answered knowledge questions was non-signi� cant (r = 0.067, p >> 0.05). An unrelated t testshowed that those scoring below the median and thosescoring equal to or higher than the median on theknowledge questions did not signi� cantly differ in theiroverall EPSS performance (t(71) = –0.366, p >> 0.05)

System acceptance

The usefulness of the EPSS for learning, revision and completing assignments each had a median scoreof 5 (with siqr 0.25, 0.25 and 0.5 respectively). Theperceived likelihood of using the EPSS, if it was

298 IETI 39,4

Table 3 Percentages of answers to knowledge questions

Q Concept Percentage correct Con� dence – mean (sd)

1 Central tendency 99 78 (25)2 Variability 21 48 (23)3 Normal distribution 65 49 (25)4 Bi-variate correlation 32 34 (21)5 Signi� cance testing with two groups 61 53 (27)6 Descriptive statistics 63 47 (26)7 Parametric test for two unrelated samples 57 45 (28)8 Level of measurement 63 43 (28)9 Validity 81 60 (27)

10 Correlation 74 45 (31)11 Sampling 76 67 (24)12 Causation 78 58 (25)13 Errors in statistical testing 17 43 (28)14 Degrees of freedom 39 37 (25)15 Parametric test for two related samples 62 41 (25)16 Design when using two related samples 84 56 (29)17 Linear regression 2 37 (26)18 Graphical representation of interval level data 83 48 (29)19 Binomial distribution 18 38 (26)20 Homogeneity of variance 52 37 (29)

Notes: Q = question; sd = standard deviation

made available through the university’s intranet, had a median score of 5 with siqr 0.25. A Wilcoxon’spaired signed-rank test showed that median scores onall four measures were significantly higher than aneutral score (represented by a value of 3 on a five-point Likert scale), all z < –5.9, p < 0.001. UsingKendall’s tau, it was established that all four system-acceptance questions were signi� cantly correlated (seeTable 5).

Discussion

Overall, knowledge levels indicated that there wasscope for improving students’ performance. Speci� careas of poorer performance were variability , bi-variate correlation, errors in statistical testing,degrees of freedom, linear regression and binomialdistribution. Further development of the EPSS couldaddress these areas in more detail within the Advisorsystem and Help facility.

Overall, students’ performance using the EPSS washigh (mean = 84.6% correct). This indicates that the

EPSS was successful in supporting students whenperforming tasks related to quantitative researchmethods. Although outcome measures of knowledgeand EPSS use were not designed for comparison,overall, students’ performance using the EPSS was higher than unaided performance on knowledgequestions (mean = 53.1% correct). Comparing stu-dents’ level of knowledge and their success on taskswith the EPSS more directly, it was found that in thearea of parametric test for independent measuresdesign (knowledge question 7 and Advisor question 2)students’ performance improved by 33%. These resultssuggest that the EPSS can be used as a means toimprove students’ performance.

There was a non-significant correlation betweenstudents’ performance in terms of (a) knowledge ofconcepts and (b) success in using the EPSS. Thiscorrelation together with the non-signi� cant differencein aided performance between the better and poorerstudents in terms of unaided performance indicates thatstudents with different levels of knowledge benefitequally from using the EPSS for revision.


Table 4 Percentages of answers to questions requiring use of the EPSS

EPSS Sub-system Question topic Percentage correct or found

Advisor 1 Measure of central tendency for parametric data 95

2 Parametric test of independent measures design 76

3 SPSS menu selections for parametric test of independent measures design 77

4 Test for relationship in ordinal data 47

Tools 1 Prospective power analysis 94

2 Retrospective power analysis 90

Help 1 Mode 100

2 Interpolation 100

3 Effect size 97

4 t test 79

Table 5 Correlations between perceived usefulness and intention to use

Learning Assignments Revision

Assignments 0.63

Revision 0.37 0.46

Likelihood of using 0.58 0.63 0.55

Note: p < 0.01 for all correlation s

The correlation between performance and con� dencewhen using the EPSS was not examined in this study.However, the significant correlation between confi-dence and performance on the knowledge questionsindicates that use of the EPSS may not only result inincreased student performance but might also producean increase in con� dence as well.

Regarding perceived usefulness, students expected that the EPSS would be useful to them for learning,revision and completing assignments. Relating tointention to use, students believed that it was likely thatthey would use the system if it was made available tothem through the university’s intranet. The signi� cantcorrelations between acceptance measures con� rmedthe association between intention to use and perceivedusefulness found in previous research (Davis andVenkatesh, 1996; van Schaik et al., 2002).

CONCLUSIONS AND FUTURE WORK

The EPSS that we have developed has extended therange of electronically supported domains to the area of quantitative research methods using a web-based implementation. This implementation makes itpossible to support a global user community of learnersand researchers with regularly updated support as issuggested in Figure 1. The global scope of our systemis in line with developments towards global educa-tional and training systems such as virtual universities(Hazemi and Hailes, 2001).

The evaluation results of the EPSS in the context ofrevision are encouraging. However, the value of theEPSS should also be assessed in other situations suchas learning and completing assignments.

The scope of the EPSS can be extended in various ways in order to make it more widely applicable. Inparticular, the range of the quantitative researchmethods included in the system could be increased.Furthermore, the scope of the domain could also beextended in order to make it more general. This couldbe achieved by supporting the process of research from‘start to finish’ rather than only research methods.Although various textbooks describing or prescribingthe overall research process are available – see, forexample, Robson (1993) – online support in the formof an EPSS is still lacking.

The functionality of the EPSS can be extended in order to support a wider range of tasks within a learningor research context. For example, support for commu-

nication between users could be provided. This couldbe implemented by linking with existing tools forelectronic communication. Furthermore, the set of toolsprovided by the EPSS and the number of links to other(online and off-line) resources could be expanded.

REFERENCES

Banerji, A K (1995) Designing Electronic PerformanceSupport Systems, PhD Thesis, University of Teesside,Middlesbrough, UK.

Barker, P G (1995) Electronic Performance SupportSystems, Special Edition of Innovations in Educationand Training International, 32, 1, 1–73.

Barker, P G and Hudson, S (1998) An Evolving Model ForMultimedia Performance Support Systems. In vanOostendorp, H and Arnold, A G (eds) Trends inCommunication: Part 4 Interface Technology –Enhancing the Quality of Life, Boom Publishers, TheNetherlands, pp. 60–73. Available online:ht tp:/ / www-scm.tees.ac.uk/ users/phi l ip.barker/maastricht/mmeps.doc

Barker, P G and van Schaik, P (1999) Mental Models andtheir Implications for the Design of Computer-BasedLearning Resources. In Chapman, G M (ed.) Computer-Based Learning in Science, Proceedings of CBLIS 99 International Conference, University of Twente,Enschede, The Netherlands, University of Ostrava Press,Ostrava, Czech Republic, pp. 1–11.

Barker, P G, van Schaik, P and Hudson, S (1998) Mentalmodels and lifelong learning, Innovations in Educationand Training International, 35, 4, 310–18.

Beacham, N A (1998) Distributed Performance SupportSystems, PhD Thesis, University of Teesside,Middlesbrough.

Branford, A J (2001) Statistics Computing Laboratory,Paper C6. In Chapman, G M (ed.) Proceedings ofComputer-based Learning in Science, CBLIS 2001,University of Ostrava Press, Czech Republic.

Brown, L A (1996) Designing and Developing ElectronicPerformance Support Systems, Digital Press, Newton,MA.

Davis, F D and Venkatesh, V (1996) A critical assessmentof potential measurement biases in the technologyacceptance model: three experiments, InternationalJournal of Human-Computer Studies, 45, 1, 19–45.

Flinders, S (2000) A Prototype Electronic PerformanceSupport System Facility to Support Web-based Infor-mation Retrieval, MSc Thesis, School of Computing andMathematics, University of Teesside, Middlesbrough.

Gery, G J (1991) Electronic Performance Support Systems– How and Why to Remake the Workplace Through theStrategic Application of Technology, ZiffCommunications Company, Cambridge, MA.

Hazemi, R and Hailes, S (2001) The Digital University –Building a Learning Community, Springer-Verlag,Heidelberg.

300 IETI 39,4

McGraw, K L (1994) Electronic performance supportsystems: integrating AI, hypermedia and CBT to enhanceuser performance, Journal of Arti� cial Intelligence inEducation, 5, 1, 3–26.

Morris, E (2001) The design and evaluation of link: acomputer-based learning system for correlation, BritishJournal of Educational Technology, 32, 1, 39–52.

Pearson, R (2001) An EPSS for Quantitative ResearchMethods and Statistics, MSc Thesis, School ofComputing and Mathematics, University of Teesside,Middlesbrough.

Reeves, T C and Raven, A (2002) Performance SupportSystems, in Adelsberger, H H, Collis, B and Pawlowski,J M (eds) Handbook on Information Technologies forEducation and Training, Springer-Verlag, Berlin, pp.93–112.

Robson, C (1993) Real World Research, Blackwell,Oxford.

Stevens, G F and Stevens, E (1995) Designing ElectronicPerformance Support Tools – Improving WorkplacePerformance with Hypertext, Hypermedia andMultimedia, Educational Technology Publications,Englewood Cliffs, NJ.

van Merriënboer, J J G (1997) Training Complex CognitiveSkills: a Four-Component Model for Technical Training,Educational Technology Publications, Englewood Cliffs,NJ.

van Schaik, P, Bettany-Saltikov, J A and Warren, J G(2002) clinical acceptance of a low-cost portable systemfor postural assessment, Behaviour and InformationTechnology, 21, 47–57.

Wild, M (2000) Designing and evaluating an educationalperformance support system, British Journal ofEducational Technology, 31, 1, 5–20.

BIOGRAPHICAL NOTES

Dr. Paul van Schaik is a senior lecturer in Psychologyin the School of Social Sciences at the University of Teesside, Middlesbrough. His main researchinterests include human–computer interaction andvirtual reality.

Robert Pearson has recently been appointed as alecturer in Psychology in the School of Social Sciencesat the University of Teesside. He obtained a � rst classhonours degree in Psychology in 2000 and wasawarded a masters degree in Information Technologywith a distinction in 2001. His research interestsinclude human-computer interaction.

Philip Barker is Professor of Applied Computing in the School of Computing and Mathematics at theUniversity of Teesside. He is an Associate Editor ofthe International Journal of Human–Computer Studies.

His research interests include electronic learning andhuman–computer interaction. He was the ProgrammeChairman for the World Conference on EducationalMultimedia, Hypermedia and Educational Telecom-munications (EDMEDIA 2002), which was held inDenver, Colorado, USA in June 2002.

Addresses for correspondence: Dr Paul van Schaik &Robert Pearson, Psychology Section, School of SocialSciences, University of Teesside, Borough Road,Middlesbrough, TS1 3BA, UK; e-mail: [email protected] and [email protected]. Professor Philip Barker, School of Computingand Mathematics, University of Teesside, BoroughRoad, Middlesbrough, TS1 3BA, UK.e-mail: [email protected]


APPENDIX – EPSS EVALUATION QUESTIONNAIRE

EPSS Evaluation

The objective of an electronic performance support system (EPSS) is to provide a set of electronic tools in orderto improve human performance within a given application domain. An EPSS for quantitative research methodsand SPSS has been developed which you will be able to use during this module. A number of different functionsare supported by the tools that are embedded in the EPSS. These included: an ‘Advisor’ system to guide usersto speci� c information on quantitative research methods, a ‘Help’ facility to allow easy access to informationstored in the EPSS and a ‘Personal Area’ which allows each individual user to have ‘Notes’ and personalized‘Performance Aids’. Tools for performing ‘statistical power analysis’ are also incorporated into the EPSS.

This questionnaire aims to obtain your view of the EPSS before actually using it.

1. Gender (please tick one): Male h Female h

2 Age (in years): _________

3. Programme of study and year (for example 1st year BSc Psychology):

4. Using computers

302 IETI 39,4

I will never understand how to use a computer.

Strongly Strongly agree disagree

+3 +2 +1 0 –1 –2 –3

Only a few experts really understand how computers work.


+3 +2 +1 0 –1 –2 –3

It is extremely dif� cult to learn how to use a computer.


+3 +2 +1 0 –1 –2 –3

Computer errors are very dif� cult to � x.


+3 +2 +1 0 –1 –2 –3

5. Doing quantitative research methods

6. Using SPSS

7. Intention to use the EPSS


I will never understand how to do quantitative research methods.


+3 +2 +1 0 –1 –2 –3

Only a few experts really understand how quantitative research methods work.


+3 +2 +1 0 –1 –2 –3

It is extremely dif� cult to learn how to do quantitative research methods.


+3 +2 +1 0 –1 –2 –3

I will never understand how to use SPSS.


+3 +2 +1 0 –1 –2 –3

Only a few experts really understand how SPSS works.


+3 +2 +1 0 –1 –2 –3

It is extremely dif� cult to learn how to use SPSS.


+3 +2 +1 0 –1 –2 –3

Assuming I had access to the EPSS, I intend to use it


+3 +2 +1 0 –1 –2 –3

Please state why.

……………………………………………….….………………………………………………………………………………………..…………………………………………………………………………………………..……………………………………………….….………………………………………………………………

8. Usefulness of the EPSS

(a) For learning

Please state why.

……………………………………………….….………………………………………………………………………………………..…………………………………………………………………………………………..……………………………………………….….………………………………………………………………

304 IETI 39,4

Given that I had access to the EPSS, I predict that I would use it


+3 +2 +1 0 –1 –2 –3

Using the EPSS would improve my performance in learning quantitative research methods.


+3 +2 +1 0 –1 –2 –3

Using the EPSS would make it easier for me to learn quantitative research methods.


+3 +2 +1 0 –1 –2 –3

Using the EPSS would enhance my effectiveness in learning quantitative research methods.


+3 +2 +1 0 –1 –2 –3

I would � nd the EPSS useful in my learning of quantitative research methods.


+3 +2 +1 0 –1 –2 –3

(b) For completing assignments

Please state why.

……………………………………………….….………………………………………………………………………………………..…………………………………………………………………………………………..……………………………………………….….………………………………………………………………

(c) For revision


Using the EPSS would improve my performance in completing assignments.


+3 +2 +1 0 –1 –2 –3

Using the EPSS would make it easier for me to complete assignments.


+3 +2 +1 0 –1 –2 –3

Using the EPSS would enhance my effectiveness in completing assignments.


+3 +2 +1 0 –1 –2 –3

I would � nd the EPSS useful in the completing of my assignments.


+3 +2 +1 0 –1 –2 –3

Using the EPSS would improve my performance in revision of quantitative research methods.


+3 +2 +1 0 –1 –2 –3

Using the EPSS would make it easier for me to revise quantitative research methods.


+3 +2 +1 0 –1 –2 –3

Please state why.

……………………………………………….….………………………………………………………………

9. General comments and problems.

……………………………………………….….………………………………………………………………

Thank you again for your help with the evaluation!Please check that you have answered all the questions.

306 IETI 39,4

Using the EPSS would enhance my effectiveness in revising quantitative research methods.


+3 +2 +1 0 –1 –2 –3

I would � nd the EPSS useful in my revision of quantitative research methods.


+3 +2 +1 0 –1 –2 –3

designing electronic performance support systems to facilitate learning

Documents