The University of Birmingham Exam PaperDatabase: an analysis of user accesses

and feedback from 19 December 1998 to 25June 1999

http://builder.bham.ac.uk

Pete Dalton and Clare NankivellCentre for Information Research and Training

University of Central England

8 November 1999

2

Contents

Contents........................................................................................................................ 2Executive summary ........................................................................................................ 31. Introduction ........................................................................................................... 62. Exam paper database usage: an analysis of access log data ........................................... 8

2.1 Introduction........................................................................................................ 82.1.1 Location of PCs with highest accesses to the exam paper database ...................... 82.1.2 Periods of highest usage .............................................................................. 102.1.3 Exam papers most frequently accessed.......................................................... 142.1.4 Successful versus unsuccessful downloads ..................................................... 16

3. Analysis of customer feedback from on-screen questionnaire ...................................... 173.1 Introduction...................................................................................................... 173.2 Results ............................................................................................................. 17

3.2.1 Number of responses – sample size .............................................................. 173.2.2 Frequency of use....................................................................................... 183.2.3 Finding out about the exam paper database ................................................... 193.2.4 Print versus electronic exam paper resources ................................................. 203.2.5 Print versus electronic exam paper resources, reasons for preference ................ 213.2.6 Evaluation of specific features of the Exam Paper database.............................. 243.2.7 Access to database ..................................................................................... 283.2.8 Other comments ......................................................................................... 29

4. Conclusions .......................................................................................................... 315. Recommendations ................................................................................................. 33

List of FiguresFigure 1: Hourly patterns of access between 19 December 1998 – 25 June 1999Figure 2: Proportion of successful and unsuccessful downloadsFigure 3: Frequency of useFigure 4: Finding out about the databaseFigure 5: Use of printed and electronic exam paper resourcesFigure 6: Printed versus electronic sources – preferencesFigure 7: Ease of useFigure 8: Time taken to find exam papersFigure 9: Screen designFigure 10: Access off campus

List of TablesTable 1: Location of 20 PCs with highest accesses to the exam paper databaseTable 2: Dates of the 20 busiest periods of accessTable 3: Hourly periods of accessTable 4: 20 most accessed exam papers

Appendix: On-screen questionnaire survey

3

Executive summary

This report details the findings of an evaluation of the pilot online University of Birminghamexam paper database. The database was set up in October 1998 as a test service by theBUILDER project team and involved digitisation of printed exam papers and presenting them asAdobe PDF files. The exam paper database consisted of 1600 exam papers from one year only(1997-8). Exam papers for most subject areas were available and were accessible via the Webon the University of Birmingham campus network only.

Evaluation was conducted both through analysis of the Microsoft Internet Information Server(IIS) logs of usage of the service over six months, which covered 42,135 accesses, and throughanalysis of an on-screen questionnaire completed by 254 individuals.

Overall, despite criticism aimed at specific aspects of the service, the evaluation providedpositive feedback concerning the exam paper database service and the indications are that it hasa great deal of potential in the future to support learning at the University of Birmingham.

Key findings

During the initial test phase, indications are that the exam paper database was frequently usedby students to support them with their study. As might be expected, the periods of most usageoccurred in the run-up to the end of term examination session in the second semester. Thisincluded dates close to the actual exam period and dates prior to the Easter vacation, whenstudents obtained papers for revision during the vacation. Although one period of heavy usagewas recorded prior to the first semester exam period, usage corresponding to this period wasless heavy than that corresponding to the second semester exam period. This may be due tomore people finding out about the exam paper database towards the second semester examperiod.

Times of peak usage were during the day concentrated on the hours between 10am and 5pm.The access statistics suggest that most users were able to download papers successfully. In theperiod examined, exam papers supporting science subjects were those most accessed, with thegreatest number of single accesses being made to papers supporting courses in the Departmentof Physics and Astronomy. Access data suggests that the service is being frequently used bystudent customers. The dedicated PC in the Main Library and other PCs in library clusters aswell as PC clusters located in specific departments and Schools were most often used to accessthe service.

The research suggests that most users were comfortable with various design features of thedatabase, such as ease of use, time taken to locate specific papers and screen design. Themajority of users rated these features as good or okay. Many respondents had been made awareof the service through a tutor and friend. In some cases, it seemed that they were not briefedsufficiently about what the service could or could not currently provide.

At present, just over half of the users who had experience of both the print and electronic exampaper resources expressed a preference for the electronic resource. Reasons for favouring theelectronic database focused on the following:

4

• Reliability that a specific exam paper from the year archived was available - i.e. not lost ormislaid

• Enabling simultaneous access for many users – considered to be particularly useful aroundbusy exam periods

• Printing – many users considered it easier to print a paper than to photocopy one from theprinted resource

• Speed of searching – some users considered that it was quicker and easier to locate specificpapers using the electronic resource.

Those that either preferred the printed exam paper resource, or provided criticisms of theelectronic resource, provided comments focusing on the following issues:

• Coverage: the exam paper database presently only provides access to one year of exampapers

• Access to appropriate hardware / software: some users had experienced difficulties gettingaccess to a limited number of computers with the appropriate software installed

• Printing facilities: the value of the exam paper database was considerably reduced whenusers did not have access to appropriate printing facilities

• Browsing: some respondents considered that the paper resource was easier to browse thanthe electronic resource.

In addition, many respondents expressed a desire for the exam paper database to be madeaccessible outside the University of Birmingham campus network so that access could be gainedfrom home during vacation periods. The indications were that, if this were made possible, itwould become a common way of accessing the database. The suggestion was that the preferencefor using the electronic database would be greatly increased if some or all of the featuresmentioned above were integrated into the service.

Recommendations

A number of recommendations for the BUILDER project team and the University ofBirmingham emerged from the research. These were:

• Provision of more information on the database explaining what it is and is not capable of,and ensuring that tutors and IS staff are suitably briefed

• Continuing to promote and publicise the service to potential users. Information provided bytutors and IS staff is an effective way to publicise the service.

5

• Provision of a more extensive online exam paper database – containing more than one yearof papers

• Putting in place the appropriate mechanisms for facilitating off-campus access to thedatabase for University of Birmingham students e.g. authenticated access via username andpassword

• Investigating the scale of provision in departments and information services of computerhardware and software to allow access to the database. This includes ensuring a sufficientnumber of PCs with Adobe Acrobat Reader are available

• Ensuring that users will be able to get access to printing facilities as this greatly enhances thevalue of the database.

6

1. Introduction

This report analyses usage of the pilot online University of Birmingham exam paper database,and also assesses feedback received from an online questionnaire presented to users of theservice for a fixed time period.

The University of Birmingham exam paper database was set up in October 1998 as a testservice by the BUILDER project team in co-operation with the University of BirminghamExaminations Office. It involved digitisation of printed exam papers and presenting them asAdobe PDF (Portable Document Format) files.

The database provides access to past exam papers via the World Wide Web athttp://builder.bham.ac.uk/exampapers. At the time this research was conducted, it providedaccess to over 1600 papers in a variety of subjects, although papers from the School ofMedicine were not included. During the period examined (19 December 1998 – 25 June 1999)the database provided access to papers from the 1997/98 session only.

Access was available to users on the University of Birmingham campus only using the Web.Papers could not be viewed off-campus. In order to view exam papers, users needed to use aPC with Adobe Acrobat Reader (at least version 2). Users had the option to read the paper onscreen or, if their PC was connected to a laser or ink-jet printer, they could print the paper out.

Copies of printed exam papers, containing many years of retrospective papers, were available inthe University of Birmingham Main Library. Photocopiers were situated close to this area. Adedicated PC with laser printer was available within the Main Library to facilitate access to, andprinting out, the online exam papers. The printed exam papers are also available on short loanfor photocopying from the relevant site library at the University of Birmingham.

Before the launch and during the period examined, the service was publicised by the BUILDERteam in a variety of ways including posters, fliers, links from the Information Services Web siteetc.

The exam paper database aimed to support teaching and learning at the University. TheUniversity operates a semester system with vacation periods in between and exam periods at theend of each semester. The dates of the 1998/99 semesters, vacation periods and exam periodsare as follows:

Semester 1: 28 September 1998 – 22 January 1999Christmas vacation: 19 December 1998 – 17 January 1999Semester 1 exam period: 18 January 1999 – 23 January 1999

7

Semester 2: 25 January 1999 – 28 June 1999Easter vacation: 27 March 1999 – 25 April 1999Semester 2 exam period: 17 May 1999 – 5 June 1999

Autumn term: 28 September 1998 – 18 December 1998Spring term: 18 January 1999 – 26 March 1999Summer term 26 April 1999 – 18 June 1999

The creation and use of the exam paper database highlights some of the issues concerning thehybrid library such as electronic delivery of information, user interface design and usage of theservice. As a result research was undertaken to analyse the pilot exam paper database service.Logs of usage for the period of 19 December 1998 to 25 June 1999 were analysed. In additionto this, the responses to a pop-up questionnaire presented to users of the service between 26April 1999 and 23 June 1999 were also analysed. This report details the results of thisresearch.

8

2. Exam paper database usage: an analysis of access log data

2.1 Introduction

Over the period of 19 December 1998 to 25 June 1999 a complete log of accesses to the exampapers was recorded, using standard Microsoft Internet Information Server (IIS) logging. Thefollowing is an analysis of this data. A total of 42,135 accesses were made to the databaseduring the period examined.

2.1.1 Location of PCs with highest accesses to the exam paper database

The IP address provides data to determine which individual PCs have been used to access theexam paper database on the University of Birmingham campus. Table 1 overleaf shows thelocation of the 20 PCs most frequently used to access the database from.

The data suggests the following points of significance:

• The IP address which has the greatest number of accesses (2017) is a dedicated PC in themain hall of the Main Library. This PC and laser printer was provided solely for the purposeof providing access to the exam paper database. The access data suggests that this has beena worthwhile investment.

• Five of the 20 IP addresses with greatest accesses are from PCs located in the Main Library,which may suggest that the exam paper database is being accessed alongside other servicesin the library.

• The PC with the second highest number of accesses (1604) is located in the School ofPhysics and Astronomy. It is interesting to note that the School courses web page contains adirect link to the exam paper database. This may be a factor contributing to the highnumber of accesses from this PC.

• The School of Computing was the most represented School in Table 1. PCs in the School ofComputing accounted for four of the 20 IP addresses with the greatest number of accesses.

• Three of the IP addresses represented are from PC clusters located in halls of residence.These PCs are considered to be on-campus and are part of the University of Birminghamcampus network.

9

Table 1. Location of 20 PCs with highest accesses to the exam paper database

Ranking IP address Number ofAccesses

Location

1 147.188.225.98 2017 Main Library [Dedicated PC for accessingexam paper database in main hall]

2 147.188.38.18 1604 Room P8, Poynting Buiding, School ofPhysics and Astronomy

3 147.188.195.13 1424 School of Computer Science PC Cluster4 147.188.213.120 810 Aston Webb Building Public PC Cluster5 147.188.192.41 528 School of Computer Science PC Cluster6 147.188.224.54 500 B18, Main Library PC Cluster7 147.188.225.65 444 Main Library8 147.188.236.18 362 Muirhead Tower Public PC Cluster9 147.188.195.12 319 School of Computer Science PC Cluster10 147.188.195.11 292 School of Computer Science PC Cluster11 147.188.55.7 272 University House Hall of Residence PC

Cluster12 147.188.225.62 259 Main Library13 147.188.26.5 245 School of Law PC Cluster14 147.188.236.24 231 Muirhead Tower Public PC Cluster15 147.188.236.22 227 Muirhead Tower Public PC Cluster16= 147.188.236.19 223 Muirhead Tower Public PC Cluster16= 147.188.55.9 223 University House Hall of Residence PC

Cluster18 147.188.225.108 217 Main Library19 147.188.213.101 203 Aston Webb Building Public PC Cluster20 147.188.55.6 199 University House Hall of Residence PC

Cluster

10

2.1.2 Periods of highest usage

The following section individually details the dates and times of greatest access to the database.

Dates of most usageTable 2 below shows the dates of the 20 busiest periods of access to the database, showing thatthe busiest day saw 1578 accesses, over two per minute.

Table 2: Dates of the 20 busiest periods of access

Ranking Date Number of accesses

1 26/3/99 15782 10/5/99 10013 24/3/99 9384 20/3/99 8765 25/3/99 8646 17/5/99 8537 11/5/99 8528 6/5/99 8019 23/3/99 77910 12/5/99 75511 27/4/99 74112= 4/5/99 74012= 13/5/99 74014 7/5/99 72415 14/5/99 67816 5/5/99 64917 28/4/99 61418 26/4/99 59619 15/1/99 59220 22/2/99 574

11

The periods of highest usage correspond significantly to exam periods and vacation periods (seesection 1 for precise dates).

The date of greatest usage is 26 March 1999. This date is the last day of the Spring term. Inaddition, there are four other dates around this period represented amongst the 20 periods ofhighest usage. This suggests that students use the exam paper database to obtain copies of exampapers for revision purposes during the Easter vacation. Shortly after the Easter vacation,beginning 17 May 1999, is the second semester exam period. It is probable that if access wereavailable remotely, that significant usage would occur during the period of the Easter vacation,as students would obtain exam papers from the database for revision from home.

Another significant period of high access occurs in the first half of May 1999 whichcorresponds to the second semester exam period. Accesses in this period account for half of the20 dates of greatest usage.

Only one period of access prior to the first semester exam period is represented in Table 2. It islikely that usage corresponding to the second semester exam period is represented more thanthat of the first semester as more time had passed since the launch of the service and morepeople had become aware of the service.

12

Times of most usageTable 3 below shows the patterns of access broken down into hours for the period examined.As Table 3 shows, the period of most access centres around 11am - 5pm, with 1pm - 2pmbeing the hour of most accesses to the database. The periods between 12am and 8am are thoseof least usage.

Table 3. Hourly periods of access

Ranking Hourly period Number of accesses1 1-2pm 54692 12-1pm 51273 3-4pm 49854 11-12pm 48555 2-3pm 47226 4-5pm 42127 10-11am 33678 5-6pm 21449 9-10am 188110 6-7pm 128911 7-8pm 86612 10-11pm 77113 8-9pm 70114 11-12am 56215 9-10pm 49116 8-9am 35117 12-1am 17018 1-2am 6819 7-8am 2620= 5-6am 1920= 6-7am 1922 2-3am 1823 3-4am 1324 4-5am 9

Figure 1 overleaf presents these patterns of usage in the form of a line graph.

13

F ig u r e 1 . H o u r ly p a t t e r n s o f a c c e s s b e t w e e n 1 9 /1 2 /9 8 - 2 5 /0 6 /9 9

0

1 0 0 0

2 0 0 0

3 0 0 0

4 0 0 0

5 0 0 0

6 0 0 012

-1am

1-2am

2-3am

3-4am

4-5am

5-6am

6-7am

7-8am

8-9am

9-10a

m10

-11am

11-12

pm12

-1pm

1-2pm

2-3pm

3-4pm

4-5pm

5-6pm

6-7pm

7-8pm

8-9pm

9-10p

m10

-11pm

11-12

am

H o u r ly p e r io d

Num

ber o

f acc

esse

s

14

2.1.3 Exam papers most frequently accessed

Table 4 shows the 20 exam papers that were accessed most frequently during the periodexamined.

Table 4: 20 most accessed exam papers

Rank Code Title of exam paper Department Accesses1 U1751 Astrophysics/Physics

(theory)/Physics(electronics)/physics option 1

Physics and SpaceResearch

346

2 U1745 Electromagnetism Physics and SpaceResearch

341

3 U0925 Globalisation & Territory Geography 281

4 U0926 Global environmental process Geography 273

5 U1735 Basic physics Physics and SpaceResearch

227

6 U1748 Nuclear and condensed matter Physics and SpaceResearch

222

7 U0179 Metabolism & Proteins Biochemistry 218

8 U2410 Logic Computer Science 206

9 U0678 Signal systems and control Electronic andElectrical Engineering

203

10 U1747 Classical mechanics Physics and SpaceResearch

197

11 U0184 Medical Biochemistry Biochemistry 195

12 U0182 Metabolism and cellularregulation

Biochemistry 190

13 U0625 International banking Economics 188

14 U1574 Calculus and algebra Mathematics andStatistics

187

15 U1737 Physics 2 Physics and SpaceResearch

186

16 U0398 Introduction to Media, Culture &Society

Cultural Studies 182

17 U1736 Particle Physics & Cosmology Physics and SpaceResearch

177

18 U1483 Criminal Law Law 169

19 U0927 Global Issues Geography 170

20 U0188 Biochemistry Synoptic Module 1 Biochemistry 164

15

The exam paper which has been accessed the most number of times (346) is one from theDepartment of Physics and Space Research. As this particular paper corresponds to fourindividual courses within the department, it is perhaps not surprising that it has been accessed agreat number of times. However, the paper ranked second which was accessed nearly as manytimes (341) is also from the Department of Physics and Astronomy and supports only onecourse. Indeed, exam papers from the Department of Physics and Astronomy were those mostrepresented in Table 4, with seven of the 20 exam papers supporting courses from thisdepartment. This bears some correspondence to the analysis of the most used IP addresses inwhich Physics and Astronomy was represented as having the IP address with the secondgreatest number of accesses to the service (see section 2.1.1.).

Exam papers supporting courses in Biochemistry are represented four times in Table 4 and twopapers from the Department of Geography feature with the third and fourth highest number ofaccesses.

Overall, exam papers supporting science subjects are those featuring most in the 20 mostfrequently accessed papers.

16

2.1.4 Successful versus unsuccessful downloads

Figure 2 shows the number of successful and unsuccessful downloads. An unsuccessfuldownload results from either an inconsistency in the exam paper database or in non-delivery ofthe Adobe PDF exam paper due to networking problems. Out of the total of 42,135 accessesmade during the period examined, 37,038 were successful and 5097 were not. A successfuldownload was the result of nearly nine out of every ten accesses to the exam paper database.

Figure 2. Proportion of successful and unsuccesful downloads

88%

12%

SuccessfuldownloadsUnsuccessfuldownloads

17

3. Analysis of customer feedback from on-screen questionnaire

3.1 Introduction

This section of the report presents the findings of an online questionnaire presented to a sampleof the users of the exam paper database, in order to obtain additional feedback about theservice. Questions asked covered:

• frequency of use of the database

• comparative value of printed versus electronic delivery of the service

• ease of use and design.

Although most questions were closed, requiring respondents to choose from a range of options,two questions provided respondents with the opportunity to give open-ended answers to aparticular question or feedback on the service in general.

A complete list of questions contained in the questionnaire can be found in Appendix 1.

The survey took the form of a pop-up online questionnaire that was presented on every tenthaccess of the exam paper database. The figures for completion of questionnaires are, however,fewer than the total amount of times they were presented, as users receiving the questionnairehad the option of skipping it entirely.

This section of the report is structured to correspond with the questions presented in the survey.

3.2 Results

The following section presents the results of customer feedback from the exam paper databaseonline questionnaire survey.

3.2.1 Number of responses – sample size

From the time the pop-up questionnaire was introduced on 23 April 1999, until 28 June 1999,it had been presented to users a total of 1973 times. Of this number 1719 had opted to skip thequestionnaire, leaving a completion rate of 13%. The total sample size (i.e. questionnaires whichwere wholly or partially completed) on which the following analyses are based is 254.

18

3.2.2 Frequency of use

Respondents were asked to select from a range of options that corresponded with the number oftimes they had used the service. Over two-thirds of the respondents had used the exam paperdatabase fewer than four times as shown in Figure 3. Very few respondents, (5%) had used thedatabase more than 11 times.

This may suggest that the exam paper database functions as a one-stop shop for studentcustomers. Students are likely to be visiting the database only once or twice to print out all theexam papers they need for revision for a particular exam period. They therefore do not need tokeep returning to access the database once they have print outs of the papers they need.

Figure 3. Frequency of use

31%

38%

20%

5% 6% once2-3 times4-10 times11+ timesNo entry

19

3.2.3 Finding out about the exam paper database

A variety of channels of publicity about the exam paper database were put in place by theBUILDER team, including posters and information provided by Information Services staff oracademic staff. Respondents were asked how they had found out about the exam paperdatabase. As Figure 4 shows, the most common way that users had found out about thedatabase was through being told about it by a member of academic staff (30%) and a significantamount of students (22%) had found out about it via a friend. Browsing the university Webguide and the IS Web guide were each mentioned by 15% of users as the method by whichthey found out about the database.

Figure 4. Finding out about the database

30%

22%2%

15%

15%

1%6%

0%4%

5%

Tutor /lecturer

Friends

IS Staff

Browsing IS WebGuide

BrowsingUniversity WebGuideOther browsing

Poster

Flier

Other

No Entry

20

3.2.4 Print versus electronic exam paper resources

Respondents were asked whether they had ever used the existing printed exam paper service (asprovided by Information Services), in order to provide the BUILDER project team withcomparative information concerning users preferences for either the printed or the electronicexam paper resource.

Use of both types of resourcesJust under half of all the respondents (126) stated that they had experience of using bothprinted and electronic exam paper services, as shown in Figure 5. However, as shown in Figure5, 107 respondents stated that they had not used the printed exam papers at all.

Figure 5. Use of printed and electronic exam paper resources

0

20

40

60

80

100

120

140

print and electronic not used print no entry

Exam paper resource used

Num

ber o

f res

pons

es

21

Preference for print or electronic exam paper resourcesOf those respondents who had experience of using both types of resource, 56% expressed apreference for the database resource and 44% expressed a preference for the printed resource.Figure 6 illustrates this.

3.2.5 Print versus electronic exam paper resources, reasons for preference

An option in the questionnaire of an open-ended question provided respondents with theopportunity to give reasons for their particular preferences. The following presents the mainthemes to emerge from these qualitative responses.

Preference for database resourceThe following presents respondents’ reasons for preferring the electronic database resource.

Figure 6. Printed vs Electronic resource - preferences

44%

56%

printedresourcedatabaseresource

22

Access issuesMost of the comments in preference for the database resource centred on various issues ofaccess. Some respondents suggested that often in the printed resource, individual papers mightgo missing whereas access to papers in the electronic database was more reliable:

“It’s always there –people walk off with the printed paper.”

“Some printed papers are lost or not available.”

“Easy access; you know it will be there.”

Others referred to the way in which the electronic database can support multiple users, whereasthe printed resource is limited to one or two copies at a particular time and involves having tophysically go to the library to obtain it:

“….many people can use several different PCs whereas only one person can get hold of the paper.”

“less limitation on availability.”

“Easier to get hold of – not a mad rush up to exams.”

“Don’t have to wait until someone else has finished with the book.”

“I don’t have to go to the Main Library.”

Some respondents commented on the ease with which they could print papers whereas to get anindividual copy of the printed paper users are required to photocopy the original source:

“I don’t have to bother going to the library and using the photocopiers.”

“I don’t have to spend time queuing for photocopiers.”

“…can just print the paper out rather than having to copy it.”

A few respondents commented on the speed and ease with which they could search the exampaper database:

“The searching is quicker on the Web than having to find the correct book in a long line ofrandomly ordered books.”

“It is quicker to access.”

“Faster to look through / find.”

“Quick and easy to search, browse and download.”

23

CostSome respondents referred to the fact that they could print papers from the database for free(in some School clusters) whereas they had to pay to photocopy the printed resource:

“Able to print them off for free!”

“….because it is cheaper; in fact it is free.”

Preference for printed exam papersThe following presents respondents’ reasons for preferring the printed resource.

More extensive exam paper collectionMany respondents who preferred the printed exam paper resource did so because it containsmore papers, covering more years than the database which presently contains only one year ofexam papers:

“….there are more papers than in the database resource.”

“It gives more than one year –more useful for revision.”

Some respondents, however, commented that they would prefer the database if it containedmore years of exam papers:

“Although I like the online version of the exam papers, the printed version is still better. This is onlybecause the database version at the moment only contains the exam papers from last year’s exams.On the other hand the printed version contains many years of examples which is quite useful.”

“….there are more past papers than in the database resource. If more files were transferred to thedatabase resource…,then naturally it would be far more convenient than running around thelibrary and queuing for photocopiers.”

Access and use of PCsA few respondents preferred the printed resource as they experienced difficulty with using orgetting access to PCs and the appropriate software:

“It works out quicker to access these in the end; since there is always a queue for the computer.”

“It is difficult to get onto the computers to be able to use them efficiently other than that no realproblems.”

“You aren’t restricted to sitting (uncomfortably) at a computer.”

“Because you don’t have to find a computer with Adobe Acrobat Reader installed and you don’thave to find a computer which is fast enough to download the information before you graduate.”

“Because the printing facilities and copying facilities of the database make it more difficult toactually be useful.”

24

PrintingSome respondents either did not realise they could print papers off or had not been able to doso, in these cases their reasons for preferring the printed resource reflected this:

“because I can write on it and take it home for further reference.”

“can make notes etc.”

“can write on it, look at it at home.”

Ease of use / browsingOther respondents felt that the printed resource was generally easier to browse and use:

“As with anything on paper it’s easier to browse (i.e. you can just flick through the papers; pickingout any past questions that might be relevant to the modules that you do).”

“quick and easy.”

“more convenient.”

3.2.6 Evaluation of specific features of the Exam Paper database.

Respondents were asked to indicate from a range of choices how they rated the following threedistinct aspects of the database:

• Ease of use

• Time taken to find papers

• Screen design

Ease of UseRespondents were asked to choose a response which best reflected their rating of the exampaper database in terms of ease of use. Half of the respondents rated the ease of use of thedatabase as good, as shown in Figure 7. A third rated the ease of use as OK. Only a smallproportion of users considered the database to be excellent, poor or fair.

25

Time taken to find papersRespondents were asked to rate the database in terms of time taken to find exam papers. AsFigure 8 shows, most respondents considered the time taken to find exam papers to be good orOK. A significant proportion of respondents (25%) had not answered this question.

Figure 7. Ease of use

4%

50%33%

9%4%

ExcellentGoodOKFairPoor

26

Figure 8. Time taken to find exam papers

35%

29%

7%

4%

25%ExcellentGoodOKFairPoorNo entry

27

Screen designRespondents were asked to choose a rating for screen design. As shown in Figure 9, themajority of respondents considered the design of the screen to be good or OK.

Figure 9. Screen design

7%

32%

38%

9%3%

11%ExcellentGoodOKFairPoorNo entry

28

3.2.7 Access to database

Presently, access to the exam paper database is only available from computers on the Universityof Birmingham campus network. Respondents were asked to consider how likely they would beto use the exam paper database if it was possible to access it off-campus in locations such as thehome or at work. Figure 10 illustrates the responses received. Generally students expressed aninterest in being able to access the database off-campus. Nearly two-thirds of the respondentswere certain that they would access the exam paper database off-campus if it was possible andnearly a quarter considered that maybe they would access the database off campus if it werepossible. Very few (4%) considered that it was unlikely that they would access it off-campus.

Figure 10. Access off campus

62%24%

4%1%

9% Certainly

Maybe

Unlikely

CertainlynotNo entry

29

3.2.8 Other comments

Respondents were asked for any general comments regarding the exam paper service. Sixty-ninerespondents provided additional comments. Analysis of these comments is presented below.Many of the comments reflect on issues raised in other parts of the survey.

Provision of more years of exam papersMany of the comments concerned making more years of exam papers available on the database.This was also addressed in section 3.2.5. Apart from requesting more years of exam papers,most of these respondents believed it was a valuable service:

“It is an excellent idea. I think that if the database could be expanded then it would be anextremely useful revision tool. When I say expanded I mean to include more years.”

“It would be helpful if you displayed exam papers from the past two / three years instead of just lastyears exam papers.”

“Would be better if exam papers from past five years were available instead of only the previousyears exam papers.”

In addition, a few respondents requested that solutions to the exam questions also be providedas part of the exam paper database service:

“Would be better if model answers were available.”

Off-campus accessA large number of respondents expressed a desire to be able to access the exam paper databaseoff-campus:

“Having it available off-campus is a necessity, especially when you live on the other side of thecountry and you need an exam paper during one of the vacations.”

“I definitely think that the whole database and viewing system should be available to externalusers, because if it isn’t there is no point in using the system. If you are going to university youmight as well photocopy the papers.”

A few respondents were concerned that there was not enough done to let them know that theservice was not available off-campus:

“I have tried accessing during holiday from home when I started my revision and was disappointedto find I could not access the database.”

“It should be made clearer that the service is not available off-campus. When I tried to use it fromhome I kept getting the same irrelevant exam –no error message.”

30

“When informing tutors/lecturers of the database, could you ask them to make it clear that it isonly available ON campus, as this holiday I was relying on getting copies of past papers from thedatabase, but waited until I got home to London. When I was 100 miles away I was told that youcan only get copies on campus!”

Campus computer access and related issuesA few respondents mentioned the difficulty of getting access to the appropriate hardwareand / or software in departments of the University:

“The PCs in my department are extremely outdated and slow so the papers take a very long time todownload. Most of the PCs do not have Adobe Acrobat Reader so the computerised database isuseless.”

“Often crashes when Adobe loads.”

“The need to use the Adobe Acrobat Reader limits the number of computers which allow you accessto the exam papers.”

“It would be better if the papers were in a standard text format and not just scanned images.”

“Waiting for the papers to load and printing….takes an absolute age.”

One respondent hoped that it would be possible to save exam papers to disk:

“How do you save them to disk?”

Other general commentsA number of other comments were made. One respondent referred to publicising of the service:“Let more people know about it before they reach their final year. It is a good service but I onlyfound out about it by chance.”

Some were general positive comments about the service:

“An extremely useful and innovative service.”

“I think that it is a useful service as paper copies are time-consuming and often difficult to find.”

“It’s a great idea and it makes past exam papers a hell of a lot easier to find than journals in thelibrary.”

“Thank you –very useful for my elective I would not have found a past paper without this service.”

Generally even where criticisms of the service were made it was acknowledged that in principleit was a useful service.

31

4. Conclusions

During the initial test phase, indications are that the exam paper database was frequently usedby students to support them with their study. As might be expected, the periods of most usageoccurred in the run-up to the end of term examination session in the second semester. Thisincluded dates close to the actual exam period and dates prior to the Easter vacation, whenstudents obtained papers for revision during the vacation. Although one period of heavy usagewas recorded prior to the first semester exam period, usage corresponding to this period wasless heavy than that corresponding to the second semester exam period. This may be due tomore people finding out about the exam paper database towards the second semester examperiod.

Times of peak usage were during the day, concentrated during the hours of 10am and 5pm.The access statistics suggest that most users were able to download papers successfully. In theperiod examined, exam papers supporting science subjects were those most accessed, with thegreatest number of single accesses being made to papers supporting courses in the Departmentof Physics and Astronomy. Access data suggests that the service was frequently used by studentcustomers. The dedicated PC in the Main Library and other PCs in library clusters and indepartments which had high number of accesses were most often used to access the service.

The research suggests that most users were comfortable with various design features of thedatabase, such as ease of use, time taken to locate specific papers and screen design. Themajority of users rated these features as good or okay. Many respondents had been made awareof the service through a tutor and friend. In some cases, it seemed that they were not briefedsufficiently about what the service could or could not currently provide.

At present, just over half of the users who had experience of both the print and electronic exampaper resources expressed a preference for the electronic resource. Reasons for favouring theelectronic database focused on the following:

• Reliability that a specific exam paper from the year archived was available - i.e. not lost ormislaid

• Enabling simultaneous access for many users – considered to be particularly useful aroundbusy exam periods

• Printing – many users considered it easier to print a paper than to photocopy one from theprinted resource

• Speed of searching – some users considered that it was quicker and easier to locate specificpapers using the electronic resource.

32

Those that either preferred the printed exam paper resource, or provided criticisms of theelectronic resource, provided comments focusing on the following issues:

• Coverage: the exam paper database presently only provides access to one year of sampleexam papers

• Access to appropriate hardware / software: some users had experienced difficulties gettingaccess to a limited number of computers with the appropriate software installed

• Printing facilities: the value of the exam paper database was considerably reduced whenusers did not have access to appropriate printing facilities

• Browsing: some respondents considered that the paper resource was easier to browse thanthe electronic resource.

In addition, many respondents expressed a desire for the exam paper database to be madeaccessible outside the University so that access could be gained from home during vacationperiods. The indications were that, if this were made possible, it would become a common wayof accessing the database. The suggestion was that the preference for using the electronicdatabase would be greatly increased if the some or all of features mentioned above wereintegrated into the service.

Overall, despite criticism aimed at specific aspects of the service, the research has providedpositive feedback concerning the exam paper database service and the indications are that it hasa great deal of potential in the future to support learning at the University of Birmingham.

33

5. Recommendations

A number of recommendations for the BUILDER project team and University emerge from theresearch. These are:

• Provision of more information on the exam paper database explaining what it is and is notcapable of, and ensuring that tutors and IS staff are suitably briefed

• Continuing to promote and publicise the service to potential users. Information provided byacademic staff and Information Services staff is an effective way to publicise the service

• Provision of a more extensive online exam paper database – containing more than one yearof papers

• Putting in place the appropriate mechanisms for facilitating off-campus access to thedatabase for University of Birmingham students e.g. authenticated access via username andpassword

• Investigating the scale of provision in departments and information services of computerhardware and software to allow access to the database. This includes ensuring a sufficientnumber of PCs with Adobe Acrobat Reader are available

• Ensuring that users will be able to get access to printing facilities as this greatly enhances thevalue of the database.

Appendix

Exam Paper Database

Your views on the exam paper database are valuable to us - they will help usdevelop the service for future years and for other departments in the University.

Please spend a few moments answering the following questions.

1. How often have you used the Exam Paper Database?

once2-3 times4-10 times11 times or more

2. How did you find out about the Exam Paper Database?

Tutor / LecturerFriendsInformation Services (IS) staffBrowsing the IS Web GuideBrowsing the University Web siteOther browsingPosterFlierOther

3. Have you ever used the printed exam papers in Information Servicessites?

IF YES

I prefer the printed resourceI prefer the database resource

Exam Paper Database

http://builder.bham.ac.uk/exampapers/index.asp (1 of 3) [11/18/1999 2:55:06 PM]

Why do you prefer this service?

IF NO

I have not used an IS printed exam paper resource

4. Please rate the following aspects of the Exam Paper Database

Ease of use

PoorFairOKGoodExcellent

Time taken to find papers you wanted

PoorFairOKGoodExcellent

Screen design

PoorFairOKGoodExcellent

5. If this service were available to you off-campus (i.e. from home / halls ofresidence / other places), how likely would you be to use it?

CertainlyMaybeUnlikelyCertainly not

6. Please make any other comments about the Exam Paper Database whichyou feel we need to take into account for future development of this service.

Exam Paper Database

http://builder.bham.ac.uk/exampapers/index.asp (2 of 3) [11/18/1999 2:55:06 PM]

Thank you for taking the time to complete this questionnaire.

For further information please contact The Exam Office.

These pages are maintained by Stephen Leigh.

Last updated - 23 April 1999

24244

Exam Paper Database

http://builder.bham.ac.uk/exampapers/index.asp (3 of 3) [11/18/1999 2:55:06 PM]