evaluation methodology - de montfort...

Interim User Evaluation Report version 1.003

The Managed Learning Environment

at

De Montfort University Report 6.1 “Interim User Evaluation Report”

Evaluation Strategy and CampusNet Evaluation Findings.

Anne Jelfs and Mary Thorpe

Institute of Educational Technology

Open UniversityWalton Hall

Milton Keynes MK7 6AA

February 2002

Printed 22/05/2023 12:41 document.doc of 28


THE MANAGED LEARNING ENVIRONMENT AT

DE MONTFORT UNIVERSITY

(JISC 7/99)Evaluation Report 1

Introduction

This is the first formal report in completion of Work Package 6: User evaluation,

undertaken by the Open University in collaboration with the project team at De

Montfort University (DMU):

John Eyre, Project LeaderPat JefferiesMark SimpsonOpen University Evaluation Team:Professor Mary Thorpe, Director Institute of Educational Technology (IET)Anne Jelfs, Evaluation Project Officer, IET

Two working papers have also been produced to date by the Institute of Educational

Technology (IET) and are attached as appendices:

Appendix 1: ‘CampusNet: a preliminary review of web pages’ (Dr. Canan Blake) and

Appendix 2: ‘Report on an evaluation of the accessibility of the De Montfort

University Managed Learning Environment Project and Demo pages’ (Dr.Chetz

Colwell).

These reports provided initial formative evaluation of the content and navigability of

the web pages (Appendix 1) and in the case of Appendix 2, an in-depth report on the

accessibility the Managed Learning Environment (MLE) for users with a disability.

The reports were completed in September and November 2001, and have been passed

to the DMU team for use in further development of the MLE.

Through a series of meetings in 2001, the development of the MLE and the strategy

for its evaluation, both formative and summative, have been discussed at a number of

project meetings involving Professor Mary Thorpe as the director of the OU

evaluation project, and since December 2001, Anne Jelfs as the project evaluator.

At the meeting on December 5th, a draft evaluation outline developed by the OU in

response to the JISC project proposal and specifically WP6 in outline, was presented



and agreed with DMU colleagues. Details of this agreed approach are incorporated in

the evaluation strategy below.

The aim of this first report is twofold:

to outline the evaluation strategy developed by the OU and agreed with the project team

to report the first round of interviews with students and staff at the Milton Keynes campus, based on the CampusNet system.

Future reports will record any changes to the evaluation strategy and also present

reports of further findings as each stage of the strategy is implemented.

Evaluation Strategy

Workpackage 6 requires four key tasks from the evaluation:

1. Determine evaluation measures with the OU and other interested parties2. Determine pilot groups3. Conduct surveys/interviews/focus groups4. Review student performance records

Two deliverables are specified from these studies – 6.1 Interim user evaluation report

and 6.2 ‘Managed Learning Environment – Usability issues’. This report taken

together with the two studies delivered and listed above, was completed between

months 14 – 17 as planned, and constitutes deliverable 6.1 the interim user evaluation

report. The remainder of the planned evaluation will feed into deliverable 6.2, to be

completed before the project ends in August 2002.

The overall goal of the OU evaluation is outlined on page 4 of the Project Proposal as

follows:

‘Assessment of the impact of the Virtual Desk (name given to the DMU MLE) on users will be conducted by the Open University Institute of Educational Technology......They will use a combination of questionnaire, focus group, interviewing data collection methods, plus student performance records.’

In delivering this goal, we aim to provide a full exploration of the perspective of

users, particularly students and teaching staff and where possible, administrative and

support staff. The approach of illuminative evaluation (see Parlett and Hamilton 1972)

is particularly helpful here, with its emphasis on the context or ‘milieu’ within which

educational innovations and initiatives are carried out. With the perspective of



outsiders, taken for granted assumptions about the academic and working context at

DMU and its impact on users of MLE specifically, can be explored as part of the

qualitative data collection. Given the existing detailed studies of the usability of the

MLE interface and environment by an ergonomics specialist and member of the team

(Mark Simpson), this emphasis on the perspective of users and the impact of the wider

DMU environment makes an appropriate contribution.

Moving to the specific data collection strategy, activities have been agreed under the

four headings in the outline of Workpackage 6 as outlined below.

Determine evaluation measures

It was agreed that the basis for evaluation measures will be derived from semi-

structured interviews undertaken with both students and staff. These will be sampled

from the areas with whom participation in use of MLE in 2002 has already been

negotiated, including: Business, Law, Marketing, Pharmacy, Media, Art and Design

and Computing. If possible, interviews with Library staff and with selected areas of

administration will also be undertaken. The content of these interviews will include

the following:

What has been your involvement in the MLE thus far?

What contribution to X (their course/s) do they expect the MLE to make this year?

What expectations do they have about how students will use the course related

elements of MLE for their course?

How often do they expect students to use these elements? How much time do they

think students might spend with these online elements?

In what ways if any do they think student learning will be helped through their

involvement in the project?

Do they have any goals for their development of the use of the MLE on their

course/area in the future?

These issues and the reasons for particular responses, will be explored through semi-

structured interviews, varied in the case of Library and administrative staff to suit

their roles.



Determine pilot groups

It was agreed that some in-depth qualitative work with individuals will be undertaken

at a point when the MLE has been launched and available for use for a period of time.

It was also agreed that work of this kind should be done on the system that is currently

available at the Milton Keynes campus, namely CampusNet. Results from this work

would then be available during the construction of the first phase of the MLE. It was

agreed that this would constitute the first stage of work with pilot groups and that the

approach would include both interview questions and observation of tasks, as follows:

a) Users were asked to access CampusNet, as they would normally do, in terms of

which areas they visit, for what purpose, and approximate time

b) Users were asked to carry out agreed tasks to identify:

User expectations and typical usages

Any problems experienced with the interface

Any functions not offered which students would appreciate

Proficiency of users and differences between them

It was also agreed that a selection of staff at the MK Campus who have used

CampusNet would be interviewed. Pat Jefferies identified both appropriate staff and

students for this study, and 5 students and 5 staff from the Computing Department

were interviewed in December 2001. The findings from these interviews are presented

in this report.

Conduct surveys/interviews/focus groups

Interviews

The second stage of interviewing will be done with a sample of students and staff at

the De Montfort University Leicester campus, once the MLE has been launched and

available for some time (start of term 2002/3 – Sept 2002). The expectation is that the

MLE will be available as a valuation Pilot, at some point in March and process

validation and initial data collection will be scheduled to take place after that, but

before preparation for the examinations becomes a major preoccupation for students.

Staff and students will be selected for interview in liaison with John Eyre and will

include two students from each of five participating departments and at least one

member of staff from each, from the Library and the administration.



The format of the student and staff interviews will combine observations of usage

together with semi-structured interviews, similar to the approach reported here on the

CampusNet interviews.

Focus Groups

These will be set up in liaison with the participating departments at a stage when the

MLE is available and a group meeting feasible in timetable terms. If possible, students

from non-participating departments will also be included in order to gauge the likely

differences of view across the campus, in areas such as level of interest and preference

for using computer based resources for study.

Surveys

The purpose of a survey of students has yet to be firmed up, but it would be helpful as

a means of providing a more representative sample of attitudes and behaviour across

all departments, complementing the qualitative work which will focus on those

departments already committed in principle to making active use of the MLE.

Exploration is underway of the potential of the work of Professor Noel Entwistle and

the Approaches and Study Skills Inventory for Students (ASSIST), as an instrument

that has been extensively validated and that can be adapted to suit the particular needs

of this project. This would provide a measure of current approaches to consider,

together with attitudes towards using a computer for study and some indication of

relationships between the two. Further consultation on both the questionnaire and the

most effective method of distributing and returning completed copies, will be

undertaken.

The work of the Evaluation Team will focus on creating and testing a set of processes

and procedures that can be used to evaluate the MLE and its take-up over time (and

beyond the end of the funded project - end of August 2002). It makes sense to match

the evaluation processes with the level of features available and the number of users

using it. An additional benefit of this approach would be that the evaluation

procedures that we define, could be used by other funded projects.

The timing of data collection from students will have to be sensitive to the

requirements of their timetables and availability before the examinations, particularly



since many are not available once these are over. The feature-set of the current

version of the MLE is also an issue to consider, as it is not worth asking potential

users about features that do not currently exist. There is therefore a relatively tight

window of opportunity for interacting with undergraduates. Post graduates and staff,

however, can be approached beyond the completion date of examinations and this

may give us some flexibility. In any event, the final report should be available by the

end of the project.

Review Student Records

Student data records will be available via the MLE, although we are not sure whether

assessment scores will be available to academic staff, and whether the impact of MLE

on student study content and/or methods is likely to make analysis of their

performance a relevant factor. We will keep this under review.



CampusNet Evaluation InterviewsThe evaluation of CampusNet at the DeMontfort site in Milton Keynes consisted of

interviews with five computer science students (three male and two female) and five

computing department academics (two male and three female). The findings from the

student data are presented in this section, followed by the data provided by the

academics. All names have been changed to anonymise the interviewee.

The first part of the interview consisted of three tasks related to navigation around the

system relevant to student needs. A copy of the interview outline for staff and

students can be found in Appendix A and full data analysis can be found in Appendix

B.

Students

Jan Mature student Female 4th Year student

Paula Young student Female 3rd Year student

Adam Mature student Male 4th Year student

Colin Young (pt. time) Male MSc 2nd Year student

Russell Young student Male 4th Year student

The tasks were designed to allow students to search for relevant study and

administrative information held within the environment.

The first task consisted of finding a telephone number for one of the

University’s financial administrative staff, who liased with students about

their fees. Only one of the students found the name easily. The other

students had difficulties because they entered both the first and last name of

the target individual. CampusNet could only search for the last name.

Students felt that this was counter-intuitive.

The students were then requested to find a room allocation. Four of the

students never found the room and one of these looked for the location in the

modules and timetable section, which was where they expected it to be.



The final task was to find a relevant exam time in the examination timetable.

Three students found this easily and two never found the timetable at all. One

of these students looked in the resource area.

The conclusions drawn from these tasks are: a) students need some training at the

beginning of their academic careers; b) assumptions built into the layout and

operation of any MLE need to be evaluated and developed to fit more closely with

user expectations and intuition. In the case of CampusNet, further development would

need to allow searching by first and second names and greater consistency of layout.

All students commented that their primary use for CampusNet was for the timetables

and the resources. They liked the ‘tab’ layout and felt it was one of the good points

about the system, however there were comments about the need for consistency in the

use of only primary colours. Mailspinner is in a totally different colour range and not

based on primary colour hues. Students found the book exchange area useful and

suggested that a Student Union site would be a valuable addition.

All students need to be able to access the site both on and off campus, which means

that some students will be using modems to connect to the Internet. Large downloads

could therefore be a problem.

One problem students identified was that they had to log on to the system twice.

Once for CampusNet and again for Mailspinner (the email system). All the students

made negative comments about Mailspinner and to achieve an acceptable web-based

email system they used either Yahoo or Hotmail. The problem with students using

Hotmail or Yahoo is that they will have less need to use CampusNet, except as a

portal to other sites. Everyone complained about not being able to create a personal

address book in Mailspinner. There were also email compatibility issues because

Internet Explorer does not regard Mailspinner as a valid email system. Students would

also like to have notification when new mail has arrived and an easier way of deleting

emails. Some of the students also mentioned the need for either a message board

system like MSN or a group work site such as Shockwave to allow for more

interaction and sharing of resources. Some students had used WebCT for one recent

module and they made comments on how it had been used. One of the students

commented on a feeling that using WebCT took away time from assignment



completion, and another student felt that the discussions conducted on WebCT were

monopolised by two or three people. This might indicate a need for a moderator.

There were a number of other items that students commented on, such as the

recommendation that the award title should be listed in the heading which in turn

should expand to take the navigator to the relevant programmes. Other student

comments included making the first page more interesting and inviting and the home

page should have a directory. One student recommended the Dynamic Systems

(DSDM) web site as a good example (http://www.dsdm.org/index.asp).

Students expected to find on the site their tutorial changes, a diary, the availability of

‘labs’ and expected to know that all changes to the system were made clear as soon as

they occurred.

Academics

The academic staff that contributed to the data collection were selected by Pat

Jefferies as individuals who had a diverse range of computing skills and differing

approaches in their use of CampusNet.

All of the academics have two email systems, CampusNet’s Mailspinner and Outlook.

Having two systems was frequently referred to as a ‘nuisance’. However, there was

some consideration of the fact that without having to gain contact lists from

Mailspinner and CampusNet, many of them would not use CampusNet.

The academics interviewed at the Milton Keynes site had previously used Word and

HTML to produce documents and web pages, however there had been a policy

decision to move to Cold Fusion, which inhibited the original producers from putting

material on the Intranet. They could no longer use Front Page or Netscape composer

and would need to edit raw html text. All of these problems led to a feeling of

frustration, disempowerment and lack of ownership and put increasing pressure on the

technical support staff to provide resources for the academic staff.

Staff did suggest a few areas for improvement including a library link, the removal of

obsolete modules and the address bar identifying the actual location rather than


http://www.dsdm.org/index.asp


‘Navtab’. By connecting the QLS system and a better match between the

management systems and the data provided on the Intranet there would be improved

administrative support. Staff also suggested that student timetables should be on the

site and that CampusNet should launch when they first log on. One academic had a

number of issues surrounding the design of the site, and criticisms included the use on

CampusNet of long wordy descriptions and large empty spaces.

In their teaching some of the academics were using conferencing as a means for group

work and collaboration. The system used was WebCT which uses threaded

discussions. One person registered a need for statistics from the conference on

student use and frequency to enable understanding of the depth and value of using

WebCT in their teaching. Another academic only made course material available as

he teaches the course and not all at the beginning. He does this because he fears that

if it is all available at the beginning of the semester, then students will stop attending

the lectures. He says “students are not good at learning just because there’s a pile of

resources there.” He thinks that students need to be taken through a process.

Finally, both academics and students need training support to allow full use of the

technology and recognition of its full potential. Without training there will be less

take up of the opportunities and less innovation in the teaching.

Key Points

Students thought the ‘tab’ system was intuitive and easily recognisable.

However, they were concerned when some tabs led to modules etc which no

longer existed.

Staff considered the layout needed improvement; of empty spaces should be

reduced and a flatter structure should be used.

Both students and staff needed more introduction to, and training on, system

use.

More help from support staff was needed to enable academics to use the site

and to put course material on the site.



Students and staff wanted an email system that allows them to create their own

email address book, has a better interface, enables multiple file attachments,

includes a search facility and is easier to delete old emails.

Staff and students want a message board and share-ware system so that group

project work etc. can be handled.

Modules, Degrees and Courses should all be up-to-date and not listed in code

order. It would be preferable if they were categorised either in terms of level,

the title of the module, in alphabetical or in relation to named awards.

Staff and students want to be able to put ‘favourites’ on the front page and the

front page to have important messages highlighted.

Students wanted a link to the library and to student financial information e.g.

fees.

Some students would like a forum area for discussion purposes.

Students would like more lecture notes on the system.

Better links between management/administration and CampusNet, so that

information is up-to-date and timely (e.g. exam results to tutors and module

details).

Staff would like CampusNet to launch when an individual ‘logs on’ to the

computer, as they felt this would encourage more usage.

Staff would like to be included when designing the environment, so that there

is a feeling of consultation and recognition of their needs.

Staff would like student profiles on CampusNet so that support and

information can be provided to students (e.g. degree classification, exam re-

sits).

In addition to points of detail made by both staff and students about the operation of

CampusNet thus far, a number of concerns of more general significance were raised,

particularly by staff. These concern the infrastructure and roles and responsibilities

surrounding the implementation of CampusNet, all of which are equally applicable to

the MLE.

First there is the concern about effective systems and links between different areas of

the university. One of the focal points of this for example is the need to ensure

automatic updating of module/course information rather than manual updating with



each change. At present there is no system for ensuring that changes originating at the

level of university administration or department, are automatically transferred to the

data held on CampusNet.

Second, the ownership of CampusNet was damaged by the move to require expertise

in ColdFusion for inputting content onto the site. Academics now have to go through

technicians in order to get their material put up and this has had a disempowering

effect and reduced academics’ enthusiasm for using the system.

This therefore begs the question of decision-making and ownership of any managed

learning system. In this case a decision appears to have been taken without

consultation among users – or without addressing at least some of their concerns in

advance. A link with the existing decision-making structures of the university may

need to be thought through in order to build in some form of monitoring and oversight

of the system. This would enable user feedback to be considered and judgements

taken about how best to improve systems and engineer the necessary changes.

Conclusions

CampusNet has a lot of valuable points to offer in designing the Managed Learning

Environment. This report, together with previous reports, suggests a number of ways

in which the system is functioning reasonably well and also where it can be improved.

The Project Team have identified a number of these points and have addressed the

issues raised by staff and students for the Managed Learning Environment at

DeMontfort University. This includes reviewing the email system, consultation with

staff and students and access to resources. Analysis of the evaluation conducted with

academics at the Leicester site is contained in another report.

References

Blake, C. (2001) CampusNet: a preliminary review of web pages

Colwell, C. (2001) Report on an evaluation of the accessibility of the DeMontfort University Managed Learning Environment Project and Demo Pages (Internal Report)



Parlett, M. & Hamilton (1972) Evaluation as Illumination



Appendix A

CampusNet – Student Users Evaluation topics

Explanation of project and my role as evaluator.

Request student to log on.

TASKS

1. Find telephone extension number for Sue Wakelin. She deals with grant queries.

2. What is the room/location for the Computer Science Yr3 lecture on Thursday? MULT2002, Dynamic Web publishing?

3. What date and time is the Computing and Ethics exam?4. Show me a task that you typically do on CampusNet.

Which sites or areas do individuals use most frequently. Are these sites ‘book marked’? Have they tried to personalise the site. How do they move from one page to another, or one resource to another? What do they think of the design layout? How easy do they find it to use. What training (if any) has been offered? Which sites have never been used?

In what ways (if any) do they use CampusNet in their studies? Has this use (whatever it is) involved extra work for them? If so, roughly how

much? Did they get help from anybody? Would they have liked help? Have students tried to put their information such as books for sale on the

system. What do you think are the benefits of using CampusNet in the ways they’ve

indicated?

If CampusNet is not being used, do they replace it with something else? What would they like to see included on the site. Have students ever accessed CampusNet through the Internet. How do they feel about privacy and the system?

CampusNet - Academic Users Evaluation topics



Explanation of project and my role as evaluator.

Request academic to log on. Which sites or areas do individuals use most frequently. Are these sites ‘book marked’? Have they tried to personalise the site. How do they move from one page to another, or one resource to another? What do they think of the design layout? How easy do they find it to use. What training (if any) has been offered? Which sites have never been used?

In what ways (if any) do they use CampusNet in their teaching? Has this use (whatever it is) involved extra work for them? If so, roughly how

much? Did they get help from anybody? Would they have liked help? What are their expectations for student use of CampusNet –

o What do they think students could use it for now?o What are their medium/long term expectations for how students could

use it in future? Have staff tried to put their learning materials (or other material e.g.

assignments?) on the system. How do they feel about using the system for teaching? (meaning

bored/disempowered/excited/unenthusiastic/none of these etc and why) What do they think students gain (or might gain) from this method of teaching

i.e. using CampusNet in the ways they’ve indicated.

If CampusNet is not being used, do they replace it with something else? What would they like to see included on the site. Have staff ever accessed CampusNet through the Internet. How do they feel about privacy and the system?



Appendix B

CampusNet Evaluation Interviews

Russell - 4 th Year Computer Science student

He uses CampusNet for lecture work and accesses it from home as well as on site.

Accesses home & DMU

Tasks: Finding the name was problematic. He put in the search for first as well as surname. The search facility does not work when the first name is included, it only requires surname.

No idea how to find room allocation.

Finds exam timetable easily.

His primary uses are email, timetables and resources. He does not customise or add favourites. He hates Mailspinner because he thinks it is awkward to use and can only have one document attachments. This means when sending various documents to his tutor he can only send one at a time and therefore has to send numerous emails. To overcome these problems he uses Yahoo mail. As part of his degree project Russell is using an on-line questionnaire and the results are emailed back to him, but because of the difficulties with Mailspinner, it is not recognised as a valid email system by Internet Explorer and the results are not sent to him. Says that a lot of students use either Yahoo or Hotmail rather than Mailspinner.Russell also says that he couldn’t create an address book in Mailspinner (demonstrates) there is no error message to say address is not completed. This means that to find an email address the student would have to leave Mailspinner and go to CampusNet to find an address.

He likes the tab layout and finds it easy to use, but has had no training. When he returned to DMU after a break he found that the system had changed, no one had informed him and the log-ins were mailed to his email university account, causing frustration.

He has looked at most of the sites once.

Suggests a Student union site would be useful

Search facility to include first and surname.

Training needs

Uses for email, resources & timetablesDocument attachments to email system.

Email compatibility issues.

Alternative email systems used.

Functionality of email system.

Changes to system need to be made clear to users.

Inclusion of Student Union site



Colin – MSc computing student, part-time 2 nd year

Use of CampusNet depends on the lecturer’s usage, but goes into CampusNet once a day. He looks for anything new such as resources and email

Work access

Tasks: Finds the name, but again does not go through pages, uses search facility

Can’t find room allocation.

Finds exam timetable easily.

He has never seen the favourites list on the front page before, and says this is because he has never read this first window. He likes the layout and tabs and finds it easy to navigate, but things aren’t always where he expects to find them. Usually he looks at the resources area but thinks the programme information resources does not have all of the courses including his!!! The modules relating to him are at the bottom of the page and are in course codes, but don’t have the programmes of study. Thinks there should be a degree heading that expands to take you down all the relevant programmes. He knows some modules overlap, but thinks these can be put under each degree programme. Likes the book exchange area.

Colin says that Mailspinner is bolted on and doesn’t follow the feel and design of CampusNet. He has two accounts on Mailspinner, one from B.Sc.& one for M.Sc. If he uses his old account then he loses CampusNet and has to use the back button to get to CampusNet. Mailspinner also does not inform you that you have new mail.

He liked it before when DMU had Win Popup, which was a message area and would like something like that again. “When I’m at (work address) it would be useful to send things for group work to someone here.” Has used Shareware at work and thinks something like that would be good.

He thinks there could be more information provided on things that affect students. Would like a Campus diary and a student message board.

There is no way to feedback about the site. No training, but thinks useful things provided like knowing that when you set the preferences you can change your log in to once, rather than once for CampusNet and once for Mailspinner. (All the other students interviewed logged in twice)

Training needs

Make first page more interesting.

Up-to-date information – needs someone to support this.Degree module profile requested.

Book exchange useful

Housekeeping of student data needed.

Message board would be a useful addition.Notification of new mail on Mailspinner.

Shareware requested.

Student message board

Feedback needed to technical support.

Training needs



Paula – 3 rd yr computing student

She uses Mailspinner and the module information frequently and always looks at the Announcements. Uses it at home as well as DMU. Says she has never explored CampusNet that much.

Home access

Tasks: Finds phone number easily.

Looks for room allocation under modules and timetables.

Doesn’t use the exam timetable in the Announcements section on the front page, although she did state earlier that she read the announcements!

She doesn’t like the colour of the Mailspinner tab because it doesn’t seem to fit. She says she tried to put favourites on front page so that she could have things like her timetable without going through the site, but says it doesn’t work. Would like all the options put on the first page

Thinks connections are not always there.

Has used the 2nd hand book site to buy books.

Would like up to date availability of the labs, so that she knows when they are free.Would like MSN messenger. Then can send documents or group projects to her colleagues. They could then share files and it would be much quicker. Email is useful, but doesn’t know if lecturer is there and able to answer her questions.

Expected to be able to ‘click’ on the CampusNet icon, but it doesn’t. Would like every page to be the same.

Training needs

Consistency of layout.

Layout or training needs?

Check connections work.Book exchange useful.

Lab timetable

Messenger system requested.

Consistency of layout.



Adam – 4 th yr full time student

Logs in from home frequently and uses it as a ‘plug in’ to the university.

Home access

Tasks: Uses search facility for phone number. Says it’s a bit of a guess but knows where to look. Finds phone number easily.

Can’t find room allocation but feels confident that he would find it eventually. Finds exam timetable easily.

Doesn’t personalise the front page says it is ‘literally a doorway’ and generally he has logged on for a purpose and then logs off. Usually he uses Mailspinner and sometimes for module information. Says each module should have lecture notes on CampusNet. Changes to tutorials etc should be on CampusNet. He reports that he frequently works at home and uses CampusNet to get information, he wouldn’t need to come in if tutorials were cancelled.

Thinks it is a pleasant design and likes tab idea (one that he uses in his work). He thinks it is a pretty good system and has looked at most sites at least once.

Uses WebCT for ethics module and he finds it useful, but feels it takes time away from assignments. However, he thinks it could be useful to ask others for support or help.

Would like a forum area.

Some students more inclined to work through the sites in ‘trial and error’ fashion. Training needs

Consistency of module provision

Changes to tutorials on web site.

Conferencing system

Conferencing.



Jan - 4 th yr student – Computing Science

She uses CampusNet for Resources and Mailspinner. Says she never uses the other features. Uses CampusNet for email at home.

Home access

Tasks: Uses search facility for phone number. Thinks that there must be an easier way to find names etc straight a way.

Can’t find room allocation at first. Trys the resources area.

Trys to find the exam time through the resources area – doesn’t know how to find the exam and doesn’t succeed. She expects resource area module section to have exam timetable.

Thinks Mailspinner could be easier to use. When the student logs on they have to set up their own folders etc. She hasn’t ‘figured out’ how to save emails that she writes and sends.

Complains about having to log into CampusNet and then log in to Mailspinner. She also doesn’t like the delete feature in Mailspinner. You have to what is this character? the mail to be deleted, then click delete which marks the mail with an X, but it still remains on the list. It is only when you use empty trash that it goes. She doesn’t know if when the ‘trash’ is emptied there is anyway to retrieve it. She feels that it would be easier to have a delete and undelete than empty trash. The system also keeps ‘refreshing’ itself which means that if some mail has a ?it disappears after the refreshment and she has to start again – very frustrating.Mailspinner also hides the tabs to get back to CampusNet.

She feels that she hasn’t enough time to search or ‘play’ with things so doesn’t interrogate the site. Says it takes a ‘fair while’ to get used to CampusNet.

Resources area most frequently used area - for course information. She thinks that the codes and module names should be the other way round. This would make it easier to find information. The module page also needs sub-headings, and the programme headings could be put under the modules. She likes hyperlinks from module information to other sites.

Jan doesn’t like the Homepage. She suggests that there should be a directory there, e.g. resources etc. there but not on the tab. Says to look at the DSDM (Dynamic Systems) website where in the table down the side as you run the mouse over it, it expands and you can see what it contains as you go down the list. http://www.dsdm.org/index.asp Whereas with CampusNet you can’t see what you want until you’ve reached the site.

Name search on first page.

Training needs

Layout or training needs?

Training needs/system needs

Email features need reviewing.

Only a few areas accessed regularly.

Codes and module names rearranged.

Improved homepage to include expansion of directory.


http://www.dsdm.org/index.asp


She says that most people won’t use WebCT because their personal opinions would be laughed at. Says it is monopolised by 2 or 3 people. She won’t use it because you can’t choose who you want to talk to, it is for everyone. Would like to share ideas with only a few, and acknowledges that it should be open to everyone, but feels that is why people don’t use it.

Monitoring of discussions on conferencing system.



Five Computing dept. Academics interviewed. Two male and Three female.

Tricia

Tricia doesn’t access CampusNet from home and doesn’t customise the front page. The most frequently used item is the mail contact list. She says that setting up a personal list is a ‘bit convoluted’.

Frequently used for email contact list. Personal contact lists difficult to set up

All the academics have two email systems. One is Mailspinner which is on CampusNet and the other is the university standard of Outlook. Tricia says that because of the problem of using 2 mail systems it is a ‘nuisance’, but if the contact lists were in Outlook it would probably stop her from using CampusNet.

She occasionally goes to the Resources module area, where she knows that a colleague puts up the project work. She does put material on CampusNet. Previously she had her own area on the site and could produce work in Word and then transfer it to HTML. Now she has to use ColdFusion and since the move to ColdFusion she has not put anything on the site. ‘Its stopped me from doing a lot of developmental work.’Uses WebCT rather than CampusNet and ColdFusion. Using ColdFusion means that when she wants to do things quickly she is unable and has to go through a team. Although she recognises that it is good for quality control she feels a loss of control of her own work.

On the site there are some modules present, which don’t exist now. It would be a small task to remove them, but now she has to ask a team to do it. Feels there should be a way to link with administration so that when modules are removed they are also removed from CampusNet. Students may think modules still available when obsolete. Tricia recognises that the engineers are busy and need additional support to maintain CampusNet.

She would like statistics on the usage of CampusNet to see what students were actually using. Feels it is pointless in putting material on sites that students don’t use.

Says she has used all the tabbed areas and suggests that the library was linked to the service area.

Thinks it’s a modern looking interface and she uses the tabs rather than the headings to find things. Says the labels sometimes don’t match headings.

The materials that she has on the site are those that were there before CampusNet was used. Her lecture notes are there.

Two email systems increases workload and dissatisfaction.

Resource area moderately used.

Feeling of dis-empowerment due to system changes.

Links between CampusNet and the administration depts.

More technical support.

Statistics

Addition of Library link to site.

Ensure that links are correct.

Has not updated her materials due to changes in system.



The modules list needs labels in the differing levels and course title should be before the code. Also codes have changed so students looking by codes are lost. The QLS system could be linked so module codes are changed automatically.

CampusNet should launch as you log on, otherwise it is not being used. Says it would make her look at the Announcements and use MailSpinner more. Once people can opt out then there is ‘move from the object of the exercise’.

When she has presented the timetables on the site there is less likelihood of the notice board timetable being removed from the board.

Demonstrates that some students have never logged on to WebCT when it is a necessary part of the course. Does not like the way that the discussions in WebCT are threaded in long lists when you first see the WebCT pages. To conduct collaborative work she would want the software to support it and students would need a shared area.

Swap over course title and course code to make it easier to find course.

Launch CampusNet when user logs on.

Feels students use the timetables on the site.

Recognition that not all students use discussion areas.

Student shared area.

Angela

Angela logs on very infrequently to CampusNet at DMU and never from home. Probably logs only once or twice a week and some weeks, not at all. She doesn’t like Mailspinner. Didn’t know she could customise the first page or bookmark.

She says that setting up a personal list is difficult and so she copies addresses of students from Mailspinner and puts them into her own email lists. She corresponds to all her students via email rather than Mailspinner.

She recognises that this system is limited but doesn’t like the way it sends messages and it limits her sending things. When she types a message she has to do it in Word and then spell check it and then paste it in. When she wants to cc a message she can’t do multi cc’s and multi-groups. She finds the style doesn’t suit her. The spell check facility missing, the groups and the copying of messages sent to her do not suit her style.

Uses CampusNet only if she has too. Doesn’t use it to find things except policy documents. She can always find other ways of finding things out.

Doesn’t put notes/lectures on CampusNet because it is not easy for academics to do it themselves. CampusNet was developed without user analysis or needs done. No feeling of

Frequently used for email contact list. Dislikes Mailspinner.Training.

Personal contact lists difficult to set up

Mail set up.

Needs spellchecker in Mailspinner.

Avoidance of CampusNet.

Difficult to put lecture notes on CampusNet. No user analysis and



ownership by academics. The engineers know what they are doing, but you have to wait for them. Doesn’t think putting lecture notes on the system is a good way of using the technology.

Doesn’t find the layout intuitive. Only one or two parts she uses as she feels she could probably find everything elsewhere. Doesn’t browse the site.

Angela thinks that project work could be an interesting activity for students through CampusNet but for the future not now.

Thinks that by not using reprographics and using only web-based lecture notes you are passing on the costs, either to another area of the costing system or to the students.

She can see the worth of assignments being put on the site. Has received electronic assignments but feels this is a question which has not been addressed by the department. There are issues such as lateness to be addressed.

There is a web site for her module at Leicester and tells her students about that site.

Feels she has to ‘dig around’ a bit to find things on the site, as it’s not intuitive as to where to find things. Would like more inclusion of the academic staff which in turn would give ownership.

no feeling of ownership.

Layout is not intuitive.

Potential use in project work.

Passing costs of reprographics to users.

Assignments

Uses Leicester’s web site.

Consultation with academics.

Gary

Gary’s main concerns are about the database underlying the system and the use of ColdFusion. He says the technicians know ColdFusion and implementing web pages, but are not database experts or systems experts or graphic designers. The system integration is poor, although he acknowledges that CampusNet is not completely useless.

He feels that there are issues of control with the system– who should have it and whether it should be centralised or individuals who have control over publishing information. With central control there are issues about who has access.

The resources page – “when people understand links they end up with very deep trees” – this page needs to be much flatter. “Look at all that empty space there.” Very long wordy descriptions.

By putting module details by years in the resource area’s first

Underlying systems need work

Central or individual control of materials and system.

Design issues



page, it would make it much flatter.The module page is stored as raw text and html. Says he can edit html but because they use ColdFusion he cannot use a standard web editor. Even if you have produced a page which doesn’t use ColdFusion he says you still have to go through ColdFusion’s security checks and you can’t use Front Page or Netscape composer. You have to edit the raw html text. Says that ‘we should be using proper editing tools’. “Lots of people don’t do this (use ColdFusion) because they don’t want to learn it.” It is a barrier.

Referring to the modules, he comments that the information is not generated anywhere it is a repository and if any of the modules go then the site is not changed automatically. The information on CampusNet is unrelated to the marking scheme. He is the module leader, but other people can access the information and change things.

The modular system changes frequently and codes change frequently but old codes are left on. There are inconsistencies of information which can be distressing to students. Changes to administration cause problems e.g. changes to resit system which were programmed into the administrative system but the academic staff didn’t know this. Regulations change and students should be informed.

Mailspinner – students are stuck with it. ‘For Internet users at home then web forms are good, but you soon get fed up with things like Hotmail’. Says you need a proper email system and these are different to web-based systems. There is no tree structure in Mailspinner. The address book management he feels is appalling and pushes people into using 2 mail systems which is counter productive. Says Mailspinner is really just a send system. Gary can use Outlook at DMU and when at home he can see the same address books but group email has to be through Mailspinner unless he has put it in Outlook. What he does is to cut and paste group mail to his own mail system.

Gary is critical of the systems used to produce data, which may have to be keyed in more than twice e.g. exam results. This can lead to errors. Wants a match between the management systems and data provided on the Intranet.

He uses his own Web page and links it from CampusNet. This page has teaching content and the material has nothing to do with the Intranet site. He puts the material on as he teaches the course, not all at the beginning as he feels students won’t attend if it’s all there. He says “students are not good at learning just because there’s a pile of resources there.”

Design issues

ColdFusion causes personal difficulties in his use.

No linking databases.

Dis-empowerment

Updating and maintaining the system issues.

Mailspinner is fine as a web based email, but not good for other use.

Administration integration with system.

Personal web page rather than use CampusNet.

Teaching materials



Students need to be taken through a process.

Ivor

Ivor logs on every morning because he uses it quite a bit. He has his own email and CampusNet available all the time. He also access at home and puts things on his favourites.

He puts information on the projects up and he maintains this after the technician puts them on the system. He provides information such as feedback from the external examiner.

He also provides similar information for staff plus other things such as policy. He customises the pages so that he can quickly update information.

He puts material on the site and tells students the links with Leicester. He uses it to look up student email addresses and to look for ‘phone numbers.

Says Mailspinner is not used by many academics and is only used for group addresses. Feels it is easier to set up groups in Mailspinner rather than Outlook. A list of students is devised from enrolment and this is sent to the engineers to put on and then the engineers give out an email account.

There is no search facility on the email. You can find a name by looking in the address system of CampusNet, but can’t cut and paste or send an email from the list.

Can’t see why student profiles are not on CampusNet so that he can provide support and information to students.

He thinks a conference area would be good for group work.

Says there are some modules, which have no information on them and people should be consistent.

Uses CampusNet a lot.

Maintains his own work on CampusNet.

Experienced ColdFusion user

No linking databases.Chance of errors.

Email search facility

Student profiles on CampusNet would be useful.Conferencing system useful.

Consistency of presentation.



Rebecca

Rebecca uses CampusNet mainly for teaching purposes and the email list management. She uses it as a resource.

She uses the staff section for Union items and for management meeting information.

She did use to bookmark, but with a new computer she hasn’t. She says that she finds the ‘address thing at the top Annoying. It always says Navtab’. So when she looks for things there is no address. Material held at Leicester is also set as Navtab.

Says she is not a programmer and she wants a service. She is not interested in learning ColdFusion and feels its all gone too ‘techy’. There are also usability issues - at first CampusNet was much more user-friendly. She could save documents as HTML and then link. Found ColdFusion difficult. Says you can’t organise your own FrontPage. “So you end up with work arounds”.

She doesn’t like the mandatory first page. Would like each site to state the purpose of the site e.g. administration resource.

Most of her students are part-time so they access CampusNet from work. Some of them cannot do this because of company policy.

Rebecca says the design is all right, but has had no training in using it. She has looked at all parts of CampusNet but has not used all the parts.

She has set up group lists for email from the lists in MailSpinner.

She says this is an MK only Intranet and it is not connected to Leicester as a resource. The MK students can get materials from Leicester but not the other way round. She thinks she would like to be able to access staff information at Leicester, which she demonstrates is not currently available to her.

She uses CampusNet as a resource, e.g. assignments and lecture notes, but doesn’t put them there herself.

CampusNet is a resource

Address bar should show location.

ColdFusion causes personal difficulties.

State what is included in the location.

Access issues.

Looked at sites but not used them all.

Group email

Connections between sites.

ColdFusion stops material production.


evaluation methodology - de montfort...

Documents