annual report 30 september 2003 - birkbeck, university of ... · qmark question mark perception...

OLAAF Annual Report 2003 1

Annual Report

30th September 2003

Project Number: 5/02

Project Title: OLAAF: OnLine Assessment And Feedback

Project Website: http://www.bbk.ac.uk/olaaf

Completion Date: September 2005

Project Director: Dr Richard Rayne

School of Biological and Chemical Sciences

Birkbeck College, University of London

Malet Street, London

WC1E 7HX

0207 631 6253

[email protected]


Summary

OnLine Assessment And Feedback (OLAAF) is a 3-year project funded under the HEFCE FDTL4. OLAAF’s

primary aim is to support lecturers in the design, delivery and evaluation of Computer Based Assessment

with Feedback. Deliverables from this project will be widely disseminated throughout the UK higher

education community. The project responds explicitly to several of HEFCE’s priorities as outlined in the

HEFCE Strategic Plan, 2003-2008 (see Sections 1.8.3 and 1.8.4 in this report). This report retrospectively

covers Year 1 of the OLAAF project and sets out a plan for Year 2.

Key activities and main accomplishments for Year 1 include the following (with relevant report section

numbers indicated):

• Establishment of a Network of OLAAF Participants and of Links with other Projects

A core London-based team has been recruited and trained (1.4.1); excellent working relationships

based on clear agreements have been established with Consortium Partner Sites (1.4.2, 1.4.4;

Appendix 1); seven enthusiastic and committed site representatives from other UK institutions have

been recruited to form an Interest Group (1.4.3 and Appendix 2). Collaborative arrangements have

been established with other FDTL projects (1.5.1, 1.5.3), LTSNs (1.5.2) and others (1.5.5).

• Production of Assessment Construction Resources and Authoring of Computer-Based

Assessments

Computer-based assessments (CBA) are presently being developed in parallel at various OLAAF sites

(1.4.5, 1.6.3; Appendices 1 & 2). OLAAF supports this innovation by providing: a range of

Assessment Construction Resources available online (1.4.6, 1.4.7, 1.6.2); centrally placed expertise

(i.e. at Birkbeck) in CBA and in evaluation of CBA; funding to support training and authoring at Partner

Sites; online technical support (1.4.4); and workshops to embed skills. The culmination of the first

year’s work will be the OLAAF Conference in January 2004, which will bring together all partner and

Interest Group sites to share successes and plan collaboration on future developments.

• Establishment of Strategies for Dissemination of Project Deliverables and for Evaluation of the

Project’s Products and Activities

A detailed Dissemination Strategy (Appendix 3 and Section 1.7) has been produced in consultation

with the project Steering Group (1.4.8), and all members of the team have worked effectively to raise

the profile of the project, both within their institutions and more widely. Dissemination has been

significantly aided by ongoing collaboration with institutional networks, other FDTL projects, relevant

Learning and Teaching Support Network Subject Centres and the Assessment Project Network set up

by the National Coordination Team (see Section 1.5).

The project will benefit from our continuous formative evaluation as well as through evaluation by

external experts. An Evaluation Strategy (Appendix 8 and Section 1.7) allows for regular evaluation

both of operational aspects of the project, and of the assessments and resources produced (see also

Appendix 7: Report of the First External Assessment Evaluation).

The project’s financial position is excellent (see Section 3) and we foresee no impediments to future activites

owing to budgetary constraints. A plan for activities in Year 2 is described in Section 2 (see also Appendix

9). We are presently on course to achieve all key objectives/targets as set for Year 2.

We look forward to an exciting second year of the project, when the computer-based assessments authored

during the first year will be integrated into module assessment strategies, and further end-user evaluation

can begin. We are certain that our commitment to maintaining principles of good assessment practice will be

rewarded by evidence of a positive impact on student learning and retention.


CONTENTS OF OLAAF ANNUAL REPORT 2003

1. Year 1 Review 1.1 Main objectives/targets for year 1

1.2 Changes to the project plan

1.3 Particular successes in year 1

1.4 Major activities undertaken in year 1

1.5 Collaborations and Links

1.6 Outputs from year 1 activities

1.7 Evaluation of the work

1.8 Learning from activities and progress made during year 1

2. Year 2 Plan 2.1 Main objectives/targets for year 2

2.2 Changes to the project plan

2.3 Help needed for year 2

3. Financial

Statement 3.1 Expenditure Statement for Year 1

3.2 Anticipated Expenditure for Year 2

4. Annexes A & B

5. Appendices 1 Partner Site Summary

2 Interest Group Summary

3 Dissemination Strategy

4 Birkbeck Magazine, Summer 2003

5 Programme for Plymouth Event

6 Flyer

7 External Assessment Evaluation Report, August 03

8 Evaluation Strategy

9 Project Timeline

10 Budget (years 1-3)

Glossary APN Assessment Project Network CBA Computer Based Assessment CBAF Computer Based Assessment with Feedback CIAD Centre for Interactive Assessment Development, University of Derby FDTL Fund for the Development of Learning and Teaching HEFCE Higher Education Funding Council for England LTSN Learning and Teaching Support Network NCT HEFCE’s National Coordination Team OLAAF OnLine Assessment And Feedback QMark Question Mark Perception TRIADS TRI-partite Interactive Assessment Delivery System


1. Year 1 Review

1.1 Main Objectives/Targets for Year 1

Annex A shows the major objectives and targets for year 1. All of the main objectives for this period

have been achieved.

1.2 Changes to the Project Plan

There were FOUR main changes to the original project plan, as described below. These were

approved by our NCT Coordinator and recorded in the quarterly reports.

New partner site: Brunel University (1.2.1)

Brunel University has joined the project as a new Partner Site. This has

enabled us to involve a new Site Leader (Dr Darren Griffin) and a PhD student

(Kavita Shah), both of whom bring considerable expertise in computer-aided

learning and assessment.

Changes of personnel at Partner Sites (1.2.2)

At Birmingham University, there have been some changes in the roles of key

supporting personnel which may have implications on this group’s ability to

contribute fully to the OLAAF project. The remaining site team are still

participating in the project, but the future development of assessments at

Birmingham may have to be scaled down slightly from their original plans. The

project will still benefit from our Birmingham colleagues’ expertise in computer-

based assessment built up over several years. Any reduction in activity at

Birmingham has been more than compensated by the activities of the additional

partner site (Brunel University).

Change of personnel: Birkbeck Earth Sciences (1.2.3)

Dr Gerald Roberts, named in the project proposal as the representative for

Earth Sciences at Birkbeck, will have a reduced involvement in the project due

to a change of circumstances. However, we are pleased to report that a

colleague from that department, Dr Karen Hudson-Edwards, will now be

involved. We are currently working with her on designing computer-based

assessments for a new distance-learning Geology course.

External assessment evaluation (1.2.4)

In our Evaluation Strategy (see Section 1.7) we have consolidated the external

assessment evaluations into three stages: August 2003, April 2004, April 2005.

This allowed the first evaluation to take into account both the delivery and the

evaluation of CBAF in the Field Biology module at Birkbeck. It was felt that

three in-depth evaluations taking two days each would serve the project better.


ACR training sessions (1.2.5)

We are developing a programme of workshops based on the Assessment

Construction Resources (ACR), to be held in autumn 2003, rather than in

Summer 2003 as originally planned. It seemed sensible to schedule the

workshops to take place once the Interest Group was properly established, and

the ACR materials had been through an initial pilot phase at Birkbeck College.

Responding to feedback from Interest Group sites, we plan to hold workshops

at convenient locations where participants from a number of different sites can

converge. The team felt that this opportunity for collaborative work would be

more beneficial for team building than holding workshops at each partner site.

1.3 Particular successes in year 1

Recovery from setbacks (1.3.1)

Despite the initial set-backs when our experienced TRIADS Authorer left the

College in Dec 2002 (see 1.4.1), we have nevertheless been successful in

keeping up with our schedule for assessment production. This is due largely to

the outstanding efforts of our new CBAF Support Officer, Ellen McCarthy, who

has quickly built expertise in the relevant software and has established excellent

working practices with members of the project team.

Addition of new Consortium Partner (1.3.2)

The opportunity to invite a new partner institution, Brunel University, to join the

project, has already contributed tangibly to our progress. Through the

partnership, Kavita Shah, a PhD student at Brunel with expertise in online

learning, has been seconded to the OLAAF project on a part-time basis. She

has been working closely with members of the Birkbeck team on assessment

authoring and evaluation issues. Furthermore, we have linked up with an

existing group of computer-aided learning enthusiasts at Brunel through their

Virtual University Project. There is excellent scope for collaboration with

several members of this group, and we anticipate this new partnership will be of

substantial benefit to the project.

Interest Group recruitment (1.3.3)

We have been pleased by the high quality of applicants to the Interest Group

(see 1.4.3), the broad range of pedagogies represented therein, and the

enthusiasm which has led some institutions to exceed their obligations and join

as a group, with representatives from across the institution.


1.4 Major Activities Undertaken in Year 1

Recruitment & Training (1.4.1)

During the early stages of the project the main priority was recruiting staff to key

positions within the core team. The Project Manager was appointed in

November, and joined the team immediately. At the time of writing the bid, a

computer-based assessment authorer was already in post at Birkbeck College,

and planned to continue this critical role within the project team. However,

shortly prior to the commencement of the project, she gave notice to leave the

College, and we were left in the difficult position of having to speedily recruit a

replacement with comparable expertise in this specialist area. After

interviewing several candidates, we were successful in identifying a suitable

person, and our new CBAF Support Officer took up the post in early February.

Training and development needs were ascertained for the core team, and

appropriate solutions were sought. The following is a list of training undertaken

by members of the team during the first year:

• 7 members of the project team (2 at Plymouth, 1 at Brighton, 3 at Birkbeck and 1 at Brunel) have been trained in the use of TRIADS software by the Centre for Interactive Assessment at Derby (CIAD) (April 2003);

• The Project Manager has attended seminars on Designing and Implementing Learning Technology Projects (8 April 2003) and Evaluating and Disseminating Learning Technology Projects (23 May 2003);

• A team-member at Brunel has attended training in Creating Accessible Multimedia, Preston (April 28

th 2003);

• The CBAF Support Officer at Birkbeck has attended training in Accessible Websites, Macromedia (3 April 2003);

• Both the CBAF support officer at Birkbeck and a colleague from Brunel will

attend training in ‘Creating the rich-media experience' with Flash MX (4 September 2003).

• Either the Project Manager or Project Director or both have attended all FDTL4 training events organised by the National Coordination Team.

Partner Site Agreements (1.4.2)

In order to establish clear working and financial arrangements, agreements

were exchanged with the six Partner Sites, detailing the obligations on both

sides, including a clear timeline of activities, a schedule for quarterly payments

(if applicable), and details of intellectual property rights. Visits were made to

each of the Partner Sites during the early months of the project, and there have

been several subsequent opportunities to meet. The site leaders have

organised site teams, in many cases including both technical and academic

staff, and working procedures and targets have been established. They have

also been responsible for raising awareness of the project by contacting

accessibility officers, learning and teaching representatives, and C&IT services

at their respective sites. Software has been distributed to Partner Sites in

accordance with the agreements.

Appendix 1 provides background information on the Consortium Partners.


Formation of Interest Group (1.4.3)

The OLAAF Interest Group has been formed in order to extend the reach of the

project beyond the Consortium by involving colleagues at other institutions

throughout the UK. The Interest Group has added a further element of diversity

to the project participants in terms of geography, subject area and pedagogy. It

also encompasses a range of computer-based assessment (CBA) authoring

tools. This will allow us to pilot our resources to ensure that they are truly

generic, with relevance both to TRIADS-users and to users of other CBA

authoring tools.

Interest Group members (CBA authoring tool shown in parentheses) and the

subject areas represented:

• British School of Osteopathy [BSO] (TRIADS), Osteopathy

• Edgehill College of Higher Education (TRIADS), Geology, Geography

• Keele University (QM PERCEPTION), Life Sciences

• Kingston University (QM PERCEPTION/ BLACKBOARD), Life Sciences, Computing, Chemistry, Mathematics, Pharmaceutical Science, Medicinal Chemistry.

• Manchester Metropolitan (WEB CT), Biological Sciences

• University of Ulster (QM PERCEPTION), Chemistry and Biochemistry

• Warwickshire College (TRIADS), Equine Distance Learning

A summary of the Interest Group is contained in Appendix 2.

Project Web Log (1.4.4)

A project web log has been set up and is being used as a discussion forum to

foster communication between groups on both technical and administrative

issues. (http://mac4.biol.bbk.ac.uk:8080)

CBAF authoring at Partner Sites (1.4.5)

Work on authoring assessments is now in progress at the majority of the

Interest Group and Partner Sites (Birkbeck – Biology and Geology, Birmingham,

Brunel, UWCM, Plymouth, London Metropolitan, Brighton, BSO, Edgehill, Keele

and Warwickshire). Authoring at Kingston will commence when Questionmark

Perception has been installed at the site in September 2003.

Assessment Construction Resources (1.4.6)

Researching and compiling material for the Assessment Construction

Resources (ACR) has been a major focus of activity during the first year of the

project. An early output was the online Context Analysis Tool, a structured

questionnaire which generates an individual assessment specification. It is

envisaged that this tool will be developed as one of several ways of navigating

the ACR, whereby relevant resources are targeted specifically to match the

user’s context. A draft version of the ACR, including the Context Analysis and


Evaluation Guide is available on a private internet site for members of the

project team and Interest Group to utilise, evaluate and contribute to. The ACR

materials were evaluated in August as part of the first external assessment

evaluation, which elicited some extremely useful suggestions. Work toward

developing the ACR in line with this advice is ongoing.

ACR testing (1.4.7)

Use of the ACR has been piloted on the Field Biology module at Birkbeck

College. The ACR has been used to plan and carry out a series of computer-

based assessments for this module, and to formulate an evaluation strategy.

Feedback from this pilot has been used to refin the materials, and also has

provided a working model from which case studies and examples can be drawn

to enhance the ACR and exemplify their use.

Between Aug 2003 and Feb 2004, further testing of the ACR will be undertaken

by other members of the project team and Interest Group who will evaluate the

resources based on their utility in planning and constructing CBAs. Advice will

also be sought from the Project Advisors, including experts on assessment,

evaluation, accessibility and statistical analysis. The ACR will be revised

according to feedback from the pilot and made available publicly on the web site

in May 2004. Further refinements will be made throughout the remainder of the

project.

Steering Group and Project Group Meetings (1.4.8)

A project Steering Group has been established and the first two meetings have

taken place as scheduled. Membership and terms of reference are included in

Appendix 2 as part of the Dissemination Strategy.

The first Project Group Meeting was held at Birkbeck College in the morning of

29th

January 2003. All Partner Sites were represented. A Steering Group

Meeting was held directly thereafter. Apart from the Project Director’s report on

progress and finances, the main agenda items were: Steering Group Terms of

Reference, Project Evaluation, OLAAF Awareness Strategy and Risk Analysis.

The second Project and Steering Group Meetings took place on 17th

September

2003, at Birkbeck College. These meetings were similarly well attended, (only

two absences). The Steering Group considered all aspects of this report in

detail, agreed that it was an accurate representation of the year’s activities and

offered several constructive suggestions regarding the presentation of the

report. The meeting concluded with demonstrations of the ACR and of

Computer Based Assessments developed during the first year.


Dissemination Strategy and activities (1.4.9)

A Dissemination Strategy (Appendix 3) has been developed in collaboration

with the Partner Sites. The OLAAF web site has been live since January 2003

at www.bbk.ac.uk/olaaf. A distinctive project logo has been designed and used

on publicity materials including flyers, posters and letterhead.

This year’s dissemination activities have included:

Articles

• Birkbeck TLT Newsletter 1, Article, Autumn 2002

• Birkbeck TLT Newsletter 2, Article, Spring 2003

• Birkbeck Magazine - 600 word article and photos, Summer 2003 (Appendix 4)

• LTSN Bioscience Web Site, brief article and link to OLAAF web site, April 2003

• LTSN Hospitality Leisure Sport Tourism Web Site– link to OLAAF site, May 2003

• Deliberations Web Site – link to OLAAF site, Spring 2003

• LTSN-01 (Med, Den, Vet) Newsletter - Introduction to OLAAF, Spring 2003

• LTSN Bioscience Bulletin – 50 word box, June 2003

• LTSN Health Science e-bulletin & web site - link to OLAAF site, Summer 2003

• LTSN Health Science Newsletter – 400 word article, Summer 2003

Presentations and workshops

• Birkbeck Cross-College Working Party, Presentation, September 2002

• Birkbeck Working Party on Teaching and Learning, presentation, October 2002

• ‘Innovations in e-learning in HE’ (day-long symposium), Birkbeck, presentation, November 2002

• LTSN Bioscience event: Exploring tools for on-line assessment - a whistle-stop tour for busy academics, presentation, November 2002

• University of Wales College of Medicine – 19th

December 2002, Presentation at E-Learning Day.

• ILT Regional Forum, Poster presentation, December 2002

• UCL Meeting of the Physiological Society, presentation, December 2002

• LTSN Bioscience Contacts Meeting, Poster, Winter 2002

• LTSN Bioscience Assessment Event, presentation, January 2003

• Birkbeck cross-college working party in ICT and Education, presentation, September 2003

• OnLine Assessment and Feedback, A one-day event supported by LTSN Bioscience, LTSN GEES, OLAAF and University of Plymouth, including workshop and demonstrations from OLAAF project, 60+ attendees expected, 21st October 2003

Flyers

• CAA Conference, Loughborough, flyers in all 150 delegate packs, July 2003

• Association for Learning Technology Conference, Sheffield, flyers in all 600 delegate packs, September 2003

Quarterly Newsletters

• 1st OLAAF Newsletter (by email to approximately 250 recipients) May 2003

• 2nd

OLAAF Newsletter (by email to approximately 300 recipients) Sep 2003

Awareness Workshops

Several awareness workshops and presentations have taken place at

Partner Sites to advertise OLAAF to potential end-users and establish links


with Learning & Teaching networks:

• Birkbeck College – October 2002, Presentation to Working Party on Teaching and Learning; November 2002, Presentation at Innovations in E-Learning day; September 2002, September 2003, Presentation to Cross-college working party on ICT and Education

• British School of Osteopathy - 14th May 2003, Presentation to 8 staff, including C&IT and Principal

• Brunel University – 17th July 2003, Presentation to Virtual University Project group, 12 staff

• Kingston University - 5th June 2003, Workshop, 12 staff attended including academic staff, learning and teaching and C&IT

• University of Brighton – 16th July 2003, Workshop session as part of University Learning and Teaching Conference, 12 staff attended including academic staff, learning and teaching and C&IT

• University of Plymouth – 7th

March 2003, Presentation to 6 staff; 21st October 2003, Workshop and demonstrations as part of Celebration of Learning and Teaching (Appendix 5)

1.5 Collaborations and Links

Other FDTL projects (1.5.1)

We have collaborated with other FDTL projects in exchanging ideas and

refining evaluation and dissemination plans as part of the NCT workshops and

the Assessment Project Network (see 1.5.3). We keep in touch with several

projects whose work overlaps with ours, for example those focusing on

feedback and formative assessment, and we intend to seek further collaboration

in the future. In particular, there is much common ground between our project

and the project ‘Improving the Effectiveness of Formative Assessment in

Science Teaching’ (FAST) (FDTL 74/02). We are members of the LTSN

Special Interest Group established by FAST, and look forward to working

closely with the FAST group.

Bioscience and other LTSN (1.5.2)

We have collaborated closely with the Learning and Teaching Support Network

(LTSN) Subject Centre for Bioscience throughout the bid writing stage, and from

the start of the project. Heather Sears, a representative of the LTSN Bioscience

Subject Centre, is a valued member of our Steering Group. We have attended

all relevant events organised by the subject centre, including making

presentations at several events, and are currently involved in organising a joint

LTSN Bioscience and GEES event on OnLine Assessment and Feedback at

Plymouth on 21 October 2003. We have contributed articles to web sites,

newsletters and bulletins in the following related Subject Centres: LTSN Health

Science, LTSN-01 (Med, Den, Vet), LTSN Bioscience and LTSN Hospitality

Leisure Sport Tourism.


Assessment Project Network (1.5.3)

We are an active member of the Assessment Project Network, established by

the National Coordination Team to support collaboration between FDTL projects

with assessment-related themes, and have attended the meetings held in

October 2002 and May 2003. Our literature survey is featured on the APN

website and we have submitted a proposal for a chapter on the OLAAF project

in a book to be produced with other members of the network. We look forward

to contributing to joint symposia at the two conferences identified by the group.

National Teaching Fellows (1.5.4)

We are fortunate to have Dr Les Jervis, a National Teaching Fellow, engaged in

the project as Site Leader for Plymouth. His breadth of experience, knowledge

of the UK HE sector, and wide array of contacts have already served to raise

the profile of the OLAAF project both within the UK and abroad.

Other bodies (1.5.5)

Each Site Leader has been responsible for forming links with their local

Disability Officers, learning and teaching centres, and central C&IT support

staff. Active participation in other relevant institutional networks is also

widespread. For example the site leader for Brighton is also the nominated

Learning Technology Support Officer for the Institute of Nursing and Midwifery;

at Brunel, project participants are active members of the Virtual University

Project, a campus-wide group of practitioners of computer-assisted learning and

assessment; and at Birkbeck project representatives sit on both the Cross-

College Working Party in ICT and Education and the Working Party on

Teaching and Learning. Participation in such groups affords opportunities to

raise the profile of the project through key networks and will ultimately facilitate

potential input into policy-making within the institutions.


1.6 Outputs from Year 1 Activities

OLAAF website (1.6.1)

The site, designed to conform to accessibility guidelines, went live in January

2003. It is a regularly updated portal containing information about the project,

links to materials we have produced, and links to other useful sites. It has

already invited international interest; the log shows visitors from the United

States, The Netherlands, Hong Kong, Australia, Norway, Ireland and China.

http://www.bbk.ac.uk.olaaf

Assessment Construction Resources (1.6.2)

Preparation of these materials has been a major focus of the Birkbeck team’s

activities so far. The ACR is currently presented as a static web site, containing

resources focusing on the practical tasks encountered in planning, authoring

and reviewing a computer-based assessment. The stages are: 1. Determine the

purpose of the assessment; 2. Analyse the context; 3. Construct test items; 4.

Plan the format of the assessment; 5. Construct the assessment; 6. Review the

assessment prior to delivery; 7. Address technical issues; 8. Evaluate the

assessment.

Each section provides a synopsis of the challenges faced at the particular stage,

with advice targeted to particularly challenging or critical sub-tasks within the

respective stages. This advice draws on the collective experience of the OLAAF

team and, as far as possible, is supported by current research findings.

Supplemental material is provided in the form of annotated reading lists and

hyperlinks to external sources. Good practice is reinforced by the provision of

editable proformas – templates and checklists that encourage staff to work

systematically through each stage of assessment construction. Examples of

proformas completed by the project team illustrate the process.

Additional planning documents, as well as case studies and example questions,

will be added to the ACR as they are generated by the Project Team and

OLAAF Interest Group. These additions will expand the range of subject areas

and pedagogical approaches exemplified in this resource.

Computer-Based Assessments (1.6.3)

A series of assessments for the Field Biology module at Birkbeck have been

designed, authored, delivered and evaluated in accordance with the ACR. At

the same time, this pilot implementation has informed the development of the

ACR, providing valuable feedback on the practicality of the resources, and

generating numerous examples. A number of other new assessments –

presently in the design stage – will be running in the Autumn term 2003 and/or

Spring term 2004 at several Partner Sites.


Flyers and posters (1.6.4)

Pamphlets giving details of the project, including how to join the OLAAF Interest

Group, have been produced and circulated at key conferences and events.

Appendix 6 is an example.

Posters to advertise the project have been produced and distributed to the Site

Leaders for display at their institutions.

1.7 Evaluation of the Work

Evaluation Strategy (1.7.1)

The Evaluation Strategy for the project is attached as Appendix 8. This was

developed from discussions at the first Steering Group meeting, and refined in

consultation with our external evaluators, Dr Joanna Bull (Eduology) and Dr

David O’Hare (Centre for Interactive Assessment Development, University of

Derby).

The first External Assessment Evaluation visit took place on 6th

August 2003.

This evaluation focused on project outputs including the draft Assessment

Construction Resources, and specific Computer-Based Assessments; the

resulting report is attached as Appendix 7. This evaluation was found to be

extremely valuable, and several recommendations were made which will inform

the future development of the outputs.

As we are a three-year project, our first External Project Evaluation will take

place in May 2004. This will aim to answer three questions:

1. Is the project moving forward at a reasonable pace towards the

achievement of its agreed outcomes?

2. Is the project being effectively managed and is the project team

functioning productively?

3. Are the outcomes of the OLAAF project being effectively disseminated?


1.8 Learning from Activities and Progress made during Year 1

Effective partnerships (1.8.1)

Despite the administrative effort involved, we have found it very useful to

elaborate explicit, written partner agreements with both our Consortium

Partners and Interest Group members. These detailed agreements make clear

the responsibilities and expectations on both sides, both financially and in terms

of expected activities and outcomes during the project. The agreements have

already proved a useful reference point in discussions, and it is hoped that by

having such clear, written guidelines from the project’s inception, we will avoid

many potential misunderstandings in the future.

Communication was quickly identified as the key to forming effective

partnerships within the project Consortium. A communication strategy was

drawn up after the first project team meeting, in which several channels of

communication were identified including face-to-face meetings, email,

telephone, newsletters, and a project web log. Despite the geographical spread

of the Consortium members, effective communication has been maintained

through a combination of these methods. We have also maximised

opportunities for contact, by putting time aside for project business at unrelated

events which members of the team happen to be attending.

Embedding changes (1.8.2)

We have identified several conditions under which computer-based assessment

is likely to continue in the partner institutions beyond the life of the project, and

the accompanying support and funding. These conditions are commensurate

with the aims of the project and progress towards their achievement has already

been made.

1. Building networks of support - opportunities for collaboration, exchange

of ideas, peer review.

2. Providing training based on highly practical resources which encourage the adoption of good models of working practice which should outlast the project and equip individuals and teams to continue developing CBA in the future.

3. Developing relevant skills and expertise in relevant CBA authoring tools and providing software and ongoing support.

4. Involvement of existing support centres and institutional networks.

5. Integration of local (i.e. Departmental) plans with Institutional Learning and Teaching Strategy and e-Learning Strategy.

Responding to HEFCE priorities (1.8.3)

The project emphasises the importance of well-constructed formative

assessment that helps learners to understand their developing strengths and

makes visible the value added through their studies (a priority highlighted in the

HEFCE Strategic Plan, 2003-2008, under ‘Excellence in Learning and


Teaching’). Our commitment to maintaining principles of good assessment

practice and transferring these to a computer-based environment also

embraces the challenge identified in the HEFCE Strategic Plan, (under ‘Meeting

customer and stakeholder needs’) of maintaining the quality of experience

offered to the student through new media.

Contributing to student success (1.8.4)

A range of approaches to assessment is encompassed by the project at various

sites, but there is a general emphasis on diagnostic and formative use of CBA,

and on the use of targeted feedback to promote student learning. There is

abundant evidence to suggest that regular, formative assessment, administered

at intervals from the beginning of the course, can have a positive impact on

student retention and achievement on the course. Regular assessment also

provides valuable feedback to staff, allowing them to take early remedial action

where necessary.

Staff at all partner and Interest Group institutions are currently engaged in

remodelling the assessment strategies of their respective modules to include

elements of computer-based assessment with feedback. Through participation

in the OLAAF project, and adherence to the principles for good assessment

design promoted in the ACR, it is hoped that evaluation of these interventions

will indicate a positive impact on student learning and retention. At the very

least, by coordinating such widespread reflective practice in diverse subject

areas, the project will be in a position to advance the debate on how CBA can

most effectively enhance student learning.


2. Year 2 Plan

2.1 Main Objectives/Targets for Year 2

A Project Timeline is attached as Appendix 9. Key milestones for year 2 include:

• Authoring of CBAF at sites (first iteration), Nov 2003

• National conference OLAAF1 and Report, Jan 2004

• Assessment Construction Resources piloted within OLAAF Network, Feb 2004

• Draft Assessment Resources available publicly on web site for use and evaluation, May

2004

The Dissemination (Appendix 3) and Evaluation (Appendix 8) Strategies introduced in the year 1

report will be carried forward in year 2.

2.2 Changes to Year 2 Plan

We have added more targets to the plan, having analysed the tasks and responsibilities for year two

in more detail. However, there are no changes to the original targets.

2.3 Help Needed for Year 2

Help needed from the NCT (2.3.1)

It would be useful to have advice on the format that our external project

evaluation report should take. Access to good examples from previous projects

would be particularly helpful. Ideally we would like this information prior to the

first external project evaluation in May 2004.

We have found the Assessment Project Network, set up by the National Co-

ordination Team, to be a valuable source of support and collaboration, offering

multiple opportunities for dissemination, including international conference

symposia and contribution to a book. We see the network as a major channel

of support from the NCT and the project will certainly benefit if this assistance is

ongoing.


3. Financial Statement

3.1 Expenditure Statement for Year 1

The Expenditure Statement for Year 1 is shown in Annex B; an alternative view of this information is

found in Appendix 10.

The total award from HEFCE over the 3-year course of the project is £249,956. It should be

emphasised that this total sum is still required to support the project over this period and, further, that

we presently anticipate no substantive changes to the total amounts allocated to the respective

named categories.

Annex B shows an underspend of £17,541 for Year 1 (against income from HEFCE of £96,090).

This has arisen almost entirely for the following two reasons: 1) the rate of expenditure in the Staff

category has changed; 2) the rate of expenditure in the Dissemination category has changed.

Please note, again, that over the 3-years of the project (as illustrated in Appendix 10) the total

expenditure is exactly as originally planned with only very minor changes between categories.

Additional explanation follows.

Changes to rate of expenditure in the Staff category (3.1.1)

This change has arisen owing almost entirely to the later than anticipated starts

of the two key members of staff supported by these funds: the Project Manager

and the CBAF Support Officer (Authorware Programmer).

The Project Manager position was budgeted to start from 1 October 2002, but

this member of staff actually started approximately 1 month later than this,

reducing the staff expenditure for Year 1 in this sub-category.

The CBAF Support Officer situation is more complex, but in short, expenditure

in this sub-category started later than anticipated, accounting for most of the

remaining Staff underspend in Year 1. (We had budgeted to start this

expenditure from 1 Jan 2003.) As explained elsewhere report (1.4.1), we had

anticipated contracting for this work an experienced person who was already in-

post at Birkbeck. Due to unforeseeable circumstances, she left the College

shortly before the start of the project. There was an inevitable delay in

recruiting a new person for this position (she started in late Feb 2003), and the

delay alone accounts for some of the underspend in Year 1. It turned out that

the new appointee was considerably less experienced than the original post-

holder, and so attracted a lower rate of pay. We were able to re-purpose the

savings, funding staff time purchase at a new Consortium Partner Site, Brunel

University (which has produced significant benefits to the project; see 1.2.1 and

1.3.2). (This change was discussed with, and approved by, our NCT Co-


ordinator, Paul Martin.) Payments to Brunel commenced only from April 2003—

a delay which accounts for most of the remainder of the Staff underspend in

Year 1.

Minor changes to the schedules of the staff time purchase payments to other

Consortium Partner Sites (but not to the originally planned total amounts)

account for the remainder of the Staff underspend in Year 1.

In all cases, funds unspent in Year 1 will be spent by the end of the project (see

Appendix 10). In effect, most of the funds unspent in Year 1 will now “spill” into

Year 3.

Changes to rate of expenditure in the Dissemination category (3.1.2)

Minor changes in the rate of expenditure—effectively a “phase shift” of about 4

to 6 months—account for the majority of the underspend in Year 1. About 60%

of the underspend will be accounted for in the first half of Year 2; the remainder

will be carried to Year 3. Appendix 10 clarifies this point.

3.2 Anticipated Expenditure for Year 2

Anticipated Expenditure for Year 2 is shown in Annex B; an alternative view of this information is

found in Appendix 10.

The totals in Annex B reveal a small (£6829) overspend relative to the originally budgeted sums. As

explained in Section 3.1, this is accounted for by changes in the rate of expenditure in Year 1. The

entire Year 1 underspend is recovered by the end of the project, as revealed in Appendix 10.


Declaration

I certify on behalf of the lead institution that the attached progress and financial report, and the information

contained therein is correct, and that the funds allocated have been applied to the purpose for which they

were made available.

Project Name:

Project Number:

Head of Institution or Nominated Deputy

Name (print)

Signed Date

Project Director

Name (print)

Signed Date


Annex A

Fund for the Development of Teaching and Learning (FDTL)

Activities and Progress

Project no. and name: 5/02 OLAAF: OnLine Assessment and Feedback

Reporting Period: 1 October 2002 - 30 September 2003

Preparatory Phase (Nov – Dec 2002)

Objective/target/outcome/ milestone

Original date to be achieved

Output to demonstrate objective/target/outcome/milestone achieved

Date achieved

If applicable, reasons for slippage in date achieved

If applicable, reasons for change to original objectives

Appoint Project Manager Nov 2002 Project manager in post Nov 2002

Finalise agreements between Partner

Sites

Jan 2003 Agreements signed by both parties and

held on record

Jan 2003

Refine project plan in line with feedback

from exploratory contacts

Dec 2002 Project plan expanded to include more

detailed breakdown of activities and target

dates

Dec 2002

Appoint CBAF support officer Feb 2002 CBAF support officer in post Feb 2002

Present at LTSN and Other Events Ongoing Presentations made at several events Ongoing

Web site live; follows W3C guidelines Dec 2002 www.bbk.ac.uk/olaaf Dec 2002

Organise web-based discussion forum Dec 2002 http://mac4.biol.bbk.ac.uk:8080/ Dec 2002

Identify Interest Group participants and

survey their requirements

Dec 2002 Participants identified and application forms

distributed

Dec 2002

Plan interactions of Partner Sites Dec 2002 Initial meetings held with Partner Sites Dec 2002


Phase 1 (Jan – Sep 2003)




Date achieved



Refinement of Partner Site objectives Feb 2003 Objectives refined and timescales agreed Feb 2003

Refinement of Interest Group objectives Feb 2003 Interest Group agreement and

responsibilities finalised

Feb 2003

Preparation of Assessment Benchmarks Apr 2003 Context Analysis Tool to assist

benchmarking

Apr 2003

Accessibility audit at all sites; contact

Disability Officers

Mar 2003 Accessibility audit information collected

from main sites

Mar 2003

Production of draft CBAF Toolkit for use by

Partner Sites

Apr 2003 Draft version of Assessment Resources

produced and piloted at Birkbeck

Apr 2003

Production of Evaluation Guide for use by

Partner Sites

Apr 2003 Draft version of Evaluation Guide produced

and piloted at Birkbeck

Apr 2003

ACR piloted at BBK Sep 2003 Assessment Resources piloted on Field

Biology course

Aug 2003

1st Annual Progress Report to NCT 30 Sep 2003 Refined by Steering Group 17th September

and Submitted

30 Sep 2003

Site Leaders establish Site Groups; web

fora initiated, facilitated by Site Leaders

Apr 2003 Site Groups established, web log in use Apr 2003

Distribute software and funding to Target

Sites

Jul 2003 Software distributed, and quarterly funding

schedules agreed

Jul 2003

Training sessions at Partner Sites to

embed ACR

June 2003 Planned to take place in autumn term Dec 2003 Approved decision to

reschedule until autumn term

Awareness workshops at Partner and

Target Sites to advertise OLAAF to

potential end-users and establish links with

Learning & Teaching networks

Jun 2003 Workshops have been held at several sites

and more are planned

Jun 2003


1st External Assessment Evaluation of

ACR and CBAFs

April 2003 External Assessment Evaluation Report Aug 2003 Approved decision to

consolidate 4 shorter

evaluations into 3 more in depth

ones

1st Steering Group Meeting Jan 2003 January 29th 2003 Jan 2003

2nd Steering Group Meeting Sep 2003 September 17th 2003 Sep 2003

Phase 2 (May 2003 – Feb 2005)




Date achieved



Authoring of CBAF at sites (for semesters

1 & 2, first iteration)

Nov 2003 Authoring of CBAF underway, with central

support from Birkbeck

Develop dissemination and publication

strategy

Sep 2003 Dissemination Strategy July 2003

Annex B


Fund for the Development of Teaching and Learning (FDTL) – Expenditure Statement Project no. and name: 5/02 OLAAF: OnLine Assessment and Feedback

Actual expenditure for year one 1 October 2002 - 30 September 2003

Income for year one

Allocation from the HEFCE: £96,090

Total for year one: £96,090*

Total budget for period

Total expenditure Variance Underspend – Overspend +

Reasons for variance and action taken/intended

Staff* 63990 51969 -12021 The rate of expenditure has changed owing to delays in assembling the full complement of personnel and changes in the schedule of payments to some partners. All of the unspent funds will be used for the originally intended purposes by project end. See Section 3 and Appendix 10 for further explanation.

Travel & subsistence 2700 2370 -330 Underspend will be used in year 2 owing to shift of some activities to year 2.

Dissemination 14050 8530 -5520 The rate of expenditure has changed. Some expenditure originally anticipated for year 1 will now occur in years 2 and 3. See Section 3 and Appendix 10 for further explanation.

Equipment 9700 9635 -65

Evaluation 700 840 +140 The evaluator required slightly more time than anticipated to complete the external assessment evaluation.

Other costs (please detail)1 4950 5205 +255 The cost of some essential software had increased by the time of purchase.

Total 96090 78549 -17541 See above. To be carried forward into year 2.

*Birkbeck College paid 45% of the total cost of the Project Manager (this is additional to the budgeted sum shown above).

1 ”Other costs” are primarily those incurred in purchase of software, but also include difficult to categorise computer accessories.

Annex B


Fund for the Development of Teaching and Learning (FDTL) – Budget statement Project no. and name: 5/02 OLAAF: OnLine Assessment and Feedback

Anticipated expenditure for year two (1 October 2003 - 30 September 2004)

Income for year two

Underspend to be carried over from year one £17,541

Allocation from the HEFCE £96,553

Total for year two £114,094

Anticipated expenditure

Variance +/– from original budget

Notes regarding allocation of funds

Staff* 77312 +1159 See Year 1 Expenditure; also Section 3 text and Appendix 10. Underspend of £1275 over the 3 years.

Travel & subsistence 6250 +450 Required to support activities in year 2. A £500 overspend across the 3 years is anticipated. This will easily be recovered from underspends other categories.

Dissemination 13020 +3720 Expenditure originally accounted for as “year 1” will occur in year 2 (and in year 3). No change in total expenditure over the 3 years. See Section 3 text and Appendix 10.

Equipment 0 0

Evaluation 3150 +350 Approximately 1 day of extra fee to be paid to Project Evaluator. Overspend of ca. £140 change over the 3 years. See Appendix 10.

Other costs (please detail)1 3650 +1150 Some software costs have increased, and some software upgrades will be needed in year 2. No

change in total expenditure over the 3-year period.

Total 103382 +6829 See above. To be carried forward into year 3.

*Birkbeck College will pay at least 45% of the total cost of the Project Manager (this is additional to the budgeted sum shown above).

1 ”Other costs” are primarily those incurred in purchase of software, but also include difficult to categorise computer accessories.

annual report 30 september 2003 - birkbeck, university of ... · qmark question mark perception...

Documents