evie project final report - university of leeds · jisc development programmes project document...

JISC DEVELOPMENT PROGRAMMES

Project Document Cover Sheet

EVIE PROJECT FINAL REPORT

Project Project Acronym EVIE Project ID Project Title Embedding a VRE in an Institutional Environment Start Date 1/11/2004 End Date 28/2/2007 Lead Institution University of Leeds Project Director Formerly Tracey Stanley, now Brian Clifford Project Manager & contact details

Formerly Derek Sergeant, now Peter Balmforth, [email protected], 0113 3431769

Partner Institutions The British Library Project Web URL http://www.leeds.ac.uk/evie/ Programme Name (and number)

VRE 1

Programme Manager Frederique Van Till

Document Document Title Final Report Reporting Period Author(s) & project role Tracey Stanley, EVIE Project Director

Peter Balmforth, EVIE Project Manager Bill Jupp and Gemma Kitchen, EVIE Project Officers

Date 30/5/2007 Filename evie-final-report_v1.1.doc URL Access Project and JISC internal General dissemination

Document History Version Date Comments

1.1 30/5/2007

of 30

mailto:[email protected]

EVIE – Final Report – Version 1.1– May 07

Table of Contents

......................................................................................Error! Bookmark not defined. EVIE Project Final Report ................................................................Error! Bookmark not defined. Table of Contents.............................................................................................................................1 Acknowledgements..........................................................................................................................3 Executive Summary.........................................................................................................................4 Background......................................................................................................................................5 Aims and Objectives ........................................................................................................................5 Methodology ....................................................................................................................................6 Implementation ................................................................................................................................7 Outputs and Results ......................................................................................................................11

SRU Search Channels and CREE Jafer Channel.............................................................................20 Personal Bookmarks .........................................................................................................................20 Funding Workflow..............................................................................................................................21 Funding Alerts RSS...........................................................................................................................21 Group Select Channel .......................................................................................................................21 Group Bookmarks..............................................................................................................................21 Collaboration .....................................................................................................................................21

Outcomes.......................................................................................................................................21 Dissemination ................................................................................................................................24 Conclusions and Lessons Learned................................................................................................25 Implications and Sustainability.......................................................................................................28 Recommendations .........................................................................................................................29

of 30


Acknowledgements This project was funded under the JISC Virtual Research Environments Programme. The Project would like to acknowledge contributions from the Project partner: the British Library, in particular from Stephen Andrews, Adam Farquhar and Paul Wheatley. The Project would also like to acknowledge contributions from all staff who have worked on the Project, including Derek Sergeant, who was Project Manager until Oct 2006.

of 30


Executive Summary The EVIE Project has developed and tested a prototype Virtual Research Environment (VRE) using a portal framework. The portal framework provides a seamless, web-based interface to a range of institutional systems and services. A wide range of tools and services have been integrated within the portal framework, including:

• Access to library services. • Collaborative working tools. • Funding alerts and related information. • Support for obtaining funding. • Tools for capturing research outputs.

User requirements were gathered from research staff, and were used to prioritise developments. Development has followed an overall ‘research lifecycle’ identified during the course of the Project. The Project has used a number of integration mechanisms, including single-sign-on (SSO) and channel and portlet development in order to bring these disparate resources together in a coherent way within a single framework. The Project has also investigated a number of issues around the development of VREs. These include an assessment of the digital preservation requirements for outputs generated within the VRE, or available through the systems integrated with the VRE. This work was undertaken in collaboration with the British Library, and leveraged their considerable expertise in digital preservation work. A recommendation was made for an external preservation services approach. The Project has also assessed the requirements for a supporting taxonomy, which would enable the tagging or classification of objects created within the VRE. The recommendation made here is that the developments in this area are still too immature to facilitate widespread adoption and use in a VRE environment. The Project has also explored a range of advanced resource discovery developments – including use of SRU/SRW to provide a resource discovery portlet within a VRE, and advanced document visualisation techniques to provide a visual approach to the presentation of search results. The conclusion is that SRU is not yet stable enough to facilitate widespread adoption though the changes that would be required to make it more viable are small. The prototype system has been extensively evaluated and tested with research staff across a range of disciplines within the institution. EVIE has demonstrated that researchers would welcome a seamless environment where they can access the tools that they need to support their research activities, but that this environment must be sufficiently flexible to enable them to integrate their own preferred tools alongside those supported by the institution.

of 30


Background Researchers in all disciplines are increasingly expecting to be able to undertake a variety of research-associated tasks online. These range from collaborative activities with colleagues around the globe through to information-seeking activities in an electronic library environment. Many of the tools which enable these activities to take place are already available within the local IT infrastructure. However, in many cases, the tools are provided through discrete, bespoke interfaces with few links between them. Researchers face a number of challenges in this environment, including multiple methods of authentication and authorisation, finding information and sharing information between applications. The EVIE Project intended to address these issues by testing the integration and deployment of key existing software components within a portal framework. A portal is intended to provide a seamless, web-based interface to a range of university systems and services. The key benefit of a portal framework is to bring together disparate resources and systems into a single environment, so that end-users can utilise these tools in an integrated fashion, thus aiding efficiency and effectiveness, and improving the overall user experience. The University of Leeds is using the Luminis Portal product (from Sungard HE) as its framework for a portal. Luminis is built upon open standards and v4 is both JSR-168 and Shibboleth compliant, based around uPortal, elements of the Sun JES stack and, optionally, Documentum for content management. Luminis is widely adopted in the UK and internationally. Currently the University has a live Student Portal based on the Luminis platform. The University of Leeds has also developed and deployed a Virtual Research Environment, known as the Virtual Knowledge Park (VKP); although this is now in the process of being withdrawn by the University. The VKP supported 11 large-scale research programmes which included national research centres, regional research networks, technology institutes and European research consortia within which there were over 200 active research networks. The University has also developed an in-house VLE – the open source Bodington system, and has extensive electronic library services. The EVIE Project therefore intended to build on this expertise by integrating the VRE, VLE, e-library and other selected corporate resources within an overarching portal framework. The integration was expected to deliver a range of benefits to researchers, including widening awareness of the tools available, increasing familiarity, uptake and use of tools, aiding ease of use, and improving the ability for researchers to share information across disparate systems.

Aims and Objectives The aims and objectives of the EVIE Project, as stated in the original project plan, were to:

• Establish a prototype VRE infrastructure based on open standards and existing software components to support a test group of researchers; including users from the School of Medicine, School of Geography, and researchers using the White Rose Grid.

• Provide a set of additional resources and services through this environment, including facilities for enhanced search and retrieval.

• Deliver simplified-sign-on functionality to enable seamless integration between the identified platforms.

• Provide a set of user validated recommendations identifying effective, scaleable and reusable mechanisms for construction of intuitive search and retrieval tools within this environment.

• Provision of enhanced resource discovery mechanisms with document visualisation techniques available to indicate relevance.

• Develop best practice for the use of taxonomy within a VRE. • Provide support for search and retrieval mechanisms across disparate information resources

within a VRE. • Identify long-term options and requirements for digital preservation in a VRE.

of 30


• Identify requirements for data integration to provide a seamless flow of information between systems integrated through the environment.

All of these objectives have been met by the EVIE Project. Additional objectives were added at various stages of the Project. These included:

• Investigation of integration of collaborative tools into a VRE framework (agreed with JISC Sept 05).

• Improve seamless access for researchers through provision of integrated access to the University repository (ePrints) system and University Publications Database (agreed with JISC Sept 05).

• Undertake a post-implementation review (agreed with JISC Sept 2006). The integration with ePrints was not implemented. An investigation of the options here highlighted that ePrints version 3 would provide the functionality required to undertake specific aspects of this work. ePrints 3 did not become available for implementation during the timeframe of EVIE, so it was not possible to pursue this work further.

Methodology The EVIE Project has been carried out over three key project phases:

1. Research and user requirements gathering (9 months) 2. Implementation (15 months) 3. Testing, evaluation and dissemination (4 months)

The Project has been developed as a series of inter-linked work-packages, broadly covering these key phases. Phase 1: Research and user requirements gathering The research phase included a 6–month user requirements gathering exercise, which incorporated a series of focus groups, semi-structured interviews with key stakeholders, and an online questionnaire. This work was undertaken jointly by the University of Leeds and the British Library. The research phase also included a scoping study on systems integration requirements, which resulted in a systems integration route-map, and a full technical and functional specification. These activities drew closely on the outcomes of the user requirements exercise, which provided us with a valuable tool for prioritising development requirements. The activity also drew extensively on previous work undertaken at the University for the VKP project, including an evaluation of collaborative working and synchronisation tools which had been produced for the VKPi. As part of this activity, the Project also produced a visual design criteria, accessibility requirements and standards requirements. A demonstrator system was also produced and tested with key stakeholders, leading to sign-off by the Project Steering Group. This is available on the EVIE websiteii. The research phase also included scoping exercises for preservation requirements and taxonomy requirements. Phase 2: Implementation The implementation phase drew closely on the outputs of the user requirements gathering exercise, and on the resulting functional and technical specification. The specification was further developed as a channel development plan – which broke the development work down into key phases for delivery. Phase 1 was a fairly small phase for delivery, aimed at being easily achieved, and during which valuable skills and lessons would be learned which could be applied to the later phases. The early phase was intended to be quickly completed, in order to allow for iterative testing and improvements and to maintain project momentum. Further implementation phases allowed for more complex

of 30


developments to be undertaken. The full channel development plan is available on the EVIE websiteiii. Phase 3: Testing, evaluation and dissemination The portal components were installed on the live Portal system at the University in order to allow for a full user testing phase to take place. A range of user testing and evaluation activities were undertaken; these included:

• Focus groups, which included a demonstration of the live system, and discussion on the various components.

• One-to-one interviews with senior University stakeholders, in order to gauge attitudes and potential for buy-in to the system.

• Extended use by a small cohort of volunteers, including academic and research staff, and research postgraduates, and evaluation of this use using a paper-based questionnaire, a series of one-to-one interviews and a focus group.

• Development of a set of case studies based around the one-to-one interviews. The results of these were fed into the report for the post-implementation review.

Implementation Most successful portals in higher education have been developed using a strong user-centred approach. The EVIE Project wanted to ensure that the prototype VRE implemented features that met a real need amongst researchers. For this reason, the over-arching methodology was to involve researchers closely in the system scoping and specification from the outset of the Project. Our user requirements gathering exercise focused on users from our selected pilot areas in the School of Geography, School of Medicine, and researchers using the White Rose Grid. Firstly, a series of semi-formal one-to-one interviews was undertaken with a small number of researchers. These initial interviews were used to gain an understanding of the typical research processes undertaken by researchers in higher education. Researchers were asked about the different systems used to support their research. A second set of more structured interviews were then carried out with a view to determining whether a general picture of research processes could be drawn up. The intention was to identify areas of consistency as well as divergence from the patterns already described. Given the Project timescales, it was possible only to conduct a very limited number of interviews. However, it quickly became apparent that there was a high degree of uniformity emerging in the responses that were received – which indicated that some of the key requirements and main phases for research activity were being identified. The Project then undertook a series of focus groups. The first focus group was structured to check the validity of the ideas emerging from the one-to-one interviews. A second focus group presented an idea of what a virtual research environment might look like and asked for feedback on this. It also set some challenges, for example attendees were asked to identify up to 25 systems that a researcher might use on a daily basis. The focus groups were deliberately targeted at the pilot user group – an obvious draw-back of this approach might be that the requirements of this group were sufficiently unique for them not to adequately represent the wider University community. It was agreed to overcome this limitation by undertaking a broader survey of researchers’ requirements across the entire University. The outputs from the one-to-one interviews and the focus groups were then fed into an online survey which was made available to all staff within the University of Leeds. The online survey provided an opportunity to elicit views on requirements from the wider research community. The survey had tightly defined goals: prioritisation of user requirements, testing the completeness of the requirements already identified and confirmation that these requirements were applicable to researchers across a wide range of disciplines.

of 30


The questionnaire was widely promoted using research support mailing lists, the campus intranet and direct targeting of certain key potential users. 121 staff completed the online survey, and responses by facultyiv are shown as follows:

Faculty Number of responses Arts 10 Business 5 Biological Sciences 14 Education, Social Science and Law 23 Environment 15 Engineering 16 Mathematical & Physical Sciences 2 Medicine and Health 22 Performance, Visual Arts and Communications 10 Joint Honours 0 Other 3 Faculty not stated 1 Total responses 121

Questionnaire returns were then analysed, along with the other outputs, and a report on user requirements produced. Following the user requirements analysis, a technical and functional specification was also produced. This was based on the priorities identified in the user requirements analysis. Some work was undertaken to map the requirements against key University systems and services. The following systems and services were identified as being relevant to EVIE, and to the priorities identified in the user requirements analysis:

• Luminis – the University had selected the Luminis platform from SungardHE (formerly SCT) as it’s enterprise platform for portal development; focusing initially on rolling out a student portal to undergraduate users.

• Active Directory – a university-wide directory service for authentication. Identified as a major component in achieving single-sign-on to a variety of institutional systems via the Portal.

• Management Information Systems – the University uses SAP R/3 to manage finance, payroll information. SCT Banner is used to manage student information.

• Outlook/Exchange – the University runs a robust and resilient Exchange server infrastructure which includes extensive use of both email and calendar facilities.

• Bodington VLE – the University has developed its own open source VLE (Bodington) which is in widespread use.

• Library systems – the University uses the Innovative Interfaces Inc Library Management System, including tools for federated search and openURL access to e-resources.

• Research Management Systems Programme – the University has developed a significant research systems landscape, including a University Publications Database (ULPD), extensive use of SAP - including for grants management, development of a Full Economic Costing tool (COSTA) and some pilot use of SAP records management.

• Collaborative tools – the University uses the locally-developed Virtual Knowledge Park (VKP) system for internal and external collaboration

• Research outputs – the University has an institutional repository (White Rose) based on the eprints platform, and shared with the Universities of Sheffield and York.

• Funding information – the University has a subscription to the Research Research funding information service, which is widely used by researchers.

At this stage in the Project, a number of changes emerged in the University systems landscape. Most importantly, the University announced the results of a review that had been undertaken into the future of virtual environments on campus. The outcome of the review was that the VKP system would be phased out by summer 2007, potentially to be replaced by an alternative set of collaborative tools if sufficient demand was identified. This had a major impact on the Project, and it was agreed by the

of 30


Project Steering Group that we should no longer focus on the integration of VKP functionality as a key component of the system, but should instead look at the potential for integration of alternative collaborative tools. This change was approved through JISC, and resources were re-allocated to enable the Project to appoint an additional project officer to do development work on other collaborative tools integration. The University also announced that the Bodington system would be phased out by summer 2008, to be replaced by a new VLE system to be procured by a tender exercise. The Project Steering Group agreed that, given the longer shelf life for Bodington, and the centrality of a VLE to the University, work should still be undertaken to integrate Bodington into the portal environment. A software development plan was created, based on requirements identifiedv. The software development plan also included a general portal channel-building strategy which identified the following issues:

• Luminis would be used to provide the framework for the Research Portal. Luminis was available in v3 at that time, which did not support JSR 168 portlets. Luminis v 4 is JSR 168 compliant, but although this was originally scheduled to be available during the lifespan of the EVIE Project, development timescales were later put back to Q1 2007. The decision was therefore taken to utilise the IChannel functionality available in Luminis v3. These channels are also supported by uPortal, so it was considered that it would be possible for the channels to be re-used by a wider portal community.

• A number of development constraints were identified which were external to the EVIE Project. For example, in relation to federated searching of library resources, the Library had selected a federated search tool (III Metafind) but had not yet fully implemented or rolled out this tool to users. The timescale for this work was outside of the control of the EVIE Team. Because of these dependencies, a phased approach to channel development was agreed.

Early developments therefore included a small number of ‘quick wins’ which were intended to assist the Project team in developing skills and confidence, as well as providing the Project with some early demonstrable outcomes. An authentication and authorisation strategy was also developed for the Project. The Project has used a range of authentication technologies, but development has primarily been centred on the use of the University Active Directory authentication service via LDAP, which is well established within the institution. The Project has also made significant use of the native Luminis Generic Connector Framework (GCF), which is a single-sign-on technology integrated with the Luminis Portal platform, enabling authentication against other websites. This has typically been used in co-ordination with Active Directory. A full report on the authentication strategy for the Project, and some of the implications of this, is available on the Project websitevi. The early phase of development included the following deliverables:

• A search channel which would be preconfigured to search The European Library catalogue and present the results within the portal channel. This was to be developed using SRU/SRW.

• A training channel which would surface an online tutorial on how to obtain funding, produced by the Research Support Unit.

• A bookmarks channel which would enable the researcher to build up their own individual list of bookmarks within the portal environment.

• A calendar channel which would surface the Outlook calendar within the portal. • A group administration channel which would enable the user to choose a particular research

group within the portal. Phase 2 for development identified the following deliverables:

• Integration of federated library search tools using the Metafind product. • Integration of the CREE JAFERvii resource discovery tool to provide searching of the ZETOC

database from the British Library. • Single-sign-on link to the University of Leeds Publication Database (ULPD).

of 30


• Integration with blogging tools for collaborative research – the product chosen for this was the ELGG blogviii, which is in widespread use for supporting both teaching and research at the University of Leeds.

• Use of the native Luminis discussion forum tools, to be presented via a tab in the Portal. • Development of tools within the portal for managing different research groups, and for content

on the tabs to change depending on the context (e.g.: depending on which research group was selected).

• Development of a personalised RSS feed to provide a feed of funding alerts from Research Research into the Portal.

• Further development of training tutorials within the Portal, focusing on different aspects of the research lifecycle.

At this stage the Project developed a click-through demonstrator which could be used to provide an overall concept for what the environment would look like. This was also developed so that it could be shown to potential users and feedback obtained as part of an iterative testing cycle. Phase 3 of development took the prototype further:

• A tool to enable users to select their research group. • A group bookmark tool to enable research groups to produce a collaborative set of shared

bookmarks. • Linking of community blogs to selected research group. • A channel which lists members of the research group. • Integration of a WIKI tool to allow for further collaboration (using XWIKI). Single-sign-on to the

WIKI. • Integration of Outlook email. • Further SRU search portlets. • Linking of ULPD to the White Rose Institutional ePrints repository to enable a single interface

for deposit and submission to be presented within the portal environment. • Single-sign-on to the Bodington VLE. • Provision of help tutorials for each aspect of the functionality.

A 4th phase for development was also identified, although it was accepted that it may not be possible to achieve this within the timeframe of the Project:

• Integration with SAP Business Intelligence to provide access to tools for grants management. • Integration with the University of Leeds COSTA tool, which supports full economic costing of

grant applications. • Combined search of ULPD and the White Rose Institutional Repository. • Multiple tools for SRU and Z39.50 federated search of library resources. • Wider integration of funding alerts. • Link to training opportunities on campus, via staff training database. • Access to document management system – to replace VKP.

In addition to software development, the Project had also identified a number of other workpackages which ran alongside the development activities:

• Digital preservation requirements analysis • Taxonomy requirements analysis • Resource discovery and document visualisation.

For the digital preservation requirements analysis, it was agreed that it was important to identify and analyse the digital preservation requirements which were unique to VREs. These included the issues surrounding research outputs and collaborative documents which might be produced within the VRE environment. The work-package leveraged and extended existing work within the British Library. The Resource Discovery and Document Visualisation work-package included the development of an experimental visual front end for resource discovery services using document visualisation techniques. This work-package was undertaken as a research activity with Professor Ken Brodlie’s

of 30


Visualisation and Virtual Reality Research Group in the School of Computer Studies at the University of Leedsix. This gave the Project access to the latest cutting edge research in the area of visualisation techniques. The Taxonomy work-package drew heavily on expertise at the British Library, where some work had already been undertaken on evaluating the use of the FAST taxonomy, currently under development at OCLCx. FAST is an adaptation of LCSH, with the intention of making the schema easier to apply and use, especially in a non-traditional context. The work-package was intended to start by evaluating the potential of FAST to provide a taxonomy which could be used in the context of a VRE. This would hopefully lead to the implementation of a prototype taxonomy which could be used for classification of outputs and other objects created in the VRE environment.

Outputs and Results Phase 1 Research and user requirements gathering: Respondents were keen to see a unified interface which brought together the various research systems and tools within a single environment. They flagged that this needed to be easy to use, and that it had to be available to them when they are off-campus. Respondents prioritised the different aspects that they required as follows:

1. Finding and acquiring published information such as articles, conference proceedings, monographs etc.

2. Finding out about funding opportunities; applying for funding; managing funding projects. 3. Collaborating with partners within the University or elsewhere. 4. Sharing or archiving research results. 5. Other activities.

Within these initial priorities, respondents flagged the following issues: Finding information:

• Respondents wanted a simple, easy tool to enable them to search across datasets to find published information. This should have a Google-like interface, and enable them to search many datasets from a single search point.

• They wanted access to native and advanced search interfaces in addition to the above requirement.

• They wanted to be able to build up personal lists for searching, and to search across those resources.

• They did not want the resources to be chosen for them by someone else (eg: their department or a librarian).

Funding:

• Respondents wanted to see automatic alerting of new funding opportunities within the portal environment.

• They wanted to see various aspects of the grants management process simplified – in particular, through access to structured bid templates with a high degree of automation for sign-off.

• They also wanted to be able to search, view and download previously submitted proposals. Collaboration:

• Respondents wanted access to their own email within the portal, plus the ability to share diaries and meeting organisers with others.

• They wanted tools to enable them to work collaboratively on documents and large files. • They wanted to be able to find out who has expertise.

Research outputs:

of 30


• Respondents wanted to be able to find the full-text of an output from the University publications listings.

• Processes needed to be put in place to make it as easy as possible for staff to upload their research outputs from a single point of entry.

Other issues raised:

• Other activities flagged as important included monitoring of financial expenditure and facilities for booking meeting rooms etc.

• Respondents wanted to be able to manage data and content across different systems. Initial priority listing: The outcomes of the user needs analysis suggested the following initial priorities for incorporation of systems and tools into the EVIE portal:

1. Tools for finding information – these would include the Library catalogue, access to Library databases, access to Google, and ability to search across multiple sources of information from a single search interface.

2. Funding – these would include alerting tools for new funding opportunities (from sources such as Research Research and RDInfo), tools to support grants submission, facilities for managing grants.

3. Collaborative tools – these would include facilities for collaboratively managing documents and web content, access to email and diaries, and discussion facilities.

4. Research outputs – these would include facilities for uploading full-text documents to a repository (such as the White Rose Institutional Repository).

5. Managing data across systems – these could include access to the VLE. The full report on the user requirements gathering findings is available on the EVIE websitexi. Phase 2 – Implementation The lack of JSR-168 compliance in the Luminis Portal was not an ideal situation for the Project, faced with the choice of either:

1. Developing EVIE within a JSR-168 framework, separate from the institutional portal, or 2. Developing EVIE within the institutional portal, building channels with IChannel interfaces but

restricting re-use. The EVIE remit was to embed a VRE in the institutional environment. It was felt that developing an EVIE demonstrator in a test architecture which was separate from the main institutional portal would introduce additional barriers to the take-up and engagement of the research community. It was therefore agreed to go ahead and use the Luminis framework. However, some additional constraints accompanied this choice. In particular, the Project faced a problem in that the institutional portal was restricted to use by University members only. This meant that collaboration tools surfaced within EVIE could only by used by internal staff. It was well understood that this was an unrealistic model for many research collaborations and that any full implementation would need to make provision for access by external users. However, these constraints were far outweighed by the benefits of working within an embedded institutional environment. The Project was able to take advantage of the significant extant functionality – e.g.: numerous SSO connectors, bookmark channels, and help and feedback channels etc. The Luminis portal supports a tab-based navigation paradigm. The EVIE prototype was therefore added into the existing portal environment, as an additional set of tabs, mapping onto the research lifecycle identified by the Project as:

1. Resource discovery

of 30


2. Obtain funding 3. Collaboration 4. Research outputs

An additional Utilities tab was also created, which included a number of miscellaneous items, such as feedback, help and other general channels. Researchers who were participating in the evaluation were given a specific ‘EVIE’ role in the portal, so that these tabs and channels were visible to them but not to any other portal users. It was not possible to provide ‘live’ access to the prototype as an output of the Project, as this would require access to the institutional portal service. However, the prototype is available as a set of ‘Captivate’ files on the EVIE websitexii. These files require Flash Player v.8 in order to run. Resource discovery tools: The user needs analysis had indicated a requirement for a tool which would enable searching across multiple sources of information from a single search interface. Leeds University Library had selected the Metafind product from Innovative Interfaces Inc (III) as its federated search productxiii. This product was therefore identified as a suitable candidate for integration with the VRE. The intention was to produce a search portlet which would surface the Metafind client within the VRE interface and enable researchers to use the full functionality within the VRE. Search results could also be potentially displayed within the portlet. The Project began an investigation of this but immediately hit a number of problems:

• The Metafind client is not well documented by III, and no technical documentation appeared to be available which would facilitate implementation.

• Metafind had not been fully adopted by the Library, and no live service was available throughout the lifetime of the EVIE Project. The Library had decided not to make Metafind available to users because of concerns with product functionality and reliability.

As the product had not been fully adopted by the Library, the Project reviewed a number of alternative solutions. One of these was the JISC-funded CREE Jafer tool. The CREE Project has developed a resource discovery tool which can be embedded within a portal environment. This is configurable to talk to any Z39.50 resource. It was agreed that the Project should adopt this tool as an interim measure, pending a further decision by the Library on its choice of federated search tool. In the event, a decision on federated search was made by the Library in Sept 2006. A new product was chosen and is currently being implementedxiv. Unfortunately this was too late for the EVIE Project to take advantage of it. The CREE Jafer portlet was implemented, with some further development to enable it to enhance record display. In order to display the portlet in the portal, the portlet was run in Pluto and displayed within a frame inside the Luminis portal. Technical implementation details and modifications to the code are available on the EVIE website in the system documentationxv. A number of SRU single search channels were also deployed. These are configured to search against a single target – these include the European Library catalogue, the Library of Congress catalogue and the British Library catalogue. These were implemented as channels in the portal, using XSL templates for presentation. The channel builds an SRU query and retrieves the content direct from the source site, and applies an XSL style-sheet to display the results. A number of development challenges were handled, and these are detailed in the system documentation on the EVIE website. The bookmarks channel was developed using an out-of-the-box channel provided by Luminis, which provides the ability for the user to add and delete bookmarks. There is also a Bookmarks Plus channel which provides the functionality for group sharing of bookmarks. Both of these have actually been developed by the uPortal community rather than by Sungard themselves. The Project also assessed a number of enhancements to this channel, provided by Luminis and uPortal developers at the University of Birmingham, University of Bristol and Lawrence Berkeley National Laboratory, which were available through Sourceforge. In the event, functionality was added from a variety of other developments. Full development details are available in the system documentation on the EVIE website.

of 30


Funding tools: A funding alert channel was developed as an RSS feed into the portal. An initial investigation took place into the availability of existing RSS feeds from the main funding opportunities websites. The University of Leeds has a subscription to the Research Research funding alerts website. It emerged that this does not currently supply RSS feeds, and alerts are currently delivered by email. Other research funders provide alerting services by a range of mechanisms – for example, BBSRC has an RSS feed. It was decided to focus the development work on Research Research, as this was the service which seems to be most widely used by research staff in the institution. Individual researchers can set up their own email alerts from Research Research. The subscription process enables them to specify their discipline, and to narrow the focus by sponsor, discipline keyword or by free text. However, it was not possible to combine keywords. In order to develop the funding alert channel, it was necessary to work with the emails generated by the subscription service to develop a personalised RSS feed. This generates the feed in the portal by retrieving new funding alert messages from the users’ email inbox and producing the feed from this. In order for this to be successful, the user must create an email folder called ResearchResearch, and ensure that all of their funding alert emails are automatically filtered into this folder. A JavaMail API is used to retrieve the emails from the folder and create an RSS feed which is then surfaced within the portal. Technical and implementation details are available in the system documentation which is on the EVIE website. A funding workflow tutorial channel was also developed. This is primarily aimed at new researchers, and provides them with a walk-through tutorial on making an application for funding. The tutorial was developed using materials supplied by the University Research Support Unit, Staff and Departmental Development Unit and from the Education, Social Sciences and Law Faculty. The aim was to produce a tutorial which provided a step-by-step guide, organised by timescales, and linked to the most useful information available elsewhere on the University website. The tutorial was developed as a simple set of web links and content presented through a channel in the portal. Collaborative tools: A community blog was implemented using the ELGG blogging tool, which was already being quite widely used by the University. The University has its own installation of ELGG, however, for the purposes of the Project it was necessary to work with a local installation of the software, thus helping to manage the process of username creation and user setup for ourselves. ELGG is not integrated with the University Active Directory system, so usernames for users are manually created on the system, following a user request. This clearly provided problems for implementing SSO from the portal into ELGG, so a local installation enabled the team to have more control over this process. Having done this, it was possible to set up single sign-on from the portal which was called by the IFRAME channel displaying ELGG. The single sign-on was provided by the General Connector Framework (GCF). The default IFRAME channel was altered to make it group aware so that the user was not only logged straight into ELGG, they were logged into a specific group. The Project also worked with the native Luminis forum tools to provide an alternative set of collaboration tools. Luminis provides a set of ‘group’ tools which enable individuals to set up groups, invite others to join, and use a range of tools for communication and sharing documents, including a discussion forum, chat facility, news facility and tools for sharing documents and images. Part of the Luminis Groups functionality was surfaced on a tab within the Portal. The message board was modified to enable it to become ‘group aware’ – that is, when the user selected a particular research group in the portal, the modified message board automatically selected this group as well. This interfaced with a separate group selection channel which was created to enable users to select their choice of research group within the portal. Technical details for both of these developments are available in the system documentation on the EVIE website. Research outputs:

of 30


The Project explored the options for providing a search interface for the University of Leeds Publication Database (ULPD) within the portal. This needed to be achieved utilising SSO to enable the application to open seamlessly within the portal environment without the need for further authentication. ULPD currently uses NTLM authentication, which means that it was not possible to implement SSO using the Luminis Generic Connector Framework because there was no way to pass NTLM credentials from the GCF to the browser. NTLM authentication had been implemented to enable seamless (no password) authentication for staff users on campus. However, ULPD forces a login when accessed from off-campus. NTLM authentication was therefore utilised in the Portal, via a simple link presented in a channel that opened a new window connecting to ULPD so that, when on campus, no authentication challenge was made. The Project also intended to implement an SSO for the White Rose Institutional Repository, and to explore the options for provision of integration across White Rose and ULPD, so that data entered into one system would be surfaced in the other. This was investigated, and it was established that the work required the facility to provide an import/export facility between ULPD and White Rose. Import/export was not provided with the current version of ePrints, which is the software used by White Rose. It was established that this functionality would be available in ePrints version 3, and it was agreed that it made most sense to wait for this, as ePrints 3 also provided additional support for portal integration. ePrints 3 was originally due in summer 2006. However, the software was not actually made available until January 2007, which was, unfortunately, too late for the Project to progress this further. Managing data across systems: The Project also undertook to integrate the Bodington VLE into the portal environment. This was achieved through the use of single-sign-on (SSO), which enables the user to seamlessly authenticate against the Bodington VLE, once they have signed on to the Portal. It was relatively straightforward to create an SSO connector for Bodington, using the Luminis GCF (Generic Connector Framework). However, this also required a significant re-assessment of the authentication mechanisms which were in use for Bodington at that time. For various historical reasons, Bodington was set up to authenticate against its own database, rather than against a central authentication database such as Active Directory. The decision was taken to change this to authenticate Bodington against the institutional Active Directory via LDAP. This greatly simplifies the process of creating an SSO connector, as both Bodington and the Portal would be authenticating against the same system. The work was taken forward in two phases. Phase 1 was the implementation of a Bodington authentication plug-in which enabled Bodington to authenticate against the University Active Directory. Phase 2 was intended to resolve a number of data issues which emerged between Bodington and Active Directory. It emerged that the user data in the Bodington database was out of sync with the data held in the institutional Active Directory. This was as a result of manual data import processes into the Bodington database, which had been taking place for a number of years. Data to the Bodington system was provided from a number of sources, including SAP (the institutional HR management system) and Banner (the institutional student information system). No real housekeeping had been undertaken on the process for a number of years. This had resulted in a number of duplicate accounts – particularly for staff who might have had more than one record in a University system – eg: as a member of staff and as a student. It was therefore necessary to undertake a complete data cleansing exercise before the SSO could be put into place. This involved a painstaking process of identifying accounts that are incorrect using a master dataset, fixing these, and identifying a long-term maintenance process. Staff data on Bodington was derived from SAP. The University had introduced a new employee id field in order to link multiple accounts which may have arisen as a result of staff having multiple roles in the institution. This was used to identify staff with multiple accounts on Bodington. The Bodington authentication plug-in was implemented as a Java class, and the source code for this was made available under the Bodington open source licencexvi. Digital Preservation:

of 30


In work-package 4, the Project analysed the digital preservation requirements specific to VREs. This work-package was led by the British Library, and leveraged and extended existing work in which they are involved. The full work-package output is available on the Project websitexvii. The analysis found that research notes and outputs are scattered across an increasingly wide range of paper and digital formats, including in emails, on personal and network drives, in the logs of instant messaging tools and in shared directories. Increasingly, outputs are starting to appear in institutional and discipline-based repositories, but this is often limited to the peer reviewed version of a published document. A VRE provides a further potential environment where researchers might be creating and making available a variety of research outputs, particularly if collaborative tools are made available within the VRE, or integrated with it. Questions were raised about how these outputs could be captured and preserved so that they can be made securely available for the long-term. As part of the analysis, the Project identified two use cases which were used to explore the type and range of research outputs being produced by researchers. Two researchers were interviewed about their creation of research outputs in relation to specific and typical research projects. One use case is centred on a researcher working in the social sciencesxviii, and the other is from an arts/computing backgroundxix. The use cases identified a wide range of types of research output which could potentially be produced or made available within the VRE. These included:

• Text-based outputs – eg: journal articles, conference papers, book chapters etc. • Project-related emails • Shared community knowledge (perhaps available through a community blog or wiki). • Contract and budget information. • Project websites – which may have both public and private views. • Project deliverables – eg: research reports, promotional material etc. • Research proposals and costing data on these. • Raw research data. • Processed research data. • Software and source code. • Sound/dialogue recordings – including both audio and video. • Working papers and consultancy reports

Crucially, the researchers were producing dynamic outputs, which changed over the lifecycle of the research project. In some cases, raw data needed to be retained for the long-term, whereas in others it could be discarded once it had been processed, or analysed. The researchers were currently using a range of different repository systems to archive their data, but some of these may no longer be available once a project has completed. The work-package therefore tackled the question of what research outputs needed to be preserved for the long term. There is already a mechanism for preserving peer-reviewed traditionally published material such as subscription journals. In the UK this is centred on the British Library which preserves physical copies of all UK published materials under legal deposit. The legislation has recently been extended to cover digital material as well as print. Metadata, and in some cases the content, of published research outputs held at the British Library is exposed through resource discovery mechanisms such as ZETOC and UK PubMedCentral. Other types of output are not, however, picked up by this mechanism, and a number of these (eg: media formats) may pose specific preservation challenges. Some researchers may already be submitting their outputs to institutional repositories, or discipline-based repositories, such as ArXiv. However, these repositories may not themselves have preservation policies in place. The Project therefore developed a proposed workflow for the preservation of outputs held in, or accessible via the VRE. The workflow assumes that the VRE is supported by an external preservation service provided by the British Library. A detailed workflow diagram exists, and is

of 30


available on the Project websitexx. The workflow presents a holistic view of the research lifecycle from initial generation of the outputs within the research processes supported by the VRE, through export of these to an external preservation service, and resource discovery via the resource discovery portal. The workflow is sufficiently generic to be applicable to other HE institutions building VRE infrastructure. The workflow assumes that the majority of research outputs would be held outside of the VRE environment, although they may have been made accessible through it, or originally created within the VRE. It recommends that the different types of research outputs would be held in the following repositories at the University of Leeds or, in some cases, externally:

• Research publications (e.g.: journal articles, preprints, conference papers, posters, book chapters etc) – the White Rose Institutional Repository.

• Digitised objects (e.g.: images, multimedia, sound, video etc) – the MIDESS repository. This is a JISC-funded initiative, through which the University is developing a repository for multimedia materialsxxi.

• Research data – should be held in a trusted data repository developed either at the University of Leeds, or elsewhere (for example, the Research Councils).

• Personal and project websites – the UK web archive. • Email, files etc – institutional SAN.

The British Library would then harvest both metadata and content from these sources using Web Services and OAI-PMH as the means of transmission. This would build on work already being taken forward by the British Library in relation to the JISC-funded Ethos Projectxxii. Research outputs would be transferred to an import server at the British Library where they would be held prior to ingest into the Digital Object Management (DOM) system which is the central British Library electronic store. Export could take place on an automatic or event-driven basis – for example, it might be necessary to transfer outputs across at the point where preservation action was required. Once on DOM, preservation actions would be performed, as well as ensuring that the output metadata was exposed to resource discovery mechanisms, as appropriate. The Project also addressed the associated metadata requirements for digital preservation, and for other functions, such as resource discovery. PREMISxxiii has outlined a very detailed set of preservation metadata requirements. However, the challenges of implementing this are considerable. Automatic metadata gathering and extraction is still in its infancy. The Project considered that researchers could not be expected to provide large amounts of metadata by hand, and that this would actually act as a deterrent to submission of research outputs as the barriers would be felt to be too great. It was therefore recommended that a much reduced metadata would be more practical, and that this should be based on the AHDS (Sherpa DP) recommendationsxxiv. Document visualisation: Workpackage 5 focused on resource discovery. This workpackage included a small research project on document visualisation techniques. The work was led by Professor Ken Brodlie in the School of Computing at the University of Leeds, and was taken forward as a 10-week summer research project, using postgraduate students from the School. Search engines such as Google present their results to the users as a flat list ordered by relevancy ranking. This approach requires the user to browse the list of results and judge each result individually. While this approach may be useful to the user who has a particular website in mind, it is often less useful to the user who wishes to find a range of sites on a particular topic, as it does not allow the user to easily obtain an overall picture of the types of results returned or the relationships between these results. A possible solution to this is to present the results as a graph which links the results to relevant keywords, presenting the user with a visual map (visualisation) from which they can select at a glance the sites of most interest to them. GooRaph is a tool for presenting a visual interface to search results. It was developed in Java 5.0 using TouchGraphxxv to display results from Google using the Java Google Web APIxxvi. If appropriate, this could be made available as an experimental interface through the VRE for researchers to test and evaluate.

of 30


Three potential presentation interfaces were initially considered. The first of these was a Java applet, shipped with the Sun J2SDK. This applet takes a list of spheres and connecting springs, and produces an animated visual model of the search. TouchGraph provides a similar model to the Sun applet, however, it replaces the spheres with textual labels, is more developed and has a richer functionality. A third option would have been to develop an entire model and presentation from scratch. This was discounted as the development effort was too great for the scope of the Project. TouchGraph was chosen because of its particular features, which meshed well with the requirements for the GooRaph interface – see the report available on the Project website for further detailsxxvii. The keyword extraction component utilises the Google search engine via the Google Web API service. This service is limited to 1000 free calls to Google per day. It was therefore determined that GooRaph is not currently viable as a live service because of the limits imposed by the Google API. The Project was therefore only able to produce a demonstrator system, which could not be deployed in the live VRE interface. The approach was taken to extract keywords from cached copies of the first 10 results using a comprehensive modified stop word list. The extracted keywords are then used in conjunction with the original search terms to classify the remaining results. Design and implementation is described in the report available on the Project websitexxviii. An evaluation of the suitability of GooRaph for a live service revealed a number of issues relating to the usability of the end-user interface and the length of time taken by the keyword extraction component to return a processed set of results, which could be improved with further work. The biggest problem encountered, however, was the time taken to run the keyword extraction component. This requires several minutes to return a processed set of results – which renders GooRaph unacceptable as a search engine in a live service. A number of potential enhancements to GooRaph were considered. One possible improvement would be to improve the algorithm for dealing with ‘empty’ matrix rows. The keyword extraction component could also be improved to use a lexical resource such as WordNetxxix to select words from a candidate list that are related to the initial search terms. Project time constraints prevented further exploration of these enhancements. A demonstration of the GooRaph visualisation is available on the EVIE websitexxx. Taxonomy: An initial analysis of the FAST taxonomy was undertaken, with a view to ascertaining its potential applicability in a VRE environment. The aim of the FAST project at OCLC is to develop a subject-heading schema based on LCSH that is easy to use, understand and maintain. LCSH has been designed for implementation primarily in a traditional library environment – however, there is an increasing requirement for standard (and specialised) taxonomies which can be used to classify objects and knowledge in other kinds of environments – for example, in relation to e-science data. FAST is intended to be used by people with no library training and minimal experience, in order to enable a broad range of users to assign subject terminology to web resources. The Project produced a report on FAST, which is available on the Project websitexxxi. The user requirements analysis exercise undertaken in Workpackage 2 of EVIE did not indicate any requirement from researchers for a means to classify objects and information created within the VRE. The Project also participated in a workshop with academic staff looking at support requirements for e-research activitiesxxxii. The role of taxonomies in supporting e-research was raised at the workshop – but the outcome was inconclusive. Researchers felt that they did not themselves have the expertise to develop taxonomies for their data; however, they also felt that library staff would not have the sufficient knowledge of the subject in order to be able to contribute effectively. The use of a large-scale taxonomy such as FAST was felt to be inappropriate for many requirements, as it was felt that it would not be specific enough to make it useful to highly specialised fields. A taxonomy within a VRE might potentially also aid the retrieval of documents, and thus enhance resource discovery. FAST is broad enough to encompass all research domains, so could potentially be used for this purpose. However, it emerged that FAST cannot be embedded within a local environment. At present it can only be queried via use of a web form on the OCLC website. The

of 30


hierarchy cannot be adapted to meet local research needs. Therefore it was felt that FAST is not currently suitable to meet these specific requirements. The conclusion drawn from the evaluation of FAST was that it would not be suitable for use within a VRE. The Project then went on to look at other taxonomy developments to assess whether an alternative option might be available including a number of emerging developments of what is being termed Web 2.0 technology. A full report on this work is available on the Project websitexxxiii.

• Semantic web ontologies: the Project looked at XML and RDF implementations of ontologies, which aim to provide a specification for machine-readable metadata for all web resources. The best-known language for defining web ontologies is OWL. The Project identified a number of live implementations of OWL. However, most of these are currently limited to quite narrow and specialist subject domains.

• Folksonomies: folksonomies are a form of social classification, or social bookmarking of the web, which enable end users to apply their own keywords (known as ‘tags’) to web resources. One successful implementation of a folksonomy approach can be found on the Del.icio.us website. Folksonomies are rapidly increasing in popularity, and it is thought that extensive use of them could accelerate the evolution of the semantic web. End users do not have to be subject experts, although this will have implications for the quality and accuracy of the classification that takes place.

Semantic web ontologies could be used within a VRE for the classification of research outputs. However, the Project found that the implementation of an ontology framework such as RDF or OWL would require substantial effort in building classification hierarchies and in providing resolution services. The long-term maintenance overheads are also substantial. The expectations on researchers actively using the framework to classify their outputs is also substantial, and would require significant cultural change. The investment in development of such a framework was felt to be inappropriate for the requirements of a pilot project. A folksonomy approach would appear to be the most appropriate and cost-effective option, although it was unclear how this would scale across a potentially institution-wide VRE including many subject areas and incorporating the needs of different research groups. Weaknesses of the folksonomy approach include the fact that there is no scalability to allow for disambiguation and tag conflict resolution. There are also no mechanisms available for maintaining and updating tags as usage of terms changes over time. The conclusion of the taxonomy work was that no suitable schemes currently exist for the effective application of a taxonomy within a VRE. The Project was therefore not able to implement and test a pilot approach. Phase 3 – Testing and evaluation Testing: Test plans for all channels are available on the EVIE website in the system documentationxxxiv. The EVIE VRE was tested for conformance with W3C’s Web Content Accessibility Guidelines (WCAG 1.0 (priority 1). The review process and outcomes are described in the EVIE Accessibility Studyxxxv, which is available on the Project website. Based on this evaluation, the VRE does not meet WCAG 1.0 conformance Priority 1. There are a number of accessibility issues – however, most of these are due to the Luminis portal framework in which the VRE tools are presented. Portal interfaces are inherently problematic from a usability perspective, in that numerous complex tools are presented on a single screen. The presentation of channel layout via nested tables exacerbates the navigational problems for users using linear rendering agents. The Luminis channel framework is based on uPortal, which does provide alternative rendering and skinning paradigms which could reduce these problems, however, these are not implemented in the current version of Luminis. However, Sungard are committed to developing increasing compliance with Section 508 of the Rehabilitation Act (29 USC 794(d)) and currently test products against Section 508 and assistive technologies. The forthcoming Luminis 4 release (scheduled for q1 2007) – which includes uPortal 2.5 – should significantly improve compliance.

of 30


The EVIE channels are accessible in themselves but require a few additional tags for completeness. Alternative forms of user help will be required for users of assistive technology. Evaluation: The full evaluation reportxxxvi is available on the EVIE website. The evaluation was small scale, involving 35 research staff and students in total. The VRE was evaluated within the framework of the Leeds portal. A new role of ‘evieuser’ was added to the portal which gave users an additional set of tabs organised according to the EVIE lifecycle aspects identified in the requirements analysis. The prototype interface was initially used for demonstrating to a small number of interviewees and then a focus group to draw feedback and stimulate discussion. The Project team recruited 27 participants for evaluating EVIE in a more realistic setting. These participants were given portal access (where they did not already have it) and given the ‘evieuser’ role. During a handover process, users were shown how to use the EVIE channels and were given a set of suggested tasks to complete. However, users were encouraged to incorporate EVIE into their regular research practice where possible. Users were able to contact the EVIE Project Team directly or pass comments via an EVIE feedback channel. At the end of the evaluation period, a questionnaire was circulated to the portal based participants. Additionally, interviews and a second focus group were organised which involved selected users of the portal based prototype. Senior stakeholder interviews took place in parallel with the portal based evaluation. The format was similar to the initial interviews but included questions tailored to the roles of senior stakeholders. It was clear from examination of the EVIE databases that the usage had not been as high as would have been preferred. One of the reasons for this was a perception that the collaborative tools would not be available for very long so investing time in their use would not be well spent. In some cases, the users already had collaborative tools available. Due to time constraints in development, the selection of tools available to users was not as rich as it might have been. However, although small, the cross section of users allowed for an adequate level of feedback.

SRU Search Channels and CREE Jafer Channel The quantitative analysis indicates that these were popular channels amongst the user group. Users benefited from:

● Being able to initiate the search from within the portals ● Being able to view the results quickly, in this case in the portal.

There were some criticisms of the usability and relevance of the channels, which emerged in the qualitative analysis. The usability issues included:

● Lack of a clear button to reset the channels ● Channels being rearranged by the resizing of the channel when the results were displayed.

Some participants would prefer to use a native web interface to search a resource because of the richer functionality available. However there was perceived variation reported in the quality of native web interfaces for resource discovery services. If usability and relevance issues were resolved, these channels would have significant value to users in a live VRE.

Personal Bookmarks Over 70% of users in the online evaluation agreed that this channel was easy to use though only half felt that it provided the functionality they required. It was generally felt that it was beneficial to have separate instances of the bookmarks channel to store different categories of links. Some users said they would prefer to use their browser bookmarks rather than a bookmarks channel in the portal. This may be a usability issue as browsers provide additional functionality to the bookmarks channel such as moving bookmarks between folders and automatic capture of URLs. It could also be a reflection of users’ current feelings towards adopting the portal as their working environment. This interpretation is supported by comments from users who felt that, although they

of 30


would currently prefer to use their browsers to store bookmarks, there may be a shift towards doing this within the portal over time.

Funding Workflow Response to the funding workflow was mixed. Generally, users were in favour of such a tool within the portal (particularly for use by less experienced researchers). However, participants were divided on the issue of whether the content provided was useful to them. Generally, users felt that the content should be controlled by the department/faculty. This reflects the diversity in procedures and requirements for funding applications across different disciplines that was identified during focus groups and interviews. Users were frustrated by the inclusion of links to information which was not available from off-campus as they expect seamless off-campus access to resources through the portal.

Funding Alerts RSS Despite the fact that only two participants in the online evaluation were able to use the funding alerts channels the overall response was positive. It was generally felt that RSS (particularly when ordered by deadline) was a preferable format to email for receiving funding alerts. There were concerns expressed by some interview and focus group participants, however, over the lifespan of the information (as with RSS new information replaces old) and that the information may in fact be less visible if the user only visits the portal infrequently.

Group Select Channel Not all users easily realised that this channel changed the selected group for other channels. However, the number who found it useful to switch groups in this way was higher although at least one user preferred the Luminis “My Modules” option as a way of navigating between groups/modules. In general, users do not prefer to use the Luminis Groups alternatives to the EVIE tools (e.g. links instead of the Bookmarks channel) suggesting that the preferred way of presenting tools generally is through the portal tabs.

Group Bookmarks The majority of participants found it valuable to have a set of shared bookmarks created as part of a set of group oriented tools. Opinions on the functionality, however, were divided. Those who had never used shared bookmarks before thought the group bookmarks channel was very useful, whereas those who had experience of other shared bookmarking tools thought the EVIE channel was lacking in functionality and found the user interface cumbersome and confusing. These users wanted to be able to include annotations when adding a bookmark and for the name of the contributor and date added to be visible to other members of the group.

Collaboration 10 respondents to the questionnaire wanted collaboration to be available through the portal as opposed to 5 who wanted collaboration through an external system available in the portal via single sign-on. There was some contradiction here with the focus groups, who were in favour of a lightweight portal that focuses on single sign-on, with the majority of functionality being in external systems.

Outcomes This section discusses the results of the Project in terms of the aims and objectives which were identified at the Project outset. A summary of how each of these aims and objectives were met is given below:

1. Establish a prototype VRE infrastructure based on open standards and existing software components to support a test group of researchers; including users from the School of Medicine, School of Geography, and researchers using the White Rose Grid.

The EVIE Project successfully established a prototype VRE infrastructure, which was built on open standards, and successfully incorporated a range of existing software components and services. This prototype implementation was tested with researchers across a range of disciplines. Thus, the Project went beyond the initial aim in this, in that a prototype was delivered to users across a broad

of 30


range of disciplines, and was thus applicable to a much wider range of potential users than had originally been envisaged. There is a clear demand for research tools to be made available in a portal environment. The questionnaire showed that 100% of respondents thought that “it was useful to have the EVIE environment available as part of the Portal”. Forty seven percent indicated they strongly agreed with this statement. Questionnaire responses to individual channels showed a strong desire to embed the EVIE channels into the existing University Portal. However, this overall positive response is qualified by the qualitative data expressed through over twenty hours of focus groups and one-to-on interviews, where more detailed and often conflicting requirements were expressed, as can be seen in the full evaluation reportxxxvii.

2. Provide a set of additional resources and services through this environment, including facilities for enhanced search and retrieval.

A wide range of resources and services were delivered through the VRE, including a range of facilities for enhanced search and retrieval. The Project was not able to successfully deliver a full federated search portal within the VRE, although this was as a result of organisational issues within the institution, rather than as a result of any technical difficulties encountered. However, the Project did undertake work on the use of SRU/SRW in a portal context, and it is anticipated that this work will be of wider interest and significance to the JISC community.

3. Deliver simplified-sign-on functionality to enable seamless integration between the identified platforms.

The Project developed SSO access from the Portal for a range of services and systems, including the Library Catalogue, the Bodington VLE, the University of Leeds Publication Database and the ELGG blogging system. This was successfully tested to show the value to users of having a seamlessly integrated environment as far as authentication is concerned.

4. Provide a set of user validated recommendations identifying effective, scaleable and reusable mechanisms for construction of intuitive search and retrieval tools within this environment.

This was achieved in relation to the resource discovery channels implemented in EVIE, although the outcomes raise questions about the scalability of SRU as a mechanism for delivering search tools, and also the usability of SRU.

5. Provision of enhanced resource discovery mechanisms with document visualisation techniques available to indicate relevance.

Document visualisation techniques were explored within the Project, and an experimental service was developed. However, this was not stable enough to be released as part of the prototype VRE.

6. Develop best practice for the use of taxonomy within a VRE. The Project explored the options for use of taxonomy within a VRE, and identified that no suitable solution currently exists; although developments such as semantic web ontologies and folksonomies may offer some scope for future developments in this area.

7. Provide support for search and retrieval mechanisms across disparate information resources within a VRE.

The Project successfully identified options for exposure of VRE content to external search and retrieval mechanisms, using Web Services and OAI-PMH protocols, and via linking with institutional repositories. The majority of this infrastructure was already in place within the institution, so no actual development was required by the Project.

8. Identify long-term options and requirements for digital preservation in a VRE.

of 30


The Project produced a detailed document on digital preservation requirements which outlined a potential workflow model and proposed preservation service managed by the British Library.

9. Identify requirements for data integration to provide a seamless flow of information between systems integrated through the environment.

The Project produced a document on the potential benefits and obstacles involved in integrating back end systems, in particular the use of LDIS (Luminis Data Integration Suite). Overall, development timescales were set back by a delay on procurement of the portal product, which was outside the control of the Project. The procurement exercise was being led by the institution, and the Project was not able to control the timescales – e.g.: for release of finances etc. As a result of delays in procurement, the development work for the Project coincided with the parallel development work taking place for the implementation and roll-out of the Student Portal. These had a negative impact on the Project, as both Project staff and institutional staff were inexperienced in the portal development environment, and were climbing a very steep learning curve. This also led to increased risk for the Project, as it was dependent on the development timescales of the Student Portal Project, which was providing the overall framework. Ideally, the Project would have been working with a stable portal environment, which had already been in live use for a period of time, and with a higher level of general expertise within the institution. The portal framework is an extremely complex one, and to some extent, the Project under-estimated the difficulties of developing tools within this environment. Development therefore took longer than expected, and could not always be achieved in a standards-compliant way. The JSR-168 standard was not fully mature, and was not supported by the currently available version of Luminis. As a result of these difficulties and delays, the Project was not able to complete a full iterative cycle of development. The Project also encountered broader difficulties with a rapidly changing institutional systems landscape. A number of key decisions or changes were taking place at the institutional-level during the course of the Project, which had a direct impact:

• The decision to withdraw the VKP system in summer 2007 led to additional work for the Project in assessing and implementing alternative collaborative working tools.

• The decision to withdraw the Bodington VLE by summer 2008 had a lesser impact, but contributed to general uncertainty within the Project.

• The rollout of ePrints version 3 did not occur within the timeframe of EVIE implementation • The lack of a stable federated searching tool in the University Library led to additional project

work in assessing and implementing alternatives. • The volatility of the research administration systems landscape led to a difficulty in scoping

the appropriate products for integration. • During the course of the Project, the University developed and roll-out a Full Economic

Costing system (COSTA), implemented SAP Grants Management, and implemented systems for supporting the forthcoming RAE.

• Neither COSTA nor SAP GM were considered stable enough for integration with the VRE during the Project period.

• Integration with other research administration systems proved very difficult because the team working on these systems was fully occupied development of these systems and development of systems to support the RAE, so did not have the time to spare to explore portal integration.

• Lack of standards compliance was also an issue in integrating research administration systems.

• Data management issues also caused problems for the Project. These emerged specifically in relation to the Bodington system, but wider issues were also present with the validity and accuracy of data in corporate systems such as SAP and Banner.

of 30


Dissemination Dissemination took place throughout the life of the Project, and a number of key internal and external mechanisms for dissemination were targeted. Internal dissemination routes included:

• The EVIE Working Group met on a regular basis throughout the Project, and provided a useful forum for providing guidance, resolving issues, as well as a mechanism for further dissemination, as Working Group members reported progress back to their colleagues.

• EVIE Steering Group met periodically throughout the Project, and provided a useful forum for input into the overall project direction as well as providing a mechanism for disseminating and supporting the Project.

• The University Research Board and Faculty Research Committees – which were targeted with reports and papers for key Project milestones, such as the results of the user needs analysis.

• Senior stakeholders - for example, the Pro-Vice Chancellor for Research was provided with a briefing on EVIE, and was given specific briefings on a range of issues which arose from the Project, including digital preservation and data storage issues and requirements for development of a full Staff Portal. The PVC used briefing material from the Project in her presentation to a ‘futures brainstorming’ event for senior managers in the University, which included the Vice Chancellor.

• Senior Library and ISS Management Teams. The Project provided monthly ‘highlight’ reports to both the Library and ISS Management Team. A presentation was also provided for all Library staff.

• The EVIE Team engaged with ISS staff on development of collaborative working tools, and with the Portal Team on wider portal development issues.

• A presentation to staff at the British Library. • General Portal briefings aimed at all University staff; over 500 staff attended these briefings in

summer 2006, where EVIE was mentioned alongside other portal developments within the institution.

• The EVIE Team attended an internal workshop on support for e-research, and also used this as an opportunity to disseminate information about the Project.

• EVIE provided regular ‘highlight’ reports to the University Research Systems Steering Group – which included stakeholders from the Research Support Unit, Central Administration, senior academic staff and ISS.

• A report on the Project was taken to the VKP Steering Group. • The Project Team were interviewed by external consultants as part of the University virtual

environments review; and the EVIE Project was featured in the review document as an innovative approach.

• News items were posted on the campus intranet, campus mailing lists and on the ELGG blog. • The Project website was actively used for dissemination activities. • The EVIE Team also met with staff from across the institution, including staff from the Staff

and Departmental Development Unit and Knowledge Transfer Unit. • The user needs analysis and the final evaluation also provided a number of opportunities for

dissemination, as staff were able to hear about the Project through focus groups, one-to-one interviews, and questionnaires and through evaluation of the prototype.

• Project team members attended a 5-year planning brainstorming session for research systems development.

• Project outputs will be widely circulated within the institution. External dissemination mechanisms included:

• A paper on “using a Virtual Research Environment to present CRIS grouped to support the research users’ lifecycle” was accepted for EuroCRIS 2006, and presented in Bergen in May 2006.

• The EVIE Team produced a report on the EuroCRIS event for Ariadne, which was published in July 2006.

• EVIE was also mentioned in Ariadne articles in July 05 and Sept 06.

of 30


• The Team gave a presentation to a workshop on eResearch, Digital Repositories and Portals at the University of Lancaster in Sept 06.

• A presentation was given to the Luminis European User Group in Dublin in December 2006. • The Team undertook substantial engagement with the JA-SIG developer community both in

the UK and worldwide. • The team provided feedback on the SRU standard to the Library of Congress and Koninlijke

Bibliotheque. • A presentation on the digital preservation aspects of EVIE was given at a DCC workshop at

the University of Leeds in February 2007. • A conference call was held with representatives from the UK Clinical Research Network,

National Institute of Health Research and the UK Biobank Project in February 2007. • An article on the Project was produced for Ariadne (forthcoming – April 2007). • The Team actively engaged in discussions with the University of Hull, University of Lancaster

and University of Oxford regarding the development of VRE tools, particularly in relation to resource discovery and collaboration.

• The Project participated in the EReSS study (cross programme activity) which looked at interoperability issues across VREs.

• The EVIE Team led breakout groups at VRE Programme meetings, and provided leadership on the collaborative tools strand of the programme.

• The Team engaged with ePrints community over development of a portlet for the White Rose Institutional Repository.

• The EVIE Project was mentioned in the Office of Science and Technology eInfrastructure Group report on Virtual Research Communities.

Conclusions and Lessons Learned Overall, the EVIE Project has been successful in achieving its aims, and in contributing to a deeper understanding of the role of VREs, both within the institution and within the wider HE community. It is clear from Phase 1 of the VRE Programme that substantial strides have been made in understanding how a VRE might contribute to supporting research. EVIE has demonstrated that researchers would welcome a seamless environment where they can access the tools that they need to support their research activities. Overwhelmingly, the researchers who took part in the evaluation process appear to value having access to a range of research and community-building tools via an integrated VRE in a portal framework – as 100% of them indicated this in the questionnaire. Single-sign-on functionality is particularly valued as the highest priority for researchers, and they tended to favour a lightweight entry point to existing rich functionality, rather than more limited functionality surfaced within a portal channel. Researchers also valued the fact that access to tools via the portal provided them with a single point of entry to the services that they needed to use on a regular basis. SSO is a cost-effective way of developing integration between a portal and external systems. The use of collaborative working tools is becoming increasingly important to support research – for this to be effective, access to the VRE by external groups and researchers has to be supported. Rich document management functionality is also required (79% of respondents asked for this). Experienced users of collaborative working tools often expressed disappointment at the limited range of functionality provided by the EVIE prototype, indicating a need for much richer functionality. Researchers cite time constraints, lack of access to suitable tools and lack of access to information about expertise as key barriers to collaborative working. However, they feel that collaborative working tools have the potential to improve the efficiency of research collaboration, and to provide improved awareness of what is going on at a wider level (e.g.: within the wider discipline). It is essential that researchers have control over the VRE. In particular, the ability to personalise the VRE to provide tailored services is considered essential. Researchers specifically stressed this in

of 30


relation to the implementation of resource discovery tools and funding alerts and tutorials – where they expect to be able to personalise these services to provide access to the specific resources and information that are most relevant to them. For example, a researcher from Computing might find search tools for IEEE Xplore and ACM Digital Library already set up. They could find a funding workflow tool already integrated with their departmental peer review college and the relevant funding bodies. Their RSS reader could be pre-populated with Computing related funding feeds. There could be preset links to departmental information on journal submissions. Coupled with the ability for the user to further customise their workspace to their specific requirements, these facilities would significantly reduce the learning curve of new researchers in the department. They also expect to be able to add annotations in the VRE, and to share these across a research group. This implies that a fine granularity of portal roles will be required to facilitate further development and that roles will need to be linked to rich personal profiles on the system. There is also a requirement to integrate not only with systems within a single institution, but also, driven by the dispersed nature of research communities, with a variety of systems and resources outside the institution. These might include:

• Collaboration tools at a collaborating institution or commercial partner. • External Web 2.0 collaboration tools (e.g.: Google Docs). • Grid services. • Broader Information Environment tools and services. • Integration with the emerging landscape of institutional and other repositories and other

publication mechanisms. Researchers expect flexibility so that they can obtain maximum advantage from the investment being made in this area. If, for example, they are not using an institutionally-supported blog, they expect that they will be able to switch off this functionality and easily plug-in their own preferred tools. This fits well with the concept of Web 2.0, but does provide challenges for centrally-supported institutional systems in trying to manage this level of flexibility. Ease of use is also a key in encouraging take-up; researchers expect interfaces to be consistent, integration with existing institutional services needs to be robust, and new interfaces should require only a shallow learning curve. Researchers will not take the time to get to know a new system if it at first appears to be difficult to use. There were a number of usability problems with the prototype system which impeded use of some of the services, and could potentially have been ironed out with earlier end-user testing. In the context of a VRE there is little scope for development of a fully fledged taxonomy or ontological solution. The expertise needed to establish such a categorisation scheme, and the degree of maintenance required mean that the costs outweigh the benefits. For subject specific VREs this may not be the case. There is a general low awareness within the institution of existing research support tools (for example, the institutional repository, research costing tools). It is likely that surfacing such tools in a VRE framework will increase awareness and use. Researchers also feel that better leveraging of institutional tools (e.g.: richer functionality, improved workflows and integration) will lead to improved adoption, but that this needs to be done sensitively, to avoid information overload. Take-up of a VRE is likely to be incremental and evolutionary. Newer researchers seem more willing to adopt new tools – such as Skype or blogging - and working practices than those who are more established. Established researchers are more likely to need ongoing support in adopting new tools. Perhaps a less tangible aspect of integration of systems is the greater cooperation that portal development requires or encourages from IT staff across campus, which, once established, can facilitate wider integration of systems. Where the starting position regarding system integration is poor it may have implications a VRE or portal implementation project.

of 30


One of the benefits of a VRE would be to help bring about cultural change. Exactly how much could be achieved through the environment is not clear but portals can offer a common interface to students and researchers alike, potentially drawing future researchers into acceptance of the range of systems presented through the portal. However, this is still not a guarantee of adoption. If the tools fail to provide the functionality that new researchers want then they will cease to use the portal. It is clear that a number of challenges remain to be addressed before VREs enter mainstream use by researchers. In particular, it will be necessary to address the cultural challenges to VRE adoption, and this can only be achieved if VRE development and adoption is promoted and resourced by institutions. Several characteristics of VREs have emerged from the work of the EVIE Project. There is a requirement for highly granular flexibility of both tools and presentation, driven by recognition of the diversity of user needs and requirements the environment is designed to support. The emerging picture of a VRE also contrasts interestingly with the prevailing current perspective of a VLE. This is usually seen as a relatively tightly integrated set of tools and services bundled in a discrete package or application. The majority of VLEs are still characterised in this way, despite recent demands for greater pedagogic flexibility. The JISC e-Frameworkxxxviii provides the main challenge to this approach, with its emphasis on a service oriented approach. The characteristics of a VRE are increasingly linked to requirements to support the development of ‘virtual research communities’, as identified in the OST Report of March 2006xxxix. This report raises the national and international profile of VRE developments, and flags them as a significant objective for future development. The viable architecture for a VRE, therefore, seems to fit well with the approach being promoted by JISC, and there is an opportunity for the next phase of the VRE Programme to make a substantial contribution to the emerging e-Framework. The increased use by researchers of social networking applications, for example, adds urgency to the requirement for VREs to adhere to open and published standards and specifications. Future VRE development must build on a range of existing components, including both collaborative and portal frameworks and standards, and must provide the ability for researchers to customise and personalise the environment. The extent to which researchers will want to do this is highly dependent on individual characteristics. For example, some researchers will welcome the provision of a set of standard tools within a VRE, which will significantly reduce the need for them to get involved in setting up tools and services for themselves. Others will prefer to be able to personalise the environment, adding their own tools and preferred services. Digital preservation and VREs: The Project produced a valuable model for the management of digital preservation in a VRE, which could be applied to any research institution implementing a VRE and maintaining a local repository to support researchers. Many of the issues associated with the local preservation of objects over the long term could be avoided if the proposed workflow was adopted, and the British Library used as a national preservation store. A number of recommendations would, however, need to be implemented locally in order for the model to have maximum value:

• An institutional policy is required, stipulating the requirements for the deposit of research outputs in a subject or institutional repository. This should seek to comply with funding mandates and emerging guidelines from Research Councils, as well as stressing the benefit to the institution and to researchers.

• The range of material held in institutional repositories would need to be expanded to include the full range of research outputs and associated supplementary material.

• A list of preferred formats for repository objects should be produced, with explanation as to the way in which various formats will be preserved. A mechanism should also be put in place for the physical conversion of non-preferred formats to preferred formats.

• Repository items should be exposed to internal and external discovery channels on submission, unless a specific item is subject to an embargo period.

of 30


• Mechanisms also need to be established to ensure that non-digital outputs (eg: items on cassette tape) are also curated and that the metadata for these is exposed to resource discovery mechanisms.

• Metadata in repositories should be quality-assured by library staff; who should also provide training and guidelines for those inputting metadata. A sign-off process involving the researcher should be implemented.

• Common interoperability standard (eg: OAI-PMH) should be adopted to ensure compatibility with British Library systems.

• Selected repository objects should be exported to the British Library preservation store shortly after submission to maximise global discovery and the opportunity for long-term preservation.

• A procedure should be put in place for monitoring of objects that have been preserved in systems external to the institutional repository. This should include a means whereby continued access to such objects can be audited.

• The rights and restrictions on use of objects in repositories should be identified. Repositories should establish rights policies sufficient to maximise access and discovery but prevent unwanted commercial exploitation and plagiarism.

• The feasibility of implementing simple extensions to metadata schema for repositories to provide support for preservation services should be assessed.

• Repository managers should monitor developments in preservation metadata; particularly in relation to automated metadata extraction tools, preservation metadata schemas, developments to repository software to facilitate preservation, external preservation services, representation information and cost-effective methods of preservation action.

Implications and Sustainability The EVIE work is intended to be taken forward at Leeds via the development of a full staff portal, which is intended to cover the full range of staff activities including teaching, research and administration. The portal infrastructure provides the University with an exciting opportunity to build a unified digital campus, providing a seamless environment to support all aspects of University life. The Portal will enable the University to align its infrastructure with the strategic goals of the institution, by linking together people, systems and communities, as well as enabling the alignment of IT systems with academic and administrative processes. The intention is to build on the work undertaken by the EVIE Project to develop the full range of staff services and resources through the portal. A business case for this development has already been created, and we have been able to draw substantially on the EVIE experiences as evidence. The business case will be taken forward as part of the University annual planning round, and funding for development should hopefully become available for the 2007/08 session. In the interim, some specific EVIE outputs will be surfaced through the current Student Portal in order to provide services targeted at research postgraduates in particular. This work will be taken forward by our existing Student Portal Team. The EVIE Project has also had a substantial impact on the development of collaborative tools and services for researchers. Following the decision to phase out the VKP, a thorough investigation of alternative collaborative tools took place, and the Project provided considerable input into this process. A business case was developed, drawing in part on the user needs analysis undertaken by EVIE, and a pilot roll-out of Microsoft Sharepoint Servicesxl is now commencing in 2007. If successful, this will eventually be integrated with the Portal to provide seamless SSO access alongside other tools and services. The University also continues with its extended pilot of the ELGG blogging systems, which continues to be actively used by both staff and studentsxli. EVIE has also contributed to a wider awareness and use of this service. The EVIE Project has produced a set of recommendations for the implementation of a digital preservation framework within a VRE xi, not least identifying the importance of an institutional policy to identify which research outputs are deposited in local and subject repositories.

of 30


The British Library is actively exploring how it might implement preservation services for the UK higher education community as part of its involvement in other collaborative initiatives in this sector. Work undertaken during the digital preservation workpackage of the EVIE Project, which considered metadata issues and the relationship with local repositories, is making a valuable contribution to this activity.

Recommendations Several characteristics of VREs have emerged from the work of the EVIE Project. There is a requirement for highly granular flexibility of both tools and presentation, driven by recognition of the diversity of user needs and requirements the environment is designed to support. A Portal framework provides a proven architecture for a VRE, in that it has the potential to provide a high level of granularity of service and the flexibility for personalisation and customisation. VRE frameworks need to be developed with flexibility in mind. Researchers increasingly expect that they will be able to take a ‘plug-and-play’ approach, in order to integrate their own personal choice of collaborative tools, library resources etc within the VRE framework. This presents support and development challenges for institutions, in that a fine granularity of roles will be required. Researcher behaviours are extremely varied and it is unwise to attempt to categorise these by discipline. VRE frameworks therefore need to be built with the flexibility to cater to a wide range of support needs and expectations. Access to the VRE framework by external collaborators is an expectation rather than an exception, and requirements for this should be considered from the outset of any project. An external preservation services approach is recommended for the curation of data and objects within the VRE environment. Such an approach would require policy decisions to be taken at a high level within the institution, and it would also be necessary to have a detailed understanding of the long-term costs and benefits of using such a service. Taxonomy developments do not appear to be sufficiently mature to facilitate their integration within a VRE, and many of the broad taxonomies currently available would not be sufficiently detailed to enable them to be used in tagging highly specialised content. Further research on the issues associated with the use of folksonomies is required. Cultural change issues may act as a potential barrier to VRE take-up within institutions. Facilitating cultural change is highly resource intensive, and is not often costed into IT development projects. High-level institutional commitment is necessary in order for a VRE to succeed. In particular there must be a demonstrable level of commitment to ensuring that existing systems are integrated within a single environment to ensure maximum functionality. Support for embedding is required over a number of years – working practices will change gradually, and a VRE will need time to become widely embedded in institutional practices. A business case for institutional VRE development should therefore span at least five years, and sustained, long-term support for innovation and development is also required at the national level in order to provide strategic direction for institutional efforts. i VKP Report on Collaboration and Synchronisation: http://www.leeds.ac.uk/evie/papersNtalks/Research%20Groups%20and%20Synchronisation.pdf ii EVIE click-through demonstrator: http://www.leeds.ac.uk/evie/click_through/index.htm iii EVIE Channel Development Plan: http://www.leeds.ac.uk/evie/workpackages/wp3/EVIEWP3_channel_dev_plan_v8.pdf iv EVIE User Requirements Analysis: http://www.leeds.ac.uk/evie/workpackages/wp2/evieWP2_UserRequirementsAnalysis_v1_0.pdf: Appendix 2 v EVIE Channel Development Plan: http://www.leeds.ac.uk/evie/workpackages/wp3/EVIEWP3_channel_dev_plan_v8.pdf vi Authentication within the EVIE Project http://www.leeds.ac.uk/evie/workpackages/wp7/authentication_EVIE.pdf

of 30


vii CREE JAFER Project: http://www.hull.ac.uk/cree/ viii University of Leeds ELGG Installation: http://elgg.leeds.ac.uk/_weblog/everyone.php ix School of Computing, University of Leeds: http://www.comp.leeds.ac.uk/vvr/ x OCLC FAST: http://www.oclc.org/research/projects/fast/default.htm xi EVIE User Requirements Analysis: http://www.leeds.ac.uk/evie/workpackages/wp2/evieWP2_UserRequirementsAnalysis_v1_0.pdf xii EVIE Demonstrators: http://www.leeds.ac.uk/evie/evieDemo.html xiii Innovative Interfaces Inc. Metafind: http://www.iii.com/mill/digital.shtml#metafind xiv OVID Searchsolver: http://www.ovid.com/site/products/tools/searchsolver.jsp xv EVIE Development Log: http://www.leeds.ac.uk/evie/workpackages/wp8/system_documentation/development_log.htm xvi Bodington VLE: http://bodington.org/index.php xvii EVIE VRE Preservation Requirements Analysis: http://www.leeds.ac.uk/evie/workpackages/wp4/EWD-24-WP4-PR01_v4.pdf xviii EVIE Research Outputs Case Study: http://www.leeds.ac.uk/evie/workpackages/wp3/EVIEWP3_researchOutputs_mp_v1.pdf xix EVIE Research Outputs Case Study: http://www.leeds.ac.uk/evie/workpackages/wp3/EVIEWP3_researchOutputs_kn_v1.pdf xx EVIE VRE Preservation Requirements Analysis Workflow Document: http://www.leeds.ac.uk/evie/workpackages/wp4/EWD-24-WP4-PW01_v3.pdf xxi MIDESS Project: http://www.leeds.ac.uk/library/midess/ xxii ETHOS Project: http://www.ethos.ac.uk/ xxiii OCLC PREMIS Working Group: http://www.oclc.org/research/projects/pmwg/ xxiv AHDS PREMIS Project: http://ahds.ac.uk/about/projects/hybrid-archives/wp44_preservation_metadata.pdf xxv TouchGraph: http://www.touchgraph.com xxvi Google APIs: http://www.google.com/apis/index.html xxvii Gooraph: Document Visualisation of Search Results: http://www.leeds.ac.uk/evie/workpackages/wp5/EDV_09_WP5_PR01_v2.1_DocVizOfSearchResults.pdf xxviii Overview of the Keyword Extraction Component: http://www.leeds.ac.uk/evie/workpackages/wp5/keywd%20source%20docs.pdf xxix Wordnet: http://wordnet.princeton.edu xxx EVIE Demonstrators: http://www.leeds.ac.uk/evie/evieDemo.html xxxi EVIE FAST Evaluation: http://www.leeds.ac.uk/evie/workpackages/wp6/FAST_EVIE_WP6_v3.pdf xxxii EVIE Research Outputs Workshop: http://www.leeds.ac.uk/evie/workpackages/wp3/EVIEWP3_researchOutputs_wshop_v1.pdf xxxiii EVIE Taxonomy Output: http://www.leeds.ac.uk/evie/workpackages/wp6/TaxonStudy_EVIE_v10.pdf xxxiv EVIE Development Log: http://www.leeds.ac.uk/evie/workpackages/wp8/system_documentation/development_log.htm xxxv EVIE VRE Accessibility Study: http://www.leeds.ac.uk/evie/workpackages/wp3/accessibility_EVIE.pdf xxxvi EVIE VRE Evaluation Report: http://www.leeds.ac.uk/evie/workpackages/wp9/evie_evaluation_report.pdf xxxvii EVIE VRE Evaluation Report: http://www.leeds.ac.uk/evie/workpackages/wp9/evie_evaluation_report.pdf xxxviii JISC eFramework: http://www.e-framework.org/ xxxix Report of the Working Group on Virtual Research Communities for the OST e-Infrastructure Steering Group: http://eprints.soton.ac.uk/42074/ xl Microsoft Sharepoint Services: http://www.microsoft.com/technet/windowsserver/sharepoint/default.mspx xli University of Leeds ELGG Installation: http://elgg.leeds.ac.uk/_weblog/everyone.php

of 30

evie project final report - university of leeds · jisc development programmes project document...

Documents