1 qa focus – supporting jisc's digital library programmes quality assurance for digital...

QA Focus – Supporting JISC's Digital Library Programmes1

Quality Assurance For Digital Library Programmes:The QA Focus Approach

Brian Kelly Hamish James

UKOLN AHDS

http://www.ukoln.ac.uk/qa-focus/http://www.ukoln.ac.uk/qa-focus/

This talk describes the work of the JISC-funded QA Focus project and outlines some options for the future

This talk describes the work of the JISC-funded QA Focus project and outlines some options for the future


An Introduction to QA

Why Is QA Needed Now? In the past JISC had standards-based philosophy but:

• Approach of "let a 1,000 flowers bloom"• Diversification as no killer application then (it could

have been Gopher!) Things are different now:

• The Web is the killer application • There is now a need for interoperability (not just

self-contained independent Web sites) • JISC Programmes increasingly intended as service • development and not 'speculative' research • There is a need for publicly-funded services to be

seen to provide value for money

Intr

od

uct

ion


QA Focus

QA Focus:• Funded under the JISC 5/99 programme (CI as contact)

• Covers technical interoperability (digitisation, Web, metadata, software development & service deployment)

• Proposal for developmental approach agreed (QA Focus won't be responsible for checking project deliverables)

• Agreement to develop QA methodology which can be applied to other JISC programmes (QA Focus was funded after other 5/99 projects had started)

• Understanding that this is a new area of work (for both JISC & ourselves) with a steep learning curve

• Initially provided by ILRT and UKOLN (1.0 FTE)

• Provided by UKOLN and AHDS since 1 Jan 2003


Approach TakenTwo possible approaches to ensuring compliance with standards and best practices: Enforce

• Inspect all project’s work • Strict auditing, with penalties for no-compliance

Encourage • Developmental:

Working with community Explaining reasons for compliance Documenting examples of best practices Providing advice on implementation & monitoring Developing a QA methodology Seeking to embed QA in working practices

This may have been initial assumption in ITT

The latter approach is felt to be preferable, especially in a HE context

The latter approach is felt to be preferable, especially in a HE context


QA Focus Activities

QA Focus activities: • Focus groups to identify key concerns (in areas of

standards, implementation and service deployment) • Surveys to profile approaches, best practices and

common problems • Focussed advice primarily addressing:

Standards and best practices Implementation issues Compliance checking

• Case studies describing examples of best practices • Development of a QA methodology • Validation of approach • Embedding QA methodology within projects


Survey FindingsSurveys of compliance of 5/99 project entry points helped to profile community and identify common problems and best practices


Documentation: Advice

Advisory briefing documents are being producedThese are:

• Brief, focussed documents

• Informed by findings of the surveys

• Over 40 briefing documents have been published to date

Advisory briefing documents are being producedThese are:

• Brief, focussed documents

• Informed by findings of the surveys

• Over 40 briefing documents have been published to date


Documentation: Case Studies

Case Studies are being commissionedThese are:

• Written by projects themselves (helping

• to achieve 'buy-in') • Describe the

solution adopted to a particular problem

• Include details of lessons learnt – not just a press release!

• Over 30 published

Case Studies are being commissionedThese are:

• Written by projects themselves (helping

• to achieve 'buy-in') • Describe the

solution adopted to a particular problem

• Include details of lessons learnt – not just a press release!

• Over 30 published


Surveys For FAIR & X4L ProgrammesQA Focus now supports JISC's

FAIR and X4L programmes

We have developed a self-assessment interface for projectsThe aim is that projects will (a) check and, if necessary, fix their home page; (b) check/fix other key pages and (c) address underlying causes of problems

"I have found the webpage very useful for testing the technical compliance and accessibility of our website" "Many thanks for this - what a really useful tool!"

"I have found the webpage very useful for testing the technical compliance and accessibility of our website" "Many thanks for this - what a really useful tool!"

http://www.ukoln.ac.uk/qa-focus/surveys/fair/web-self-assessment/

QA

Fo

r W

eb


Putting it Together

We have: • Explained why projects should implement standards • and best practices (they don't always know) • Provide case studies on approaches by projects • Described implementation architectures • Described various checking tools

But how do we ensure projects : • Check deliverables for themselves • Have systematic checking procedures

There is a need to describe: • Tools & architectures for projects for check

compliance with standards & best practices • Embed systematic QA procedures

To

ols


Example - Link Checking ToolsWe should all be familiar with link checking tools. But:

• Do such tools check all types of links What about links

to external style sheet files, JavaScript files, etc.?

Are broken links really broken?

• Are such tools integrated with normal working practices?

To

ols


Describing Tools and Approaches

As an example of approach which provides easier access to testing tools, see the ,tools approach deployed on UKOLN Web site. This approach:

• Applies to all resources on Web site

• Covers HTML and CSS validation, link checking and various other tests

• Some recursive apps (,rvalidate)

• Easily implemented with single line redirect

Approach is documented at <http://www.ukoln.ac.uk/qa-focus/documents/papers/iadis-2003/>


Promoting Use Of Compliance- Checking Tools

As an example of approach which provides easier access to testing tools, see the ,tools approach deployed on UKOLN Web site.

This approach:• Applies to all resources on

Web site

• Covers HTML and CSS validation and various other tests



As an example of approach which provides easier access to testing tools, see the ,tools approach deployed on UKOLN Web site.

This approach:• Applies to all resources on

Web site

• Covers HTML and CSS validation and various other tests




Self Assessment Toolkit

Further Deliverables

We are developing a self-assessment toolkit for projects to use, by individual projects or across project clusters

The toolkit consists of: • Self-assessment questionnaires • Examples of QA procedures • Documented examples of use of testing tools • Advice on standards and best practices • Case studies • FAQs • …


Self Assessment Toolkit (1)


Self Assessment Toolkit (2)

The toolkit is also available in paper format, suitable for use in workshops, meetings, etc.

This approach will be used in the forthcoming QA workshop

Toolkits on Metadata and Service Deployment will be used


Next Steps – MetadataQA for metadata is at early stages (little documented work found). Thoughts on QA for metadata: Syntax Check syntax on embedded DC Cataloguing Rules Ensure projects have appropriate cataloguing rules Interoperability Ensure metadata can interoperate with third parties Fitness for purpose Ensure metadata is appropriate for

its purpose Change control Ensure architecture for managing

metadata can cope with change control (cf. Exploit Case Study)

See for example <http://www.ukoln.ac.uk/qa-focus/documents/briefings/#metadata>

See for example <http://www.ukoln.ac.uk/qa-focus/documents/briefings/#metadata>


Next Steps – Software

QA for software development is mature, but there may

be a need to engage in culture change within the sector

Thoughts on QA for software:

Specification Process for developing systems spec, agreeing spec with stakeholders, etc.

Methodology Documentation of systems development methodology (cf UML case study)

Testing Documentation of testing methodology

Deployment Awareness of deployment challenges

See for example <http://www.ukoln.ac.uk/qa-focus/documents/briefings/#software>

See for example <http://www.ukoln.ac.uk/qa-focus/documents/briefings/#software>


Next Steps – Service Deployment

QA for service deployment is important to protect

investment in project funding (but there is a lack of

awareness in some cases). Development Architecture Early public documentation of

development architecture

Service Architecture Awareness of likely service architecture and liaison with service

Security Awareness of security concerns

Licensing Licensing issues

ScenariosSoftware developed using cool open software environment – but JISC Service has no knowledge of environment. Resources digitised, but copyright clearance not documented. Service refuses to take delivery due to liability concerns.

ScenariosSoftware developed using cool open software environment – but JISC Service has no knowledge of environment. Resources digitised, but copyright clearance not documented. Service refuses to take delivery due to liability concerns.


The QA Focus Methodology

The QA Focus methodology which we are advising

projects (and JISC) to deploy: • Based on well-established QA principles • Seeks to be lightweight and achievable • Seeks to provide benefits to the projects themselves, as

well as (downstream) interoperability benefits to JISC programmes

Consists of: • Documented policies • Procedures for monitoring compliance with policies • Audit trails • Sharing and collaboration


Applying QA – Standards

Policy: Open standards used where possible. Proprietary formats with published spec and cross-platforms viewers and authoring tools may be acceptable

Monitoring: Project manager responsible Documentation: See QA Focus briefings 11, 31, … Exceptions: If open standards not available,

immature or costly to implement proprietary standards may be used. The project manager should document such decisions: why proprietary format was used, migration strategies to open formats in future, architecture for migration and indication of costs


Applying QA – Web Standards

Policy: The Web site will use XHTML 1.0 and CSS 2.0 standards

Architecture: The Web site will be based on XHTML templates and SSI Monitoring: New and updated pages validated using ,validate and ,cssvalidate. Every month ,rvalidate will be used

Exceptions: HTML derived automatically (e.g. Save As HTML in PowerPoint) need not comply with standards. The files will be stored in a standard directory to enable such files to be excluded from checks.


Applying QA – Accessibility (1)

Policy: The Web site will strive to attain WAI A guidelines. Consistent accessibility shortcuts will be used. An accessibility policy will be published.

Architecture: The Web site will be based on XHTML templates which comply with WAI A.

Monitoring: New and updated pages will be validated using ,bobby. A monthly batch checker will be used and audit reports published (to enable any trends to be spotted).

Exceptions: A list of permitted exceptions will be provided.

Note that University of Edinburgh Library have a useful Web accessibility policy


Applying QA – Accessibility (2)

Policy: The organisation has no accessibility policy Authors are free to implement their own accessibility shortcuts (if at all)

Architecture: No centralised policy covering authoring tools or architecture will be provided

Monitoring: No monitoring will be carried out

Note that SENDA legislation is helping to encourage organisations to develop appropriate policies and procedures

Note that SENDA legislation is helping to encourage organisations to develop appropriate policies and procedures


A Matrix Approach To Selection

Strict Compliance With Standards & Best Practices

• Projects must implement musts in standards and guidelines document

• Strict policing & penalties for non-compliance • cf. NOF-digitise programme

Deployment Of A Standards Culture • Standards document describes

recommended/preferred standards • Projects should complete selection matrix and seek

approval on choices • Selection will reflect issues such as (a) the standard

(maturity, complexity, etc.) (b) implementation issues (availability of tools) (c) organisational culture and (d) programme requirements


Recommendations

QA Focus recommendations to JISC will include: • Project proposals should describe approaches to

selection of standards, technical architecture & QA • Once approved initial project report should document:

Standards chosen (e.g. see "ESDS Web Standards Policy" case study and "Matrix for Selection of Standards" briefing document)

Justification for any deviations from open standards (e.g. see NOF-TAS FAQ)

Technical architecture Intended service deployment environment

This process is not intended to be bureaucratic: case studies show that some projects are already engaged in such activities


Validating Our Approach

Feedback on our approaches gained by publishing work in peer-reviewed papers / conferences:

• Developing A Quality Culture For Digital Library Programmes1, EUNIS 2003: gave an overview of QA Focus work (paper selected for publication in Informatica)

• Ideology Or Pragmatism? Open Standards And Cultural Heritage Web Sites1,2, ichim03: described application of QA Focus methodology for selection of standards

• Deployment Of Quality Assurance Procedures For Digital Library Programmes1,3, IADIS 2003: described deployment of QA Focus approaches in CDLR, Univ. of Strathclyde

• A Proposal For Consistent URIs For Checking Compliance With Web Standards1,3, IADIS 2003

1 Written by UKOLN and/or AHDS staff 2 Co-authored with TechDis staff 3 Co-authored with CDLR staff


Thoughts On Timeliness of QA

Early adopters

Hype

Despondency

Realism & service deployment

QA implemented

Maturity Software

Digitisation

Web

Metadata

Web Services

Semantic Web


What Of The Future? (1)

After The QA Focus Funding Ceases QA Focus Deliverables

• The QA Focus deliverables (documents, papers, tools, etc.) will continue to be hosted at a stable URL for an extended period (3-10 years)

QA Focus Recommendations • Recommendations to JISC may be implemented

in future JISC programmes and technical advice, etc. carried out by Programme Managers

• Recommendations may be ignored and(a) alternative approach deployed (b) QA issues not formally addressed



Option 2 Funding Continued

QA Focus Deliverables • The QA Focus deliverables (documents, papers, tools, etc.)

will be maintained, new areas addressed, existing areas addressed in more depth, etc.

• Remit extends beyond 5/99, FAIR & X4L QA Focus

Recommendations • Recommendations to JISC implemented in future JISC

programmes • Technical advice, involvement with programmes and

projects, etc. carried out by QA Focus team working closely with Programme Managers



Option 3 Alternative Approaches

Hardline QA • JISC wishes to implement a hardline approach to

QA • Remit of MU extended to include technical aspects

of project deliverables • No QA • JISC decides that implementing a formal QA

infrastructure is inappropriate • Projects left to 'do their own thing'

But what are the implications of these contrasting approaches?

But what are the implications of these contrasting approaches?


The Big Picture

The QA Focus approach could potentially be applied to a variety of audiences:

• QA for current and new IE programmes • QA for all JISC programmes • QA for JISC • QA for JISC RSCs • QA for JISC Services • QA for UK HEIs and FEs • QA for CIE (and links with other sectors)

Clearly this raises many issues!


Conclusions

We have described: • The approaches to QA which have been taken by

the QA Focus project • The resources which have been developed for • the community by QA Focus • A systematic methodology for QA • Options for future work in this area

1 qa focus – supporting jisc's digital library programmes quality assurance for digital...

Documents

jisc programmes qa focus

qa focus wont

fair x4l programmes

jisc funded qa focus

focus groups

fairwebselfassessment

standards best practices

web slide