smart searches … smarter metadata

Smart searches … Smarter metadataEvan Bailey and Sue Carpenter

Knowledge Sharing Services

Outline

Testing of intelligent search

Does it provide an alternative to human-assigned metadata?

Background

Our users

The Teaching and Learning Exchange

Our metadata standards

Processes for author-created metadata capture

2/22

Our users

NSW public school teachers

TAFE NSW teachers

What we know about what they want

Easy to access online learning resources

Re-usable components

Mapped to NSW curriculum

Like Google

3/22

TaLe | Teaching and Learning exchange

Online delivery of learning resources for NSW Department of Education

Schools

TAFE

Parents and Community

Search and browse by learning area and educational level

Easy preview and download

4/22

5/22

6/22

7/22

8/22

Our metadata

Metadata underpins TaLe

Standardisation

Interoperability

DET Learning Resource Metadata

NSW DET vocabularies

Crosswalks

9/22

Making metadata invisible

Workflow integration

Content creators capture metadata

Easy to use template - The Learning Reference Repository

Automated and default values

QA and training by librarians

10/22

The Learning Reference Repository

Easy to use metadata capture template

Workflow integration with resource publishing system

Describe and publish resources

11/22

12/22

13/22

14/20

Getting buy-in

High level of compliance

Easy to use tool for metadata capture

Workflow integration

Quick feedback in TaLe search results

High quality author-created metadata

They get it! Metadata matters!

Captures subject expert’s knowledge – efficient and effective

15/22

The promise of the new

Intelligent search engines

Make sense of unstructured information

Conceptual and contextual understanding

Related concepts

Improved efficiency?

Let the machine do it!

Eduction

Generation of metadata values by the search engine

16/22

Proof of concept testing

Can search engine eduction provide a satisfactory alternative to human-assigned metadata on Curriculum Support site, in terms of assigning metadata values for stages of learning and key learning areas?

Test environment

Curriculum Support resources (24,000)

Test cases Comparison with human-assigned metadata Review of search engine-assigned values

17/22

Results

Search engine eduction alone not reliable enough

Did not assign values where it should have – false negatives 30%

Assigned values where it should not have – false positives 25%

Users would lose confidence in search

Time and resources needed to train the engine

Aid human-assigned metadata – not a replacement

18/22

Characteristics of educational resources

Student resources

Context not always explicit

Trigger terms not present

19/22

The value of evidence

Strengths of the existing process

Content experts = Expert taggers

Easy capture tool

Metadata standardSpecific terminology for NSW curriculum

Metadata adds value

20/22

What’s next

Watching brief and more testing

Intelligent search with intranet content

Implement related concepts

Social tagging

21/22

More … for later

TaLe http://www.tale.edu.au

Who are we?

The Centre for Learning Innovation http://www.cli.nsw.edu.au/cli/index.shtm

Knowledge Sharing Services http://www.cli.nsw.edu.au/cli/focus/standards.shtm

evan.bailey@det.nsw.edu.ausue.carpenter@det.nsw.edu.au

Our metadata profile

The DETLRM22/22