digital libraries made easy 2004 samla convention roanoke, virginia november 12, 2004 edward a. fox...

Digital LibrariesMade Easy

2004 SAMLA ConventionRoanoke, VirginiaNovember 12, 2004

Edward A. FoxDigital Library Research Laboratory & Dept. of Computer

Science, Virginia Tech, Blacksburg, VA 24061

fox@vt.edu http://fox.cs.vt.edu

http://fox.cs.vt.edu/talks/2004/

Acknowledgements (Selected)• Sponsors: ACM, Adobe, AOL, CNI, CONACyT, DFG,

IBM, Microsoft, NASA, NDLTD, NLM, NSF (IIS-9986089, 0086227, 0080748, 0325579; DUE-0121679, 0136690, 0121741, 0333601), OCLC, SOLINET, SUN, SURA, UNESCO, US Dept. Ed. (FIPSE), VTLS

• VT Faculty/Staff: Debra Dudley, Weiguo Fan, Gail McMillan, Manuel Perez, Naren Ramakrishnan, Layne Watson, …

• VT Students: Yuxin Chen, Shahrooz Feizabadi, Marcos Goncalves, Nithiwat Kampanya, S.H. Kim, Aaron Krowne, Bing Liu, Ming Luo, Paul Mather, Fernando Das Neves, Unni. Ravindranathan, Ryan Richardson, Rao Shen, Ohm Sornil, Hussein Suleman, Ricardo Torres, Wensi Xi, Baoping Zhang, …

• Leonid Kalinichenko: for advice shaping this tutorial

Other Collaborators (Selected)

• Brazil: FUA, UFMG, UNICAMP

• Case Western Reserve University

• Emory, Notre Dame, Oregon State

• Germany: Univ. Oldenburg

• Mexico: UDLA (Puebla), Monterrey

• College of NJ, Hofstra, Penn State, Villanova

• University of Arizona

• University of Florida, Univ. of Illinois

• University of Virginia

Outline

1. Introduction

2. Historical Perspective

3. Topical Perspective

4. Software Solutions

5. Advanced Issues

For More Information• Magazine: www.dlib.org• Books: http://fox.cs.vt.edu/DLSB.html (1994)

• MIT Press: Arms, plus related by Borgman, Licklider (1965)• Morgan Kaufmann: Witten... (several), Lesk (2nd edition soon)

• Conferences• ECDL: www.ecdl2005.org• ICADL: http://icadl2004.sjtu.edu.cn• JCDL: www.jcdl2005.org

• Associations• ASIS&T DL SIG• IEEE TCDL: www.ieee-tcdl.org (student awards, consortium)

• NSF: www.dli2.nsf.gov• Labs: VT: www.dlib.vt.edu, http://ei.cs.vt.edu/~dlib/

Domain Concepts (theory)

DL Architecture

instance of

Modeling Language (Meta-Model)

used to compose instance of

abstracted from

represented by

interpreted as

instance of

Running DL

Actors “Real” World

“real” world object

represented by

interpreted as

Digital LibrariesShorten the Chain from

Editor

Publisher

Consolidator

Library

Reviewer

DLs Shorten the Chain to

Author

Reader

Digital

LibraryEditor

Reviewer

Teacher

Learner

Librarian

A Digital Library Case Study

• Domain: graduate education, research

• Genre:ETDs=electronic theses & dissertations

• Submission: http://etd.vt.edu

• Collection: http://www.theses.org

Project: Networked Digital

Library of Theses & Dissertations

(NDLTD) http://www.ndltd.org

DLs: Why of Global Interest?

• National projects can preserve antiquities and heritage: cultural, historical, linguistic, scholarly

• Knowledge and information are essential to economic and technological growth, education

• DL - a domain for international collaboration• wherein all can contribute and benefit• which leverages investment in networking• which provides useful content on Internet & WWW• which will tie nations and peoples together more

strongly and through deeper understanding

Libraries of the FutureJCR Licklider, 1965, MIT Press

Nation

Community

5S Definition: Digital Libraries are complex systems that

• help satisfy info needs of users (societies)

• provide info services (scenarios)

• organize info in usable ways (structures)

• present info in usable ways (spaces)

• communicate info with users (streams)

SynchronousScholarly Communication

Same time, Same or different place

Asynchronous, Digital Library Mediated Scholarly Communication

Different time and/or place

Computing (flops)Digital content

Locating Digital Libraries in Computing andCommunications Technology Space

Digital Libraries technologytrajectory: intellectualaccess to globally distributed information

less moreNote: we should consider 4 dimensions: computing, communications,content, and community (people)

D ig ita l L ib ra r y C o n te n t

A rtic le s ,R e p o rts,

B o o ks

T e xtD o cum e n ts

S p ee ch ,M u s ic

V id eoA u d io

(A e ria l)P h o tos

G e og rap h icIn fo rm ation

M o d e lsS im u la tio ns

S o ftw a re ,P ro g ra m s

G e no m eH u m a n,a n im a l,

p la n t

B ioIn fo rm ation

2 D , 3 D ,V R ,C A T

Im ag es a ndG ra p h ics

C o nte n tT yp e s

AmericanSouth.Org – Roles, ContentSOLINET Libraries (Data

Providers)Scholars

Intellectual Organization Controlled vocabulary Metadata extension

development

Collection Decisions Selection Criteria

Selection Criteria Controlled

vocabulary

Central Server Maintenance Local Server Maintenance Provision of Context

Metadata Repository Metadata Creation/Maintenance

Organizational Structure and

Annotation Tools

Central Interface Design/Maintenance

Local Interface Design/Maintenance

Selection of Other Annotation

Central Indices Creation/Maintenance

Local Indices Selection of Thesauri

Coordination of Metadata Gateway

Development

Gateway Implementation Concept Mapping

Digital Objects

Content Area Description Audio

Digital

Finding Aid

MSS Other

African-American cultural life 6 4 6 9 4 12 3 10 18 72

Agricultural crisis of late 19th century

1 1 3 1 1 4 8 19

Codification of segregation laws 1 3 2 1 1 8 16

Configuration of white supremacy 1 3 3 3 1 9 20

Cultural values and activities 3 1 5 17 4 15 1 5 20 71

Disenfranchising movements 1 2 2 1 2 1 6 15

Educational movements 6 1 1 18 6 21 3 5 27 98

Emergence of Holiness & Pentecostal Groups

1 1 1 7 10

Emergence of new musical forms 3 1 1 1 2 8

Emergence of organized groups expressing farmers concerns

2 2 1 8 13

… … … … … … … … … … …Total Each Format 41 14 51 161 38 133 13 79 301 831

Outline

1. Introduction

2. Historical Perspective• Computing-related (ACM-DL,

CSTC, CITIDEL), NSDL

• DLI, Workshop Results: Chatham

5. Advanced Issues

CS -> CSTC -> CRIM• NSF and ACM Education Committee are funding

a 2 year project “A Computer Science Teaching Center” - CSTC - http://www.cstc.org/

• College of NJ, U. Ill. Springfield, Virginia Tech

• Focus initially on labs, visualization, multimedia

• Multimedia part is also supported by a 2nd grant to Virginia Tech and The George Washington University: http://www.cstc.org/~crim/ (with curricular guidelines also under development)

CS Teaching Center (CSTC)

• Instead of building large, expensive multimedia packages, that become obsolete and are difficult to re-use, concentrate on small knowledge units.

• Learners benefit from having well-crafted modules that have been reviewed and tested.

• Use digital libraries to build a powerful base of support for learners, upon which a variety of courses, self-study tutorials & reference resources can be built.

• ACM support led to Journal of Educational Resources in Computing (JERIC), accessible from www.cstc.org

Browsing (1)

Browsing (2)

Computing and Information Technology Interactive Digital Educational Library (CITIDEL)

• Domain: computing / information technology

• Genre: one-stop-shopping for teachers & learners: courseware (CSTC, JERIC), leading DLs (ACM, IEEE-CS, DB&LP, CiteSeer), PlanetMath.org, NCSTRL (technical reports), …

• Submission & Collection: sub/partner collections www.citidel.org

www.CITIDEL.org

• Led by Virginia Tech, with co-PIs:• Fox (director, DL systems)• Lee (history)• Perez (user interface, Spanish support)

• Partners• College of New Jersey (Knox)• Hofstra (Impagliazzo)• Villanova (Cassel)• Penn State (Giles)

English

Spanish

Nominated

Editor reviewed

Multimedia

LLaanngguuaaggee TTooppiicc

QQuuaalliittyy

Identified by crawl

Peer reviewed

Algorithms

Multi-dimensional Categorization

DIGITAL LIBRARY SERVICES

REPOSITORIES

USER PORTALS

Overview of CITIDEL architecture

Annotations

OAI Data

Harvester

EDUCATORS

ADMINISTRATORS LEARNERS

Multilingual Searching

Revising Annotating Filtering Browsing Administering

Filtering Profiles User Profiles

Union Metadata

OAI Data

Provider

Remote and Peer Digital Libraries (eg. NSDL -CIS)

PORTALS

SERVICES

REPOSITORIES

Digital library architecture for localand interoperable CITIDEL services

CITIDEL Technology Features•Component architecture (Open Digital Library)

•Re-use and compose re-deployable digital library components.

•Built Using Open Standards & Technologies

•OAI: Used to collect DL Resources and DL Interoperability

•XSL and XML: Interface rendering with multi-lingual community based translation of screens and content (Spanish, …)

•Perl: Component Integration

•ESSEX: Search Engine Functionality

•Very fast, utilizing in-memory processing

•Includes snap-shots for persistence

•Multi-scheming

•Integrates multiple classifications / views through maps, closure

Cluster Search Results from CITIDEL

Cluster NDLTD-Computing

CITIDEL -> NSDL

• A collection project in the

• National STEM (science, technolgy, engineering, and mathematics) education Digital Library – NSDL

• National Science Digital Library

• www.nsdl.org

• (Next slides courtesy Lee Zia, NSF)

Supports:

Content

(profiles)

(metadata)

(protocols)

Learning communities

Customizable collections

Application services

Enables:Environments for

• Communication

• Collaboration

• Creation

• Validation

• Evaluation

• Recognition

• ...

• Discovery

• Stability

• Reliability

• Reusability

• Interoperability

• Customizability

• ...

of Resources

NSDL ProgramTracks

• Core Integration: coordinate a distributed alliance of resource collection and service providers; and ensure reliable and extensible access to and usability of the resulting network of learning environments and resources

• Collections: aggregate and actively manage a subset of the digital library’s content within a coherent theme / specialty

• Services: increase the impact, reach, efficiency, and value of the digital library in its fully operational form

• Targeted (Applied) Research: have immediate impact on one or more of the other three tracks

• Pathways: large efforts across broad ranges of areas or approaches or users

NSDL Information ArchitectureEssentially as developed by the Technical Infrastructure Workgroup

referenceditems &

collections

referenceditems &

collections

Special Databases

NSDLServicesNSDL

ServicesOther NSDLServices

CI Services

annotation

CI Services

discussion

CI Services

personalization

CI Services

authentication

CI Services

browsing

Core Services:information retrieval

Core Collection-Building Services

harvesting

Core Collection-Building Services

protocols

Core Services:metadata gathering

Portals &ClientsPortals &

ClientsPortals &Clients

Usage Enhancement

Collection Building

User Interfaces

NSDLCollections

CoreNSDL“Bus”

Outline

1. Introduction

2. Historical Perspective• Computing-related (ACM-DL, CSTC,

CITIDEL), NSDL

• DLI, Workshop Results

5. Advanced Issues

Borgman et al.:Workshop Report onSocial Aspects ofDigital Libraries: http://www-lis.gseis.ucla.edu/DL/

InformationLifeCycle

Information Life Cycle

AuthoringModifying

OrganizingIndexing

StoringRetrieving

DistributingNetworking

Retention/ Mining

AccessingFiltering

UsingCreating

AuthoringModifying

OrganizingIndexing

Storing

Archiving

NetworkingAccessing

Filtering

Creation

DistributionUtilization

Significance

Similarity

Pertinence

AccuracyCompletenessConformance

Seeking

SearchingBrowsingRecommending

Relevance

Timeliness

Accessibility

Believability

Inactive

Active

Discard

RetentionMining

Semi-Active

Preservability

Timeliness

Preservability

Describing

Benefits

• Ease of use

• Effectiveness

• “The benefits of digital libraries will not be appreciated unless they are easy to use effectively.” - IITA Workshop report

Application

Domain

Related Institutions

Examples Technical Challenges Benefit / Impact

PublishingPublishers, Eprint

archivesOAI Quality control, openness Aggregation, organization

Education

Schools, colleges, universities

NSDL, NCSTRL Knowledge management,

reuseabilityAccess to data

Art, Culture

Museum AMICO, PRDLA Digitization, describing,

catalogingGlobal understanding

ScienceGovernment,

Academia, Commerce

NVO, PDG, SwissProt, UK

eScience,European Union Commission

Data modelsreproducibility, faster reuse, faster

advance

(e) Governme

Government Agencies (all levels)

Census Intellectual property rights,

privacy, multi-nationalAccountability, homeland security

(e) Commerce

, (e) Industry

Legal institutionsCourt cases,

patents Developing standards

Standardization, economic development

History, Heritage

Foundations American Memory Content, context,

interpretation

Long term view, perspective, documentation, recording, facilitating, interpretation,

understanding

Cross-cutting

Library, Archive

Web, personal collections

Multi-language, preservation, scalability, interoperability, dynamic

behavior, workflow, sustainability, ontologies,

distributed data, infrastructure

Reduced cost, increased access, pereservation, democratization, leveling, peace, competitiveness

Reagan Moore

Ed Fox

Outline1. Introduction

3. Topical Perspective• Key concepts: do, mdo, coll, catalog, repository, service,

archive, DL

• Interoperability: federated search, harvesting, OAI

• Architecture: distributed, clusters, LOCKSS

• Digitization, preservation

5. Advanced Issues

Ourside Key Set, but Important: Interfaces

• 5S perspective: spaces, scenarios

• Taxonomy of interface components

• Workflow

• Visualization

• Environments

• Design

• Usability testing

Also Important: Epub, SGML, XML

• 5S perspective: streams, structures, scenarios

• Authoring

• Rendering, presenting

• Tagging, Markup, DOM

• Semi-structured information

• Dual-publishing, eBooks

• Styles (XSL, XSLT)

• Structure queries

Also Important: Databases

• 5S perspective: structures, streams, scenarios

• Extending database technology

• Structured and unstructured info

• Multimedia databases

• Link databases

• Performance

• Replicated storage, I2-DSI (details following)

Also Important: Agents

• 5S perspective: societies, streams, spaces, scenarios, structures

• Protocols

• Knowledge interchange

• Negotiation, registries

• Distributed issues

• Webbots (automatic indexing)

• Ontologies (standard upper)

Also Important: Economics

• 5S perspective: societies, scenarios

• E-commerce

• Sustainability

• Preservation and archiving• DLF, Besser, Lorie, Gladney

• Self-archiving

• Open collections

• Economic models, business plans

Also Important: IPR

• 5S perspective: societies, scenarios

• Intellectual property rights

• Legal issues

• Terms and conditions

• Copyright

• Patents, trademarks

• Distributed rights management

• Security

Also Important: Social Issues

• 5S perspective: societies, scenarios• Cooperation, collaboration• Annotation, ratings• Digital divide• Educational applications• Cultural heritage• Museums (AMICO)• Organizational acceptance• Personalization• Internationalization

What is Key depends on yourDL Definition

• Library ++ (library+archive+museum+…)

• Distributed information system + organization + effective interface

• User community + collection + services

• Digital objects, repositories, IPR management, handles, indexes, federated search, hyperbase, annotation

Our Perspective on Key Concepts

• Recall the 5S approach• Minimal digital library• Metamodel for minimal digital library• Metamodel for “born digital standard” DLs• Metamodel for architectural DL

• Here, focus on key concepts in minimal DL

Digital Objects (DOs)

• Born digital

• Digitized version of “real” object• Is the DO version the same, better, or worse?• Decision for ETDs: structured + rendered

• Surrogate for “real” object• Not covered explicitly in metamodel for a

minimal DL• Crucial in metamodel for archaelogy DL

Metadata Objects (MDOs)

• MARC

• Dublin Core

• RDF

• IMS

• OAI (Open Archives Initiative)

• Crosswalks, mappings

• Ontologies

• Topics maps, concept maps

Other Key Definitions

• coll, catalog, repository, service, archive, (minimal) DL

• See Gonçalves et al. in April 2004 ACM Transactions on Information Systems (TOIS)

structures (2) streams (1) spaces (4) scenarios (7) societies (10)

structural metadata specification (11)

descriptive metadata specification (12)

repository (19)

collection (17)

(20)indexing service

structured stream (15)

digital object (16)

metadata catalog (18)

browsing service (23)

searching service (21)

digital library (minimal) (24)

services (8)

sequence (A.3)

graph (A.6) function (A.2)

measurable(A.10), measure(A.11), probability (A.12), vector(A.13), topological (A.14) spaces

event (6) state (5)

hypertext (22)

sequence (A.3)

StreamsStreams Structures SpacesSpaces ScenariosScenarios SocietiesSocieties

indexingindexing

browsingbrowsing searchingsearching

servicesservices

hypertexthypertext

Structured Stream

ArchObj

ArchColl

ArchObjArchObj

ArchCollArchColl

Arch Metadata catalogArchDO

ArchDRArchDRArchDCollArchDColl Minimal ArchDL

SpaTemOrgSpaTemOrg

StraDiaStraDia

Arch Descriptive Metadata specification

Descriptive Metadata

specification

Streams

video digitalobject

Repository

CollectionCatalog

describes

stores

is_version_of/ cites/links_to

Service

Scenario

extends

reuses

ServiceManager

operationexecutes

participates_in

recipient

Scenarios

Societies

inherits_from/includes

association

Topological

ProbabilisticMetric

Measurable

Measure

describes

employsproduces

employs

produces

Structures

Spaces

Vector

contains

metadata specifications

is_a is_a

precedes

happens_before

redefinesinvokes

contains

Models Examples Objectives

Stream Text; video; audio; image Describes properties of the DL content such as encoding and language for textual material or particular forms of multimedia data

Structures Collection; catalog; hypertext; document; metadata; organization tools

Specifies organizational aspects of the DL content

Spaces Measure; measurable, topological, vector, probabilistic

Defines logical and presentational views of several DL components

Scenarios Searching, browsing, recommending

Details the behavior of DL services

Societies Service managers, learners, teachers, etc.

Defines managers responsible for running DL services; actors that use those; and relationships among them

Browsing Collaborating Customizing Filtering Providing access Recommending Requesting Searching Visualizing

Annotating Classifying Clustering Evaluating Extracting Indexing

Measuring Publicizing

Rating Reviewing (peer)

Surveying Translating

(language)

Conserving Converting

Copying/Replicating Emulating Renewing

Translating (format)

Acquiring Cataloging

Crawling (focused) Describing Digitizing

Federating Harvesting Purchasing Submitting

Preservational Creational

Add Value

Repository-Building

Information Satisfaction

Services

Infrastructure Services

SearchingBrowsing

queryanchor

Society

Collection, {digital object}

Recommending Filtering Binding Visualizing Expanding query

user model query/category {digital object}

{digital object} {digital object}

binder

InformationSatisfaction Services

space query’

fundamental

Rating Training

Infrastructure

Services (Add_Value)

composite

Requesting

handle

e e e{(digital object, actor, rate) }

p p p p p

classifier

e ee e

Indexing

transformer

Requirements Analysis Design Implementation Test

5S 5SLOO ClassesWorkflow Components

DLEvaluation

5SGraph 5SLGenFormalTheory/Metamodel

DL XMLLog

5SLGen: Automatic DL Generation

5S Meta

Model5SLGraph

DL Expert

DL Designer

5SL DL

5SLGen

Practitioner

Researcher

TailoredDL

Services

Teacher

componentpool

ODLSearch,ODLBrowse,ODLRate,ODLReview,

…….

Requirements (1) Analysis (2)

Implementation (4)

Design (3)

archive, DL

5. Advanced Issues

Interoperability through Standards

• Protocols/federation• Z39.50, CIMI• Dienst, NCSTRL• OAI protocol

• Metadata• TEI: inline, detailed (structure in stream)• MARC: two-level, fine-grained• Dublin Core: high-level, 15 elements• RDF: describing resources/collections, annotation• OAMS -> DC and others used in OAI

Interoperability and IR

• Information storage and retrieval

• Search, Retrieval, Resource Discovery

• Boolean vs. natural language

• Search engines

• Indexing, phrases, thesauri, concepts

• Federated search and harvesting, OAI

• Integrating links and ratings

• Crawlers, spiders, metasearch, fusion

Open Archives Initiative (OAI)

• Advocacy for interoperability• Standard for transferring metadata among

digital libraries• Protocol for Metadata Harvesting (PMH)

• Simplicity• Generality• Extensibility

• Support for PMH => Open Archive (OA)

OAI = Technical Umbrella forPractical Interoperability…

ReferenceLibraries

PublishersE-Print

digital libraries made easy 2004 samla convention roanoke, virginia november 12, 2004 edward a. fox...

edutalks2004 slide

dlib slide

digital libraries

deeper understanding

vt students

info services scenarios

penn state

vtls vt facultystaff

Documents

número: 19 año: 2004 el escÉptico digital sumario

ee141- spring 2004 digital integrated...

ee141-spring 2004 digital integrated...

inscripcion al samla

"samla och leverera digitalt kulturarv" 2013-10-07 (arab)

michael wright digital photography 2004

determinación de nitratos en suelo por snedd -samla ·...

the digital divide: ict development indices 2004

sp2004f lecture07-01 digital signal...

ee141- spring 2004 digital integrated circuits

ee141-fall 2004 digital integrated circuits

the digital photography revolution 235 billion digital...

digital coe digital coe 2016 1998 2004 2006 2008 2010 2012...

digital fluoroscopic imaging · 2004-07-26 · ••...

vii jornada de actualización y capacitación samla-proinsa...

the global digital divide revisited (2000-2004)

digital consumer roadshow 2004: 7400 logic replacement

the digital divide: ict development indices 2004 › en ›...

personuppgifter och lagen fou-staben 2017...

sgc en laboratorios de la red samla...trabajo final 1111...