DLF/IMLS OAI Project Update & DLF Aquifer UpdateDLF Fall Forum 2006

David Seaman, DLFKatherine Kott, DLF AquiferKat Hagedorn, University of MichiganTom Habing, UIUCPerry Willett, University of MichiganLiz Milewicz, Emory University

IntroductionThe DLF/IMLS grant is in its final stages and was designed to test 2nd generation OAI tools, services and training opportunities, with guidance from scholars and technical experts: http://www.diglib.org/architectures/oai/imls2004/Includes the creation of best practices for shareable metadata, OAI data provider implementations, and tools and strategies for using and enhancing/extending the OAI protocol.A report by Martha Brogan on OAI service and data providers: http://www.diglib.org/pubs/dlf106/ .

IntroductionComprehensive training documents used in several training sessions for potential OAI data providers.Two enhanced registries for data providers and service providers.Two searchable portals that collect all DLF OAI data providers and all DLF OAI data providers that are using the MODS metadata format.An effort at clustering metadata topically for inclusion into the DLF OAI data provider searchable portal.

OAI Scholars Advisory Panel

May 23, 2006Washington DChttp://www.diglib.org/architectures/oai/imls2004/

Advisory PanelJohn Carlson, University of VirginiaGail McMillan, Virginia Polytechnic Institute and State UniversityKenneth Price, University of Nebraska, LincolnStephen Railton, University of VirginiaBruce Rosenstock, University of Illinois at Urbana-ChampaignRoy Rosenzweig, George Mason University Martha Nell Smith, University of MarylandCrandall Shifflett, Virginia Polytechnic Institute and State University Will Thomas, University of Nebraska-LincolnAllen Tullos, Emory University

AgendaReviewed research progress to date

A9Asset action packageBest practices in metadata creationThumbnailsClustering by collectionMODS portal

Discussion itemsDatesVisualizationClusteringPersonal collections

CommentsWill Thomas suggested that what has been done already outruns what the scholars can do with it. Scholars aren't being encouraged to practice digital scholarship and contributing to this is the fact that a great deal of the use of digital objects is focused on pedagogy, as are thetools. But what about scholarly research and production? Visualization is important (for different reasons, in different ways) for scholars, students and K-12 pupilsWe need the voice of the scholar matched with the technology.

Full report available at:http://www.diglib.org/architectures/oai/imls2004

/OAIAdvisors060523final.htm

Wiki of OAI Best Practiceshttp://oai-best.comm.nsdl.org/cgi-bin/wiki.pl?TableOfContents

Shareable metadata and data provider implementations sections developed extensively: soon to be published by DLF“Tools and Strategies for Using and Enhancing / Extending the OAI Protocol” section: working on before end of the grant/year

descriptions of software packages for data providers and serviceprovidersmetadata management tools, e.g., UIUC ThumbGrabber, UM OAITransform, CDL Date Normalizerlisting of registries for both data and service providersdescription of extensions to the OAI protocol

“Tools” Best PracticesOngoing!

in flux at the moment, awaiting assistance from various quarterswill continue to add and update as new information becomes availablehope that it becomes a comprehensive resource for both data providers and service providers

Requests by Institutions considering OAI Implementation

Administrative: Why implement OAI? What resources are needed (staff, equipment, time)? Technical: How can we implement data or service providers with existing infrastructure?Metadata: What are appropriate standards for shareable metadata? General: Give us clear and brief documents, we don’t have time for tomes.

OAI Training MaterialsBrief documents (1-2 pages each) Standalone pieces, focused on one element of the training program“See Also” references link the series togetherCurrent training documents online: http://www.diglib.org/architectures/oai/imls2004/training2/index.htm

OAI Training ProgramCombined overview with one-on-one assistanceFlexible structure supports varying types & levels of expertiseOAI Training Materials served multiple purposes

-----* More workshops, more consultation

Successes, Barriers, & LessonsPositive response to workshop formatOAI Training Materials served multiple purposes

-----High demand for tools info and trainingRequests for more workshops & consultationResources pose challenge to implementation

-----1-page handouts meet many informational needsFind new ways to connect OAI consultants with institutions beginning OAI implementation

A Tale of Three A Tale of Three RegistriesRegistriesPlus a few other thingsPlus a few other things

By Tom Habing By Tom Habing thabing@uiuc.eduthabing@uiuc.edu

There was the best of registries for There was the best of registries for OAI Data ProvidersOAI Data Providers……

Enhanced metadataEnhanced metadataIMLS DCC IMLS DCC Collection Description Collection Description MetadataMetadata Application ProfileApplication ProfileHumanHuman catalogingcataloging of of DLFDLF--membermember data providersdata providersData Data feedsfeeds for Michiganfor Michigan’’s OAIster services OAIster serviceBrowseBrowse by by subjectsubject termsterms

ImprovedImproved Export FormatsExport FormatsRSSRSSOAI Data ProviderOAI Data Provider

Continued growth Continued growth –– currently 1164 repositories (~40% growth currently 1164 repositories (~40% growth since start of project)since start of project)http://http://gita.grainger.uiuc.edugita.grainger.uiuc.edu/registry//registry/

There was a registry of OAI Service There was a registry of OAI Service ProvidersProviders……

An OAI Service Provider is defined as a system which An OAI Service Provider is defined as a system which is reliant on data which is harvested via the OAIis reliant on data which is harvested via the OAI--PMHPMHThe current registry doesnThe current registry doesn’’t strictly comply t strictly comply Currently about 100 entries, 40 were collected from the Currently about 100 entries, 40 were collected from the openarchives.orgopenarchives.org list or harvested via Google, the list or harvested via Google, the remainder were manually enteredremainder were manually enteredMinimal metadata, basic browse and searchMinimal metadata, basic browse and searchhttp://gita.grainger.uiuc.edu/registry/services/serviceshttp://gita.grainger.uiuc.edu/registry/services/servicesbrowse.aspbrowse.asp

There was a registry of DLFThere was a registry of DLF--member member collectionscollections……

Based on the IMLS DCC Collections RegistryBased on the IMLS DCC Collections RegistryData from the previous Michigan registryData from the previous Michigan registry

With some cleanup and enhancements by DLFWith some cleanup and enhancements by DLF

BrowseBrowseSubject, Place, Time Period, Title, Institution, Subject, Place, Time Period, Title, Institution, ……

Basic SearchBasic SearchOAI data provider for harvesting recordsOAI data provider for harvesting recordshttp://http://dlf.grainger.uiuc.edu/dlfcollectionsregistrydlf.grainger.uiuc.edu/dlfcollectionsregistry//

There was some other stuff tooThere was some other stuff too……

A9 gateway to the DLF MODS search portalA9 gateway to the DLF MODS search portalPreliminary experiments with EmoryPreliminary experiments with Emory’’s MetaCombine s MetaCombine clustering serviceclustering serviceAutomated Thumbnail Generation (Open Source Tool)Automated Thumbnail Generation (Open Source Tool)Education and OutreachEducation and Outreach

OAI Training SessionsOAI Training SessionsDistributed Services Registry Workshop at the University of Distributed Services Registry Workshop at the University of WarwickWarwickNSF NSF –– DLF DLF –– JISC/UKOLN Digital Library Service Registry JISC/UKOLN Digital Library Service Registry (DLSR) Workshop(DLSR) Workshop

Reporting in support of DLF Aquifer projectReporting in support of DLF Aquifer project

There is some time left for moreThere is some time left for more……

Allow selfAllow self--registration in the data and service registration in the data and service provider registriesprovider registriesTurn thumbnail generation into web serviceTurn thumbnail generation into web service

Some Related LinksSome Related Links

http://gita.grainger.uiuc.edu/registry/http://gita.grainger.uiuc.edu/registry/http://gita.grainger.uiuc.edu/registry/services/http://gita.grainger.uiuc.edu/registry/services/http://dlf.grainger.uiuc.edu/dlfcollectionsregistry/http://dlf.grainger.uiuc.edu/dlfcollectionsregistry/http://imlsdcc.grainger.uiuc.edu/http://imlsdcc.grainger.uiuc.edu/http://opensearch.a9.com/searches.jsphttp://opensearch.a9.com/searches.jsphttp://sourceforge.net/project/showfiles.php?group_id=47963http://sourceforge.net/project/showfiles.php?group_id=47963&package_id=159364&package_id=159364

DLF Portalhttp://www.hti.umich.edu/i/imls/

Created to showcase and experiment on the OAI repositories from DLF institutionsCurrently 59 repositories, almost 2.5 million recordsChanges to search interface, metadata tagging, and sorting happened here firstBased on recommendations from the Scholars’ Advisory Panel

MODS Portalhttp://www.hti.umich.edu/m/mods/

Created to encourage OAI data providers to create MODS recordsCurrently 5 institutions make some or all of their sets available in MODS(DSpace just made MODS available, so potentially many more to come.)Highly detailed records-- MODS:DLXSResearch into addition of thumbnails

Topical clusteringEffort to create smaller sets of metadata for ease of end-user retrievalScholars’ Panel thought topically was best to work on initiallyWorking with University of California Irvine researcher David Newman, who has a “Topic Modeling” tool

Vocabulary

Tool builds a vocabulary from the metadata values in Title, Subject and Description (oai_dc)

scientific repositories contained more consistent and enriched metadata (from previous experiment)task for this prototype will be building topics using non-scientific and scientific repositories together

Learning topicsTool uses vocabulary to “topic model” or cluster groups of like metadata records together

a. with a small number of records to learn what the clusters/topics areb. with all the records to classify them according to the learned topics

Incorporate in recordsResult will be metadata records containing topic labelsSearch and browse will be re-built to incorporate searching by topic labels

clusters/topics will be mapped to UM Library’s High Level Browse classificatione.g., as used at http://www.lib.umich.edu/ejournals/

DLF Portal: Re-engineeredAll work will be prototyped in the DLF Portal by the end of the yearPrototype will kickstart re-design of OAIster in spring 2007Suggestions on interface design welcome!

ConclusionImportance of best practices to transmit what we have learned (the hard way)Training helpful to transform the willing into active participantsTools and services critical – the registries and portals are useful even while in prototype stagesMODS rulesClose contact with scholars save us from ourselves.

DLF Aquifer HighlightsKatherine Kott

Aquifer DirectorDLF Fall Forum

November 10, 2006

Where we are

• Clear purpose within DLF mission• Key outcome

– Making content available where scholars work• Creating schemas and communities of practice for

digital library interoperation• Developing a better understanding of what services

content consumers want and need

Aquifer metaphor

Pooled collections

Course management system

Commercial search service

Federated search

Citation management

Middleware layer

Upcoming work

• Two year project to begin April 2007 adds dedicated project staff

• Implements architectural framework• Implements American Social History Online

collection through four channels• Course management system• Federated search• Commercial search service• Citation management software

• Includes assessment

Accomplishments and activities

• Redesigned the DLF Aquifer website– Products and experiments listed on Products

web page• Final release of MODS implementation guidelines• Framework diagram and explanation

– Publications in D-Lib Magazine• General DLF Aquifer article June 2006• Asset Actions article October 2006

• Cornell, USC and Yale joined

DLF Aquifer desiderata

• Funding for metadata remediation and enhancement

• Funding for cross platform tools integration or development

Within the working groups

• Adding collections– Submission agreement– Gap analysis report

• Melding two product experiments (portal and asset actions)

• Creating capacity for usability and assessment best practices work, built on existing DLF efforts

• Developing and documenting best practices for collaborative development

Connection with other DLF initiatives

• Services framework– Glossary extraction– Collaboration between services framework and

technology/architecture working group prompted work on

• Architectural development• Development method

• Synthesizing gap analysis work with OAI/IMLS grant findings

In conclusion

• Working on things DLF does well– Building communities of practice– Recommending best practices– Incubating innovation

• Continuing to map DLF Aquifer within the broader organizational context

• Accomplishing core work of the initiative through the upcoming two year project

Thank you…

For more informationhttp://www.diglib.org/aquifer/

kkott@clir.org