cwspace archiving courseware websites to dspace using a content packaging profile & web services...
TRANSCRIPT
CWSpaceArchiving Courseware Websites to DSpace
Using a Content Packaging Profile
& Web Services
©
Digital Library Federation Spring Forum, 2006
April 11, 2006 — Austin, Texas
William Reilly, Rob Wolfe — MIT Libraries
2006
DLF Spring 2006 MIT's CWSpace 2
Project Overview
Vision: Increase value in Institute publication: OCW– Also: Increase
value in Open Source platform: DSpace
* CLE = Collaborative Learning Environment
Goal: Archive OCW to MIT’s DSpace– Also: Make content
available to CLEs *
CWSpace: MIT iCampus project, nearing completion
Method: Interoperability of systems– Use Content Packaging standard– Use Web Services standard
DLF Spring 2006 MIT's CWSpace 3
Perceived Need: inter-operability™
• Obvious observation: Higher Ed. Technologies increasingly seeking integration with Library & Repository holdings
• Not so obvious(?): Libraries seeing their charter to include collection, preservation, dissemination of Teaching & Learning materials (?)
DLF Spring 2006 MIT's CWSpace 4
InterOp: What’s Available
DLF Spring 2006 MIT's CWSpace 5
InterOp: What’s Achievable 1 of 3
DLF Spring 2006 MIT's CWSpace 6
InterOp: Challenges 2 of 3
DLF Spring 2006 MIT's CWSpace 7
InterOp: Possible Resolution? 3 of 3
DLF Spring 2006 MIT's CWSpace 8
OCW Materials in DSpace = New Ground
Educational Materials Digital Archive
New kind of content: educational, teaching, learning
POLICY - Cost/merit evaluation of archival treatment. Preservation and IP considerations
New emphasis (ideally) on re-use, re-purposing: aggregation, disaggregation
GRANULARITY - Challenge to digital archive; how flexible can it be? (cf. CMS)
New kind of structure, composition: website, compound digital object, rich metadata
COMPLEXITY - Need to package multi-file content, plus metadata
Content PackagingNew needs for system-to-system access: CLEs, Image Tools
BACK-END / FRONT-END - Need for networked access
Web Services
DLF Spring 2006 MIT's CWSpace 9
Content Packaging
DLF Spring 2006 MIT's CWSpace 10
Content Packaging: Steps Toward a Profile
• Model OCW Content
• Choose Packaging Specification
• Profile That Specification
DLF Spring 2006 MIT's CWSpace 11
Content Package
Package Interchange File (PIF)
• Usually a .zip file• Contains a manifest• Manifest references
files (some in, some out of the package)
DLF Spring 2006 MIT's CWSpace 12
Object Models
• Mapping one world to another
• Content Package design considerations
• Use, re-use, subsequent disseminationOCW DSpace
SIP DIP
…
…
DLF Spring 2006 MIT's CWSpace 13
OCW to DSpace mapping
DLF Spring 2006 MIT's CWSpace 14
Which Packaging Spec.?
• Content packaging specifications are generic loose envelopes to carry content, metadata
• Choice of package is largely determined by user community
• Dspace to use METS internally• Higher Education uses IMS
Content Packaging
• METS• IMS-CP• MPEG-21
DIDL• XFDU
DLF Spring 2006 MIT's CWSpace 15
METS and IMS-CP
DLF Spring 2006 MIT's CWSpace 16
IMS-CP & OCW Content Model
DLF Spring 2006 MIT's CWSpace 17
IMS-CP & DSpace Object Model
DLF Spring 2006 MIT's CWSpace 18
OCW to IMS-CP to DSpace
DLF Spring 2006 MIT's CWSpace 19
Profiling Package Standards
Every Implementation of IMS-CP or METS will:
• Constrain the Standard• Extend the Standard• Interpret the Standard
The sum of these adjustments is the implementation profile
The Standards themselves encourage profiling in a systematic way:
IMS Application Profile Guidelines Technical Manual
http://imsglobal.org/ap/apv1p0/imsap_techv1p0.html
METS Profile Schema and Guidelines
http://www.loc.gov/standards/mets/mets-profiles.html
DLF Spring 2006 MIT's CWSpace 20
Application Profiles
DLF Spring 2006 MIT's CWSpace 21
Application Profiles
DLF Spring 2006 MIT's CWSpace 22
Application Profiles
DLF Spring 2006 MIT's CWSpace 23
Application Profiles
DLF Spring 2006 MIT's CWSpace 24
Application Profiles
DLF Spring 2006 MIT's CWSpace 25
Profiled Content Package…
Now with Package defined…
…need to move that across the network from producer to consumer… DSpace
Content Provider
DLF Spring 2006 MIT's CWSpace 26
Web Services
DLF Spring 2006 MIT's CWSpace 27
DSpace dev: LNI & PKGers
DLF Spring 2006 MIT's CWSpace 28
Lightweight Network Interface
0. What Was The Problem?
1. What Is the LNI?
2. DSpace & WebDAV
3. Example of Use: Submit OCW
4. Future Development…
DLF Spring 2006 MIT's CWSpace 29
What was the Problem…?Content Providers wanted…• Networked Access: Programmatically access the DSpace object model, but not
running on the DSpace server
Basic (Lightweight Network Interface (LNI))
[ Access, Deposit, Limited Mods. ]
• Submit
• Retrieve Item w. Metadata
• Build own U/I
• Collection Mgt, Map Items
Advanced (possible future LNI dev.)
[ Replace, Withdraw, Delete ]
• Specialist metadata
• Automated metadata
• Quality control checks
• Format migration …
DSpace
Content Provider
DLF Spring 2006 MIT's CWSpace 30
What Is the LNI?
The "Lightweight Network Interface" is:
• A remote API to control DSpace
• It's essentially WebDAV: Extensions to HTTP
– A SOAP (with WSDL) equivalent is provided
• It's a mapping: —
– a "complete and comprehensive" view of the DSpace public object
API…
(the "business logic" layer)
– …as mapped to WebDAV semantics
DLF Spring 2006 MIT's CWSpace 31
DSpace: Model• DSpace digital repository• Differs from file system• Submit URI not known• Workflow• …
DLF Spring 2006 MIT's CWSpace 32
WebDAV: Model, Basics• Each Resource
named by a URL
• Properties are "name value" pairs
• Properties are extensible
• Property values can be found, and modified
DAV = "Distributed Authoring and Versioning"
URL
DLF Spring 2006 MIT's CWSpace 33
WebDAV: Model, Fuller• Resources
can be Collections or leaf nodes
• Collections can nest
• Everything can be discovered descending from the DAV server's root '/' resource
DLF Spring 2006 MIT's CWSpace 34
DSpace Mapped to WebDAV• WebDAV
DLF Spring 2006 MIT's CWSpace 35
DSpace Implemented as WebDAV
1. Properties– WebDAV– Custom for DSpace
2. LNI URLs
3. WebDAV Methods
DLF Spring 2006 MIT's CWSpace 36
Properties: WebDAV
• DAV:creationdate• DAV:displayname• DAV:getcontentlength• DAV:getcontenttype• DAV:getlastmodified• DAV:resourcetype• …
http://www.webdav.org/specs/rfc2518.html
Akin to file system properties,with add'l. metadata
DLF Spring 2006 MIT's CWSpace 37
Properties: Custom DSpace (1 of 3)
• ALL DSpace Objects:– dspace:type
• "BITSTREAM", "ITEM", "COMMUNITY" etc.
– DAV:resourcetype • "COLLECTION" etc.
– dspace:displayname– dspace:current-user-privilege-set
DLF Spring 2006 MIT's CWSpace 38
Properties: Custom DSpace (2 of 3)
• Community DSpace Objects– dspace:logo– dspace:short_description– dspace:side_bar_text– dspace:handle– …
DLF Spring 2006 MIT's CWSpace 39
Properties: Custom DSpace (3 of 3)
• Bitstream DSpace Objects– DAV:getcontentlength = getSize()– dspace:format = getFormat().getID()– dspace:sequence_id = getSequenceID()– …
• Many more DSpace Objects (some 10)– With many more Properties (some 87)
http://wiki.dspace.org/LightweightNetworkInterface
DLF Spring 2006 MIT's CWSpace 40
URLs: DSpace LNI• Community
– http://uni.edu/dspace/dav/dso_1721.1$46
• Collection– http://uni.edu/dspace/dav/dso_1721.1$349
• Item– http://uni.edu/dspace/dav/dso_1721.1$5543
• Bitstream– http://uni.edu/dspace/dav/dso_1721.1$5543/bitstream_13
The LNI URLs for DSpace objects are only for use with the LNI.
• They are therefore not to be used:
• as persistent URLs (e.g. bookmark, store for later use)
• as interactive URLs (e.g. browser)
DLF Spring 2006 MIT's CWSpace 41
WebDAV Methods in LNI• PROPFIND - very powerful request with multiple functions:
– list names of properties– return property values– recurse through 1 or more levels of a "collection" resource.
• PROPPATCH - changes value of or deletes properties on one resource.• GET - retrieves contents of a resource• PUT - replace or add a new resource to a collection• COPY - map a resource ("Item") to a different Collection.
As noted, equivalent SOAP methods are provided
Not Implemented: Locking, Versioning.
DLF Spring 2006 MIT's CWSpace 42
Current DSpace (pre-LNI)Content Provider, currently
• Batch import (& export)
• End user Web U/I
• dspace-admin Web U/I
• SRW, OAI-PMH
DSpace
Batch Importer
OCW Comm.
"Biology (7)"
"Physics (8)"
DSpace Object API
Web U/I Web, butNon Auto-
mated
Non Std.Non Web
Workflow Adm
Item
DLF Spring 2006 MIT's CWSpace 43
DSpace with LNIContent Provider using LNI:• LNI SOAP end-point URL
• LNI WebDAV URLs
• dspace-admin (partial)
• SRW, OAI-PMH– (Not replicated in LNI)
• Still available: – Batch import (& export)– End user Web U/I
DSpace
DSpaceLNI.class
OCW Comm.
"Biology (7)"
DSpace Object API
DSpace LNI WebDAV
- Web.- Stds.
- Batch-able.
"Physics (8)"
Item
DLF Spring 2006 MIT's CWSpace 44
Submission: OverviewOpenCourseWare Client
Needs 3 Values:
1. LNI SOAP end-point URL http://userid:[email protected]/dspace-lni/lni/DSpaceLNI
2. Handle for OCW Community = hdl:1721.1/125
3. Dept. Name (MIT "Course" Number) for Course to be submitted = "Physics (8)"
DSpace
DSpaceLNI.class
OCW Comm.1721.1/125
"Biology (7)"
DSpace Object API
DSpace LNI WebDAV
Lang. -Indep.
Platform -Indep.
"Physics (8)"
Item
DLF Spring 2006 MIT's CWSpace 45
Submission: 1. LookupRequest: OCW Comm. HANDLE
LNI lookup() = http://dspace.mit.edu/dspace-lni/dav/lookup/handle/1721.1/125
Response: OCW Comm. LNI URL http://dspace.mit.edu/dav/dso_1721.1$125
DSpace
DSpaceLNI.class
OCW Comm.1721.1/125
dso_1721.1$125
"Biology (7)"dso_1721.1$197
"Physics (8)"dso_1721.1$253
DSpace Object API
DSpace LNI WebDAV
Lookup
DLF Spring 2006 MIT's CWSpace 46
Submission: 2. CollectionsRequest: PROPFIND on LNI URL for children DSpace
Collections' "displaynames":<propfind>
<prop>
<DAV:displayname/>
</prop>
</propfind>
Response: MULTISTATUS<multiStatus>
<resource>
<href URI="dso_1721.1$197"/>
<displayname>Biology (7)</displayname>
</resource>
<resource>
<href URI="dso_1721.1$253"/>
<displayname>Physics (8)</displayname> </resource>...
DSpace
DSpaceLNI.class
OCW Comm.1721.1/125
dso_1721.1$125
"Biology (7)"dso_1721.1$197
"Physics (8)"dso_1721.1$253
DSpace Object API
DSpace LNI WebDAV
Iterate…
DLF Spring 2006 MIT's CWSpace 47
Submission: 3. Submit (PUT)Request: PUT to DSpace Collection LNI URL,
including specification of Packager Type (IMS-CP):
PUT /dspace/dav/dso_1721.1$253?package=OCW-IMSCP
....package contents in body...
Response: LNI URL for DSpace Item in "Location" Header:
HTTP/1.1 201 OK
Location: http://dspace.mit.edu/dspace/dav/dso_1721.1$836
....other headers....
DSpace
DSpaceLNI.class
OCW Comm.1721.1/125
dso_1721.1$125
"Physics (8)"dso_1721.1$253
DSpace Object API
DSpace LNI WebDAV
IngestItemdso_1721.1$836
DLF Spring 2006 MIT's CWSpace 48
Submission: 4. Item HandleRequest: PROPFIND on DSpace Item LNI URL for the DSpace
Handle:
<propfind>
<prop>
<dspace:handle/> </prop>
</propfind>
Response: Properties Requested
=== PROPERTIES Successfully returned:
dspace:handle = "hdl:1721.1/836"
DSpace
DSpaceLNI.class
OCW Comm.1721.1/125
dso_1721.1$125
"Physics (8)"dso_1721.1$253
DSpace Object API
DSpace LNI WebDAV
Reverse Lookup
Itemdso_1721.1$836
1721.1/836
OCW Content Mgt. System
Stores DSpace Handle
SQL Server Table: hdl:1721.1/836
DLF Spring 2006 MIT's CWSpace 49
Submission: SummaryIn this 4-step OpenCourseWare
submission example, we've seen:• SOAP endpoint URL• Dspace Handles• Dspace LNI URLs• LNI Lookup() function• WebDAV Properties• WebDAV Methods• PROPFIND (LNI request)• MULTISTATUS (LNI response)• LNI "Reverse Lookup"• WebDAV Headers (Location;
Success/Fail)
DSpace
DSpaceLNI.class
OCW Comm.1721.1/125
"Physics (8)"1721.1/253
DSpace Object API
DSpace LNI WebDAV
Item1721.1/836
DLF Spring 2006 MIT's CWSpace 50
Future Development
• Other Use Cases, Needs– Basic, Advanced …
• Other DSpace objects– Bitstream Format Registry– Metadata Registry …
• Other WebDAV methods– PUT (for Replacement)– DELETE– MKCOL …
CWSpaceArchiving Courseware Websites to DSpace
©
Digital Library Federation Spring Forum, 2006
April 11, 2006 — Austin, Texas
William Reilly, Rob Wolfe — MIT Libraries
2006
Questions, & Etc.
http://cwspace.mit.edu/docs/ProjectMgt/Reports/MIT-CWSpace-DLF-Spring2006.ppt
V. 20060410_1510