Workshop onQuality/Selectivity of the DLESE

Collections

• Framing the Question• History of the Discussion

Kim Kastens, June 30, 2003

Framing the Question

• DLESE has Broad Collection and a Reviewed Collection. We are (mostly) talking about the Broad Collection.

Framing the Question• Resources enter the DLESE Broad Collection via

two routes:

– Individually, via the DLESE Cataloging Tool (the “Community Collection”)

– As part of an aggregated or themed collection, a collection accessioned into DLESE in its entirety.

• We are concerned with quality and relevance of resources entering via both routes.

Framing the Question• Anyone can submit a resource to DLESE

via the cataloging tool, which is an open set of web forms.

• This has given rise to concerns that “junk” could get into the DLESE Collections.

Framing the Question• This workshop needs to make

recommendations on two issues:

– What should be the criteria by which resources are approved for inclusion in the DLESE Broad Collection?

– What should be the procedures by which these criteria are implemented?

Framing the Question: Criteria

• Resources submitted for the DLESE Broad collection currently must meet two criteria:

– The resource is relevant to Earth System Education

– The resource works (i.e. it has no conspicuous bugs).

Framing the Question: Criteria• Other possible criteria that have been

suggested for the DLESE Broad Collection:• No cost or low cost for educational users• Resource is in English• No commercial message• No intrusive advertising• No blatant religious message• No blatant political message• No blatant errors of fact• Educational effectiveness• Well documented

Framing the Question: Procedures

• With respect to procedures, we have two issues:

– By what process shall we identify problematic resources?

– What shall we do when we find a problematic resource?

Framing the Question: Procedures

• By what process shall we identify problematic resources?

– Ask the resource contributor (current system)?

– Screening by the community? – Screening by paid staff?

Framing the Question: Procedures• What shall we do when we find a

problematic resource?

– Exclude it from the collection?

– Include it in the Broad Collection but annotate it?

This is what an annotation might look like in theDiscovery System:

Framing the Question: Procedures

Screening bypaid staff

Screening bycommunity

Problematicresourcesexcluded fromlibrary

1 2

Problematicresourcesannotated inDiscoverySystem

3 4

History of the Discussion• Coolfont: August 1999

– Collections Policy drafted

– There shall be a Reviewed Collection and an “Unreviewed” Collection

– Collection Committee established

History of the Discussion• Coolfont: August 1999 (cont’d)

– Rationale for Reviewed Collection: • Users’ Perspective: “…. recognized, efficient source for

quality teaching and learning materials.”• Creators’ Perspective: “…. a recognized stamp of

professional approval at the level of publication in a peer-reviewed journal.”

– Rationale for the “Unreviewed” Collection:• “Users are seeking materials on a huge range of topics.

The DL provides added value by being inclusive while providing powerful search and classification capability.”

History of the Discussion• Coolfont: August 1999 (cont’d)

– Criteria for Reviewed Collection:

• Accuracy, as evaluated by scientists• Importance/significance• Pedagogical effectiveness. • Well documented. • Ease of use for students and faculty• Inspirational or motivational for students• Robustness/sustainability

History of the Discussion• Coolfont: August 1999 (cont’d)

– No Criteria established for “Unreviewed” Collection

– After debate, it was decided that there would be a human-mediated step between submission of resource and ingestion into library.

History of the Discussion• Spring 2000: Academic Career

Recognition Task Force Web Survey

– Seven selection criteria for the Reviewed Collection met approval of prospective DLESE users, resource creators, and department Chairs.

History of the Discussion• Mid-late 2000: Collecting began

– DPC: testbed collection for exercising metatdata framework

– Montana State: Dave Mogk & students

– Foothill College: Chris DiLeonardo & students

History of the Discussion• October 2000: Collections Meeting at

Boulder:

– DLESE Community Cataloger tool introduced to non-DPC collecting groups (AGI, Montana State, others?)

History of the Discussion• November 2000: Steering Committee

Meeting at Lamont:

– Contentious discussion about “filters” at the gateway to the Broad Collection

– Agreement on only two of the discussed “filters”: (1) relevant to Earth System Education(2) “It works”, e.g. no conspicuous bugs

- Contentious discussion of how to apply “filters”; clarity seemed to emerge when John Snow described a “holding tank” system used in his history group.

History of the Discussion• Nov-Dec 2000: Steering Committee

Meeting at Lamont (cont’d): - Meeting Minutes:

• “The general concept of a 30-day public comment period on new resources was agreed to. This will allow a time for the community to review resources….”

• “In the short term, partners collecting resources …. will review them to make sure they are appropriate

• “….the Collections Committee, collection proposal team and the DPC will work together to investigate mechanisms for encouraging review….”

History of the Discussion• February 2001 Collections Meeting:

– Joint meeting of Collections Committee, “Collections Partners”, and Community Review System Editorial Review Board

– DLESE Community Cataloging Tool open to the world

– Collections Committee drafted Deaccession Policy

History of the Discussion• February 2001 Collections Meeting (cont’d):

– Collections Committee discussed “filters” at gateway to Broad Collection. Imperfect consensus:

• Relevance Filter – Is the resource relevant to Earth System Science education?

• Integrity Filter – Are there no blatant errors of fact in the resource? – Are there no blatant political, religious, or commercial messages in

the resource? – Does it function reasonably; i.e., seem to be basically bug-free?

History of the Discussion• April 2001 Steering Committee meeting at

Biosphere 2:

– Collections Committee/DPC Collections group presented fleshed out version of the “holding tank” or “provisional status” plan.

– Many questions and issues. Who are reviewers? How mobilized and overseen? No $ to oversee the “army of filterers.”

– No resolution.

History of the Discussion• July-Aug 2001 Steering Committee meeting at Flagstaff:

– 850 resources in library. Metadata QA streamlined.

– “Mike Mayhew indicated a concern …about the broad collection. ….Where is the quality control in developing the collection? Do we dilute the value of library with variable quality?”

– Holding tank idea revisited, in simpler form without “designated reviewers”

– Action item: “Boyd …. will develop a draft proposal/set of guidines to implement a holding tank in which resources are discoverable in the system and identified as accessioned within a 30-day period with some mechanism to accept comments. The proposal for implementation will not include a designated reviewer”

History of the Discussion

• February 2002 Steering Committee meeting at Boulder:

– Draft Collections Accession Policy presented

– Revised throughout spring

– DLESE oversight would be review of review process, rather than review of individual resources

History of the Discussion• July 2002 Steering Committee & Annual meeting

at Cornell:

– Deaccession Policy approved– Interim Collection Accession Policy approved– First annotation service demo’d within DLESE– Faulker reported that NSDL content philosophy was:

“Educational value …to be manifest in capabilities for annotation and selective filtering, rather than an accession threshhold”

– Possibility raised that annotation option might be solution to ongoing dilemma about quality of DLESE Broad Collection.

History of the Discussion• Fall 2002:

– Sumner et al focus group study of Educators’ perceptions of Quality.

– Best Practices for Resources summited to the DLESE Reviewed Collection begins to take shape.

History of the Discussion

• Spring 2003:

– Ad hoc Collections group met in Boulder, worked on how to implement Interim Collections Accession Policy and on Pathways to Reviewed Collection document

– June 13: 12 collections met documentation requirements to be accessioned as collections.