mdeditor.org

ALASKA DATA INTEGRATION WORKING GROUP

Open Source Metadata ToolkitISO Forum and Metadata Working Group

April 11, 2018

Josh Bradley, US Fish & Wildlife Service, joshua_bradley@fws.govDennis Walworth, US Geological Survey, dwalworth@usgs.gov

mdeditor.org

Presentation Outline

●●●●

mdeditor.org

About ADIwg* (established 2010)

●○

●○○○

●○○○○

●○

*Pronounced add-ee-wig

mdeditor.org

Challenges with Standards and Tools● FGDC/CSDGM● ISO

○ 19115 (2003)/19115-2/19139○ 19115-1/19115-3○ 19157, 19119, 19110

● Ecological Metadata Language (EML)● Dublin Core● Project Open Data Metadata Schema v1.1

(Data.gov)● USGS ScienceBase sbJSON● ?????

XMLJSON

● Balancing complexity vs utility● Conversion without data loss● Adapting to changes over time● Barriers for non-technical authors● Integration with current (web) technologies

mdeditor.org

ADIwg Metadata Toolkit

*Alaska Data Integration working group (add-ee-wig)

mdeditor.org

Introducing the... ADIwg ToolKit

Defines

github.com/adiwg/adiwg-metadata

ISO, FGDC, HTML ...The ADIwg ISO 19139/19115-3 metadata profile. The mdTranslator converts mdJSON into XML adhering to this profile (and other formats, too!).

Extensible standard for encoding project anddata metadata, based on JavaScript Object Notation(JSON).

Open-source Ruby software application for translating between metadata standards. Metadata is input in one of the supported ‘reader’ formats and output in one of the supported ‘writer’ formats.adiwg.org/mdTranslator/mdtools.adiwg.org/#translator-page

Codelists for the ADIwg mdJSON and ISO metadata profile.github.com/adiwg/mdCodesmdtools.adiwg.org/#codes-page

JSON Schemas describing the mdJSON standard; Used for structural validation of mdJSON documents.github.com/adiwg/mdJson-schemasmdtools.adiwg.org

mdeditor.org

Web application for authoring and editing metadata, for both projects and datasets.

mdeditor.org

Introducing the... ADIwg ToolKit

github.com/adiwg/adiwg-metadata

ISO, FGDC, HTML ...The ADIwg ISO 19139/19115-3 metadata profile. The mdTranslator converts mdJSON into XML adhering to this profile (and other formats, too!).

Codelists for the ADIwg mdJSON and ISO metadata profile.github.com/adiwg/mdCodesmdtools.adiwg.org/#codes-page

mdeditor.org

Web application for authoring and editing metadata, for both projects and datasets.

Open-source Ruby software application for translating between metadata standards. Metadata is input in one of the supported ‘reader’ formats and output in one of the supported ‘writer’ formats.adiwg.org/mdTranslator/mdtools.adiwg.org/#translator-page

Defines Extensible standard for encoding project anddata metadata, based on JavaScript Object Notation(JSON).

JSON Schemas describing the mdJSON standard; Used for structural validation of mdJSON documents.github.com/adiwg/mdJson-schemasmdtools.adiwg.org

mdeditor.org

Metadata Standard: mdJSON● Based on JSON (JavaScript Object Notation), a

lightweight data-interchange format

● Fully defined and complete structural validation using json-schema (http://json-schema.org)

● Extendible, validator will ignore additional properties

● Uses standard GeoJSON for encoding extents

● Version 2.0 compatible with latest ISO metadata schema (19115-3)

● Version 2.5 will improve compatibility with FGDC CSDGM

Extensible standard for encoding project and data metadata, based on JavaScript Object Notation (JSON).

JSON Schemas describing the mdJSON standard; Used for structural validation of mdJSON documents.

github.com/adiwg/mdJson-schemas

Defines

mdeditor.org

Introducing the... ADIwg ToolKit

Defines Extensible standard for encoding project anddata metadata, based on JavaScript Object Notation(JSON).

Codelists for the ADIwg mdJSON and ISO metadata profile.github.com/adiwg/mdCodesmdtools.adiwg.org/#codes-page

JSON Schemas describing the mdJSON standard; Used for structural validation of mdJSON documents.github.com/adiwg/mdJson-schemasmdtools.adiwg.org

mdeditor.org

Web application for authoring and editing metadata, for both projects and datasets. github.com/adiwg/adiwg-metadata

ISO, FGDC, HTML ...The ADIwg ISO 19139/19115-3 metadata profile. The mdTranslator converts mdJSON into XML adhering to this profile (and other formats, too!).

Open-source Ruby software application for translating between metadata standards. Metadata is input in one of the supported ‘reader’ formats and output in one of the supported ‘writer’ formats.adiwg.org/mdTranslator/mdtools.adiwg.org/#translator-page

mdeditor.org

Metadata Translation Support●

●

●●

●

mdeditor.org

DesignDesign: mdJSON in => FGDC out

mdeditor.org

Design: FGDC in => mdJSON out

(FGDC CSDGM)

mdeditor.org

Introducing the... ADIwg ToolKit

Defines Extensible standard for encoding project anddata metadata, based on JavaScript Object Notation(JSON).

JSON Schemas describing the mdJSON standard; Used for structural validation of mdJSON documents.github.com/adiwg/mdJson-schemasmdtools.adiwg.org

mdeditor.org

Web application for authoring and editing metadata, for both projects and datasets. github.com/adiwg/adiwg-metadata

ISO, FGDC, HTML ...The ADIwg ISO 19139/19115-3 metadata profile. The mdTranslator converts mdJSON into XML adhering to this profile (and other formats, too!).

Open-source Ruby software application for translating between metadata standards. Metadata is input in one of the supported ‘reader’ formats and output in one of the supported ‘writer’ formats.adiwg.org/mdTranslator/mdtools.adiwg.org/#translator-page

Codelists for the ADIwg mdJSON and ISO metadata profile.github.com/adiwg/mdCodesmdtools.adiwg.org/#codes-page

mdeditor.org

mdeditor.org

mdtools.adiwg.org

● Interactive documentation tool for the ADIwg mdJSON schemas

○ Examples○ Mappings

● mdJSON Validator tool to validate using the mdJSON schemas

● Translate using the mdTranslator web API

● View codelists in HTML table, JSON, or CSV formats.

mdeditor.org

Introducing the... ADIwg ToolKit

Defines Extensible standard for encoding project anddata metadata, based on JavaScript Object Notation(JSON).

JSON Schemas describing the mdJSON standard; Used for structural validation of mdJSON documents.github.com/adiwg/mdJson-schemasmdtools.adiwg.org

github.com/adiwg/adiwg-metadata

ISO, FGDC, HTML ...The ADIwg ISO 19139/19115-3 metadata profile. The mdTranslator converts mdJSON into XML adhering to this profile (and other formats, too!).

Open-source Ruby software application for translating between metadata standards. Metadata is input in one of the supported ‘reader’ formats and output in one of the supported ‘writer’ formats.adiwg.org/mdTranslator/mdtools.adiwg.org/#translator-page

Codelists for the ADIwg mdJSON and ISO metadata profile.github.com/adiwg/mdCodesmdtools.adiwg.org/#codes-page

mdeditor.org

Web application for authoring and editing metadata, for both projects and datasets.

mdeditor.org

Metadata Editor Features● No installation, use existing web browser● By default, data stored locally only● Built-in validation and error reports● Manage multiple records and relationships● Re-use metadata components

○ metadata records○ contacts (person or group)○ data dictionaries

● Integration with mdTranslator web service● Import/Export from/to local filesystem● “Profiles” for different data types or content● Publish to external catalogs (optional)● Import extents (shapefile, kml, csv, more...)

mdeditor.org

Metadata Editor Features● No installation, use existing web browser● By default, data stored locally only● Built-in validation and error reports● Manage multiple records and relationships● Re-use metadata components

○ metadata records○ contacts (person or group)○ data dictionaries

● Integration with mdTranslator web service● Import/Export from/to local filesystem● “Profiles” for different data types or content● Publish to external catalogs (optional)● Import extents (shapefile, kml, csv, more...)

mdeditor.org

Metadata Editor Features● No installation, use existing web browser● By default, data stored locally only● Built-in validation and error reports● Manage multiple records and relationships● Re-use metadata components

○ metadata records○ contacts (person or group)○ data dictionaries

● Integration with mdTranslator web service● Import/Export from/to local filesystem● “Profiles” for different data types or content● Publish to external catalogs (optional)● Import extents (shapefile, kml, csv, more...)

mdeditor.org

Metadata Editor Features● No installation, use existing web browser● By default, data stored locally only● Manage multiple records and relationships● Built-in validation and error reports● Re-use metadata components

○ metadata records○ contacts (person or group)○ data dictionaries

● Integration with mdTranslator web service● Import/Export from/to local filesystem● “Profiles” for different data types or content● Publish to external catalogs (optional)● Import extents (shapefile, kml, csv, more...)

mdeditor.org

Metadata Editor Features● No installation, use existing web browser● By default, data stored locally only● Manage multiple records and relationships● Built-in validation and error reports● Re-use metadata components

○ metadata records○ contacts (person or group)○ data dictionaries

● Integration with mdTranslator web service● Import/Export from/to local filesystem● “Profiles” for different data types or content● Publish to external catalogs (optional)● Import extents (shapefile, kml, csv, more...)

mdeditor.org

Metadata Editor Features● No installation, use existing web browser● By default, data stored locally only● Manage multiple records and relationships● Built-in validation and error reports● Re-use metadata components

○ metadata records○ contacts (person or group)○ data dictionaries

● Integration with mdTranslator web service● Import/Export from/to local filesystem● “Profiles” for different data types or content● Publish to external catalogs (optional)● Import extents (shapefile, kml, csv, more...)

mdeditor.org

Metadata Editor Features● No installation, use existing web browser● By default, data stored locally only● Manage multiple records and relationships● Built-in validation and error reports● Re-use metadata components

○ metadata records○ contacts (person or group)○ data dictionaries

● Integration with mdTranslator web service● Import/Export from/to local filesystem● “Profiles” for different data types or content● Publish to external catalogs (optional)● Import extents (shapefile, kml, csv, more...)

mdeditor.org

Metadata Editor Features● No installation, use existing web browser● By default, data stored locally only● Manage multiple records and relationships● Built-in validation and error reports● Re-use metadata components

○ metadata records○ contacts (person or group)○ data dictionaries

● Integration with mdTranslator web service● Import/Export from/to local filesystem● “Profiles” for different data types or content● Publish to external catalogs (optional)● Import extents (shapefile, kml, csv, more...)

mdeditor.org

Toolkit Implementation

● Integration into existing systems● mdJSON can be used independently

as a validated format for data transfer

● mdJSON schemas can be extended● mdTranslator can be used without

mdJSON

● Only requires a web browser● Focus on metadata content● Custom profiles may be used to

simplify user experience● Easy to manage metadata by project● Share and co-author metadata● One editor for all content types

mdeditor.org

LCC* Network Science Catalog: Case Study● Collate all projects and

associated data products

● Adopt Alaska Data Integration working group (ADIwg) standards and tools

● Build integrated workflow with ADIwg tools and ScienceBase

○ mdJSON is common format

○ mdTranslator integrated into ScienceBase

● Currently supporting all 22 LCCs & Network office

● Arctic LCC Project Tracking System exports mdJSON

Project TrackingSystem

ScienceBase

LCC Network Science Catalog

ISO XML19115-2

*Landscape Conservation Cooperatives, www.lccnetwork.org

sbJSON

mdeditor.org

Transitioning to ISO

FGDC CSDGMor

FGDC CSDGM

or

ISO 19115-2

ISO 19115-2

mdeditor.org

Transitioning to ISO: new 19115-1 writer

ISO 19115-1

ISO 19115-2

ISO 19115-2

FGDC CSDGMor

FGDC CSDGM

or

ISO 19115-1

mdeditor.org

Projected TimelineDeliverable Due Date

Release 1.0 mdEditor (now in beta) 7/1/2018

Release FGDC CSDGM reader Done!!!

Release FGDC CSDGM writer Done!!!

Complete User Documentation for mdEditor 6/1/2018

Release ISO 19115-3 writer 8/1/2018

Release 1.1 mdEditor● Taxonomy will be supported● Optional database storage (CouchDB)

Fall 2018

mdeditor.org

Resources●

○○○

●○○○

●○○○

mdeditor.org

Summary

●●●●

Josh Bradley, US Fish & Wildlife Service, joshua_bradley@fws.govDennis Walworth, US Geological Survey, dwalworth@usgs.gov

mdeditor.org

Questions?