creating using and managing metadata across the …...import and export metadata - - yes1 no/yes2...

Marten Hogeweg, Adam Martin

Creating, Using and Managing Metadata across the Enterprise

Adam Martin

Good Metadata Matters

Good Metadata Cultivates Trust (with)in the Organization

Data Creation

Map

Analysis

Derived App

Decisions

What were the sensor specs?

What’s the level of uncertainty?

When was this data last updated?

What models were used to process the data?

How would I recreate them?

Which layers are used in the map and who created them?

Whose model are we using?

How can I reproduce and validate this analysis for myself?

Outcomes

…and is important at each step as data become decisions.

Imagery Lidar

3DVector

Tabular

Real-Time

(IoT)

Big Data

Unstructured

Enabling Search, Discovery and Use

Hub Sites Exploratory

AppsDCAT

Catalog Feed

Effectively and freely sharing your derivative work with any machine or person

Open Cloud Big Data

Stores

Metadata

Hub Sites Exploratory

AppsCatalog

Feed

Public Web Services

Census – Small Business Builder

Imagery Comparison Viewer

Metadata

Big Data Stores

on an Open Cloud

HarvestersCrawlers

Bots

DCATCSW

JSON-LD

GeoServices REST APIWFS, WMTS

DHS – HIFLD Open

USDA - Forest

FAA - UAV

ArcGIS

Enabling Search, Discovery and Use

> 4,500 sites

> 264,000 public datasets

> 1,500 organizations

DCAT

/data.json

ArcGIS Hub

Catalog Interoperability: Publish Once!

How do users find your data?User Acquisition Sources

• 35% Search Google

• Click on a trustworthy looking source

• 60% of search-based users land on datasets!

• 26% Referred from a trusted source

• From an already trusted .gov website

Source: Hub Google Analytics, Jan-Dec 2019

Are people looking for “authoritative” data or sources?

The Tools Help You Create Metadata, You Are the One Creating Good Metadata

• Fill out all metadata fields on data portals

• Keep page titles no longer than 50-60 characters

• Optimize the content of abstracts

• Do not include lists of keywords [in the abstract]

• Check whether you can influence the URL

• Avoid special characters where these are not displayed correctly

• Keep the same URL if your data is updated

• Remove out-of-date pages

• Avoid duplication of metadata

• Use tools and tests to understand users

• Use tools to identify the best keywords for your subject

• Apply these recommendations to all your pages

https://www.gov.uk/government/news/new-guide-helps-find-and-classify-geospatial-datasets-across-search-engines

Marten Hogeweg

Creating Metadata in ArcGIS

Metadata is Everywhere

Portal

Author Serve Discover Use

Metadata Lifecycle in ArcGIS

• All items have metadata

• Metadata in ArcGIS is stored in ArcGIS XML

• Metadata styles determine how you display, edit, validate, export metadata

ArcMap ArcGIS Pro Portal for ArcGIS / ArcGIS Online

ArcGIS Platform

Read Write

Stored as ArcGIS XML

Standard formats

Standard formats

Flow of Metadata Publishing to ArcGIS Enterprise

• Create metadata for data in registered enterprise geodatabase

• Add data to an ArcGIS Pro map

- Layer reference’s data’s metadata

• Publish map to ArcGIS Enterprise as a feature layer

- Data with metadata is referenced

• Individual layers in the feature layer reference the data’s metadata

• Update metadata in the enterprise geodatabase, immediately available online

Flow of Metadata Publishing to ArcGIS Online

• Create metadata for data

• Add data to an ArcGIS Pro map

- Layer reference’s data’s metadata

• Publish map to ArcGIS Online as a hosted feature layer

- Data with metadata is copied

• Individual layers in the feature layer reference the copied data’s metadata

• Update metadata locally, overwrite item to make updates available online

Metadata Support in ArcGIS Desktop

ArcMap 10.x

PRO 2.0

PRO 2.1 PRO 2.2

PRO 2.3

PRO2.4

PRO 2.5

Portal/Online

Hub

Metadata styles for geospatial metadata standards

✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ -

Display and edit full metadata ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ -

Map layers have metadata - ✓

(limited)✓ ✓ ✓ ✓ ✓ ✓ -

Import and export metadata ✓ - - ✓ ✓ ✓ ✓ Yes1 No/Yes2

Manage metadata programmatically ✓✓

(limited)

✓

(improved)✓ ✓ ✓ ✓ ✓ ✓

Manage metadata with geoprocessing tools

✓ - - - - - ✓

(arcpy)- -

Publish map with full metadata ✓ - - ✓ ✓ ✓ ✓ ✓ N/A

Customize the metadata editor ✓ - - ✓ ✓ ✓ ✓ TBD 3 N/A

ISO 19115-1/-2/-3 - - - - - - ✓

(limited)TBD 4 TBD4

Metadata Support in ArcGIS Pro

• Metadata styles for geospatial metadata standards

• Display and edit full metadata

• Metadata for map layers

• Import and export metadata

• Save and filter content to files

• Copy ArcGIS metadata

• Upgrade CSDGM metadata

• Integration via Pro SDK

• arcpy.metadata!!

https://pro.arcgis.com/en/pro-app/arcpy/metadata/metadata-class.htm

Metadata Support in ArcGIS Data Reviewer

• Schema Validation

- Item Metadata

- Stand-alone Metadata File

• Metadata Element Content Validation

- Pre-defined Expressions

- Custom XPath

Metadata Support in ArcGIS Server

• ArcGIS REST Services

• OGC Services

• INSPIRE Services (via ArcGIS for INSPIRE)

• Iteminfo & Metadata

Metadata Support in ArcGIS Online and ArcGIS Enterprise

• Metadata styles for geospatial metadata standards


• Scores quality of content

• Metadata for item layers

• Export metadata

• Copy ArcGIS metadata

• Save content to file

Metadata Support in ArcGIS for Hub Sites

• Leverage existing metadata for public items


• Scores quality of metadata content

• Search metadata

• Export metadata

• Leverage metadata for additionalaccess options

Desktop Web Device

Server Online Content and Services

ArcGIS Online

Respects Item Permissions –

Robust Access Model

WFS, WMS, KML, SHP, GeoServices

Hub Sites: From a single catalog, many portals into the data

Metadata Export to Third Party Catalogs

https://enterprisecontentnew-usfs.hub.arcgis.com/data.json

• All Online Hub Sites have /data.json

• Default DCAT format

• Configurable to support US GSA Project Open Data (data.gov)

https://enterprisecontentnew-usfs.hub.arcgis.com/data.json

Marten Hogeweg

Need more tools?

Metadata Support in Esri Geoportal ServerEnabling discovery and use of geospatial resources in heterogeneous environments

• Search and view geospatial resources

• Manage, publish, and store metadata

• View live map services

• Download data

• Born 2003

• Open Source since 2010

Metadata in the JavaScript APIhttps://github.com/Esri/arcgis-js-api/tree/master/dijit/metadata

• Part of JS API

• Used in ArcGIS Online/Portal for ArcGIS

• One Page JS/HTML App

• Load/Save to Local File

• Transform/Validate Metadata

• Support any XML Specification

- GeoPlatform Profile of ISO 19115-1

- SDSFIE-M

- BLM Profile of FGDC CSDGM

- DataCite

https://github.com/Esri/arcgis-js-api/tree/master/dijit/metadata

Marten Hogeweg

Metadata Profiles and Styles

Metadata Standards ( Then → Now / Future )

Ted Haberman : hdfcgroup.org

ISO 19115 (-1) Profiles

• A metadata profile for precision agriculture based on ISO 19115 standard.

• WMO Core Profile of the ISO 19115 Metadata Standard

• Transformation of HDF-EOS metadata from the ECS model to ISO 19115-based XML

• Draft African metadata profile of ISO19115• Samoan profile• NOKIS – an ISO 19115 Based Metadata System• The UDK and ISO 19115 Standard• Latin American Profile• Núcleo Español de Metadatos (NEM)• UK GEMINI• Biological profile• Coastal zone profile• S100 Metadata profile (IHO)

• Australia-New Zealand Profile

• North American Profile of ISO 19115:2003 (NAP)

• CAN/CGSB-171.100-2009

• INCITS 453-2009

• INSPIRE Metadata Directive• ISO19115/ISO19119 Application Profile for

CSW 2.0• C3Grid ISO 19115 Metadata Profile• Marine Community Metadata Profile of ISO

19115• Energy Industry Profile of ISO 19115-1• USGIN ISO 19139 Profile• DISDI Geospatial Metadata Profile (DGMP)• National System for Geospatial Intelligence

Metadata Foundation (NMF)

Metadata Styles

• FGDC CSDGM Metadata

• Content Standard for Digital Geospatial Metadata (CSDGM)

• ISO 19139 Metadata Implementation Specification

• ISO 19115:2003 / ISO 19139:2007

• ISO 19119:2005 / OGC CSW 2008

• ISO 19110:2005 / ISO 19110:2005 Amd.1 2011 Annex E

• North American Profile of ISO 19115:2003 (NAP)

• INSPIRE Metadata Directive

• ISO 19115-1/19115-3!!

Creating Your Own Metadata Style – ArcGIS Pro Metadata Toolkit

• Document Metadata XML for the Metadata Editor

• Change Pages in the Metadata Editor1

• Change Values in Drop-down Lists in the Metadata Editor

• Change Validation Rules

• Change Elements in a Metadata Page2

• Add Custom Metadata Element to a Page and to the ArcGIS Metadata Format

• Add Custom Pages to the Metadata Editor

• Change How Metadata is Displayed

• Change the Translators Used to Import and Export Metadata3

https://github.com/Esri/arcgis-pro-metadata-toolkit

There will be code!

https://github.com/Esri/arcgis-pro-metadata-toolkit

Marten Hogeweg

Roadmap Feedback

Problem Oriented Features

Personalized Discovery

Smarter Facets

Advanced Search

Federated SearchBrowse Connected Content

Auditing page w/adv user/content search

Unified Metadata Editor (Across Platform)

INSPIRE Compliant Catalog and Item MD Export Options

ISO Compliant Catalog and Item MD Export Options

Improved Spatial Search

Automated Metadata Population

Default Metadata Options

Batch MD Editing Features

Easier MD Importing

Improved Imagery Discovery

Extendable Search

Improved Catalog Export Options

Deeper Search (MD, Semantic Relations)

GP – GeoPortal Server

DR – Data Reviewer Extensions

[Don’t] SEO My Public Data

Custom Profile Support (inc.

INSPIRE)

Print Your Certificate of Attendance

Print Stations Located in 150 Concourse Lobby

Tuesday12:30 pm – 6:30 pmExpoHall B

5:15 pm – 6:30 pmExpo SocialHall B

Wednesday10:45 am – 5:15 pmExpoHall B

6:30 pm – 9:30 pmNetworking ReceptionSmithsonian National Museumof Natural History

Download the EsriEvents app and find

your event

Select the session you attended

Scroll down to “Survey”Log in to access the

surveyComplete the survey and select “Submit”

Please Share Your Feedback in the App

creating using and managing metadata across the …...import and export metadata - - yes1 no/yes2...

Documents