UNCLASSIFIED

UNCLASSIFIED

DoD Metadata RegistryDoD Metadata Registry

Dr Glenda Hayesghayes@mitre.org

MITRE/DISA PEO-GES9 Oct 2008

Presentation to Presentation to Federal Data Architecture Federal Data Architecture SubcommitteeSubcommittee

UNCLASSIFIED 2

Agenda

• Net-Centricity – DoD Net-Centric Data Strategy

• What is Metadata?– Structural and Semantic Metadata– Discovery Metadata

• DoD Metadata Registry and Clearinghouse (DoD MDR)

UNCLASSIFIED 3Recipe for Agility

Net-Centric Tenets

DoDDoDNet-CentricNet-Centric

Data StrategyData Strategy

InformationInformationAssuranceAssurance

StrategyStrategy

Global ConnectivityGlobal Connectivity(Transformational(TransformationalCommunications)Communications)

Core EnterpriseCore EnterpriseServicesServices

(NCES)(NCES)

UNCLASSIFIED 4

DoD Net-Centric Data Strategy

4

TrustedTrusted

InteroperableInteroperable

ResponsiveResponsive

AccessibleAccessible

UnderstandableUnderstandable

VisibleVisibleIs an information resource discoverable by most users?

Is it available on the network, and are tools readily available to use it?

Can it be intelligibly used? Are the semantics well documented?

Can it be combined or compared with other information? Can it be mediated?

Are the source, security level and access controls of the data available to users?

Is the data what users need? Are robust user feedback mechanisms in place to improve it?

The DoD Net-Centric Data Strategy, issued by the DoD CIO May 9, 2003, is the plan to make the Department’s information resources:

CC/S/As must institutionalize processes to accomplish these goals

UNCLASSIFIED 5

Quiz: Which of these is Metadata?

An XML Schema? A WSDL?

A database structure?

A picklist?

An index card in the library?

MS-Office product’s properties?

Email classification and release conditions?

Notes on the back of a photo?

iPod playlist?

Date and reviewer for a taxonomy?

Product ratings & comments on Amazon?

UNCLASSIFIED 6

Metadata Supports NCDS Goals

Title: ~~~~~~~~~~Description: ~~~~ ~~~ ~~~

Author: ~~~ ~~ ~~~~~~

Subject: ~~~ ~~~ ~~~~

InformationResource

ResourceMetadata

Structural & SemanticMetadata

book

index card

template, dictionary, thesaurus

Analogy

=

=

=

UNCLASSIFIED

Additional questions, please use the Comments and Feedback link at http://metadata.dod.mil 7

PEO – GES Data Services Office

• Mission– Provide tools, techniques, and performance standards to enable

the DoD Data Strategy– Support DISA DoD Data Strategy compliance effort

• Products & Services– Chair DoD Metadata Working Group– Provide NCES Metadata Services support– Oversee operation of the DoD Metadata Registry– Develop and promote Data Strategy Enablers for Communities of

Interest (COIs)– Provide Data Strategy input to NCES Acquisition Milestone

Documents– Support NCES Product Branch, Testing & Evaluation,

Operations, Sustainment, and NetOps activities– Support DISA CIO in extending the Enterprise Information

Environment (EIE) technical procedures and standards for DISA internal systems

UNCLASSIFIED 8

Make Data Understandable

“I need a tank” What does it mean?

Register Terms, Definitions, & Relationships

One reason Government Agencies and Military Services have trouble operating jointly is that they speak different languages.

Registering and using taxonomies improves precision search and recall

UNCLASSIFIED 9

DoD Metadata MgmtEngineering-level Processes

DOD Developer

Namespace Managers &

WGs

DoD Metadata Registry

Participates in

Participates in

Consults, Subscribes to & Submits

to/Downloads from Metadata Registry

Namespace Managers &

WGs

Namespace Managers &

WGs

DISAData Services Office

Governs/CoordinatesWithin Namespace

Operates,

DoD Metadata WG

Maintains

Hosts

Discusses

• DoD MDR Focus Group• DDMS Focus Group• Taxonomy Focus Group

UNCLASSIFIED

• Distributed Configuration Management

• 7 “Super” Namespaces

10

MDR Governance Namespaces

UNCLASSIFIED 11

DoD MDR v7.0

Public areasPublic areas

UNCLASSIFIED

Login

12

UNCLASSIFIED

Advanced Search

13

UNCLASSIFIED

Search Results

14

ratings

Addto

Briefcase

UNCLASSIFIED

Information Resource Details

15

User-definedURLs

UNCLASSIFIED 16

Navigate through Relationships

UNCLASSIFIED 1717

MDR Support for Mediation

tacrep-2-kml.xsl(stylesheet)

tacrep.xsd(schema)

MDR OrganizesComponents for Mediation

HasXSLT

notional

Leverage Registered Components & Associations

tacrep.xml ebXML RegistryQuery service

tacrep.kml

schemaLocation=“.../tacrep.xsd”

UNCLASSIFIED 18

DoD Metadata Registry (MDR)https://metadata.dod.mil

• Purpose– Provide an on-line repository which enables developers to reuse, understand, integrate with,

and share existing data assets (metadata)• Targeting web services, databases, and vocabularies

– Mandated for the publishing of semantic and structural metadata • DOD Directive 8320.2

– Provides a portal for developer access and web services for machine-to-machine access

• Key Facts– Over 8,000 users and 180,000 assets registered– Over 900 Programs of Record supported– Serving the DoD, DHS, IC, NASA and NATO– Hosted on NIPR, SIPR, and JWICS (in progress)– User driven via DoD Metadata Working Group and Feedback links– Implements the ebXML standard for Metadata Registries (in DISR)– User Name/Password or Single Sign-On through DKO– Governance structure provided by Joint Enterprise Services ERB

• Primary Benefits– Enables reuse and governance of data assets– Foundation for other services; e.g. mediation– Allows the COI data assets to exist after the COI disbands

• Primary Audience – DoD Capability Developers, COIs

UNCLASSIFIED 19

DoD MDR History

2000 2008

COE XML Registry

DoD XML Registration

MemoDoD Metadata Registry MDR on

SIPR

2002 20062004

v2.1 v3.1 v4.1

IC Info SharingData Standards Coordination

Activity (DSCA )

v7.0

MDR Implements ebXML RIM

& RS

v5.0v1.0

v6.1

UNCLASSIFIED 20

MDR Users

• Developers – Re-use and subscribe to registered data components, and/or

register new ones they have created

• Community of Interest (COI) Metadata Governors– Configuration Manage (CM) registered component (e.g.,

posting new metadata versions, version change notification etc.)

• Acquisition Policy Makers – Use Metadata Registry metrics for acquisition oversight (e.g.,

reflecting program participation, specific data component re-use etc.)

• Applications– Interface with registered components via Metadata Registry

Web Service (ebXML) to exploit reference tables, transformations, and XML schemas at Run Time

UNCLASSIFIED 21

MDR Size and Scope

• Instances on 2 security enclaves– Unclassified and Secret

• Users – 10,186

• Information Resources – Submission Package – 1,263– Schemas (e.g., DTD, XSD, etc) – 4,814– Translations (i.e., XSL, XSLT) – 180– Services (i.e., WSDL) – 555– XML samples – 251– Taxonomies (i.e., OWL) – 178– Reference Data Sets – 5,709– Amplifying Documents (e.g., DOC, ER1, etc) – 5,268

• 900 Programs of Record, 700 Organizations

Unclassified MDR Inventoryas of 8 Oct 2008

UNCLASSIFIED 22

Making Data Visible, Accessible, and Understandable through Metadata

DataStore

Web S

ervice

NCES ServiceRegistry

Audience: developer, application program

Construct & Submit Package to MDR. Contains WSDL,

schemas, stylesheets, taxonomies & any other

pertinent information

Construct & Submit Package to MDR. Contains WSDL,

schemas, stylesheets, taxonomies & any other

pertinent information

1

Register service metadata & endpoints

Register service metadata & endpoints

DoDMetadataRegistry

Audience: warfighter, intelligence, business user

Web Interface(HCI)

Register discovery metadata IAW DDMS in Federated Search-

enabled Catalog

Register discovery metadata IAW DDMS in Federated Search-

enabled Catalog

2

MDR Submission Pkg

FederatedSearch

Interface DefenseOnline

developer

warfighter

Uses registered resources

NCESEnterprise

Catalog 3

UNCLASSIFIED 23

Net-Centric Publisher

XSDXSDWSDLWSDLDDMSDDMSServiceEndpointService

Endpoint

Simplifies Metadata Publishing

UNCLASSIFIED 24

Practical Utility for Taxonomies

• Applicable for – Content– Services– Structural Metadata

• Aid precision search via DoD Discovery Metadata Specification (DDMS)

E v e n t

P hys ica l A rea G eo grap h ica l A rea

T er ri to r ia l S ub d iv ision P h ys ica l C o m m un ity C o un try

G eo po li tica l O rga n iz a tion

L oca tion G u id an ce

D o D C o re T axo no m y

FIPS.xsdCountry.sql ISO.xsd

<ddms> : <Subject>…/DoDCore#Terrorist_Event</Subject></ddms>

UNCLASSIFIED 2525

MDR Support for Taxonomies

PoliticalOrganization

TerroristOrganization

ForeignTerroristOrganization

AlQaida

MyCOItaxonomy

DoD Core

taxonomy

<ddms> : <Subject>…/MyCOI.owl#AlQaida</Subject></ddms>

Organization

UrCOItaxonomy

TerroristGroup

Producer

View

Group

al-Qaeda

equals

equals

Consumer

View

UNCLASSIFIED 26

DoD Core Taxonomy v0.75c

UNCLASSIFIED 27

UCore v2.0 Default Taxonomy

UNCLASSIFIED 28

FEA-BRM as OWL-encoded Taxonomy

UNCLASSIFIED 29

Framework for Registry Interoperability

Inter-Registry Object References

Federated Queries

Centralized Authorityw/Local Caching

Object Relocation

Architecture Products

Enterprise SearchAggregator

Service

DDMS-based Query Service

Unclassified(U) MDR

Secret(U) MDR

MDR UDDI

Top Secret(U) MDR USTRANSCOMUSMTF

fully

automatedpush

OnlinePackage

Registration

MDR

More attention needed

(S)

(TS) (S)

MDR

Adapted from OASIS Cooperating RegistriesAdapted from OASIS Cooperating Registries

UNCLASSIFIED 30

Useful Links

• DoD CIO Data Strategy Homepage– http://www.defenselink.mil/cio-nii/datastrat/

• DoD Metadata Registry– http://metadata.dod.mil

• DoD Discovery Metadata Specification– http://metadata.dod.mil/mdr/irs/DDMS

• NCES Techguide– http://metadata.dod.mil/mdr/ns/ces/techguide/

• NCES Developer Community – https://www.us.army.mil/suite/page/384284

• COI Toolkit– https://www.us.army.mil/suite/page/479547

• Intellipedia– https://www.intelink.gov/wiki

UNCLASSIFIED 31

www.disa.mil

UNCLASSIFIED

UNCLASSIFIED

UNCLASSIFIED 32

Make Data Visible“Documents”

– Documented via• Keyword, Text, RSS/Atom, DDMS

– Metadata Cache• (traditional) Search Index,

OpenSearch• (RSS/Atom) Syndication Feed,

Google Data API• (DDMS) Metadata Catalog, Federated

Search Service

Web Services– Documented via

• WSDL + XSD(s)– Metadata Cache

• ebXML Registry• UDDI Registry

– Styles• SOAP• REST

REST = HTTP/HTTPS + URI + XML

REST

UNCLASSIFIEDDDMS: Leverages Industry Standards

33

DoD Discovery Metadata Specification (DDMS)

DDMS Attributes

Security

Title

Identifier

Creator

Publisher

Contributor

Date

Rights

Language

Type

Source

SubjectGeospatial Coverage

Temporal Coverage

Virtual Coverage

DescriptionFormat

Configuration managed by DoD Metadata WG

DDMS endorsed by Executive Order 13388

“Further Strengthening The

Sharing Of Terrorism Information To

Protect Americans”

Data Catalog

(historical)

****

*

* mandatory

OGC: GML

W3C: Date & Time

W3C: OWL

IC: ISM

W3C: Date & Time

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

UNCLASSIFIED 3434

* mandatory

DDMS endorsed by

Executive Order 13388

“Further

Strengthening The

Sharing Of Terrorism

Information To

Protect Americans”

DoD Discovery Metadata Specification (DDMS)

Configuration managed by DoD Metadata WG

Data Catalog

(historical)

Utility Beyond “Make Data Visible”

DDMS is enabler to multiple Data Strategy Goals

DDMS Attributes

Security

Title

Identifier

Creator

Publisher

Contributor

Date

Rights

Language

Type

Source

SubjectGeospatial Coverage

Temporal Coverage

Virtual Coverage

DescriptionFormat

****

*OGC: GML

W3C: Date & Time

W3C: OWL

IC: ISM

W3C: Date & Time

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

ISO: Dublin Core

Make Data Accessible (practically)

Make Data Accessible (practically)

Make Data Accessible

(responsibly)

Make Data Understandable

Make Data Understandable

Enable Data to be Trusted

Enable Data to be Trusted

UNCLASSIFIED 35

DDMS Home Pagehttp://metadata.dod.mil/mdr/irs/DDMS

Current Version: 2.0Current Version: 2.0

UNCLASSIFIED 36

Federated Search Use Case

FSFS EC

NCES Fed Search

Aggregator

Capabilities:

Enterprise Services – DECC Columbus

Interfaces:

EC

FS

Enterprise Catalog

Federated SearchExternal Applications, Services, and Data

Sources

Users

NCES Enterprise

Catalog

NCES Service Discovery

NCES Enterprise Services

Management

User Logs into DOL

Portal, DKO, or COI

Application

User Authorized by NCES Security

ServicesUser Submits Search Query

Query is federated and

results returned.

Aggregated results

returned

Results viewed by

user

Fed Search Aggregator

discovers data sources from

Service Discovery

NCES Security Service

FS FS

FS

Community of Interest

FS FS

FS FS

Program of Record

COI, POR, C/S/A Data Sources populated from applications, databases, web content, etc.

For immediate discoverability users may

post metadata to the Enterprise Catalog

Enterprise Web Content is crawled and

indexed

DS

DS DS

DS

DS DSDS

DS(Web Enabled)

DS(Web Enabled)

Data SourceDS

DECC Columbus & San Antonio

Federated Search enables information sharing within and between PORs and COIs

UNCLASSIFIED 37

Make Data Accessible

Considerations1. Practical & Responsible2. Application & Browser3. Documents & Systems

ICISM = IC Information Security MetadataRBAC = Role-based Access ControlABAC = Attribute-based Access Control

ICISM = IC Information Security MetadataRBAC = Role-based Access ControlABAC = Attribute-based Access Control

UNCLASSIFIED 38

Releasability Rules

NotionalExample

Attribute-Based Access Control

Security Clearance: SCitizenship: USRank: MajorAOR: AF

classification="TS" ownerProducer="USA GBR" SCIcontrols="SI TK" FGIsourceProtected="ISR" disseminationControls="OC REL PR" releasableTo="USA AUS CAN GBR" declassDate="2002-08-01" derivedFrom="Source Document"

UNCLASSIFIED 39

Enterprise Metadata Initiatives

• Senior Enterprise Services Governance Group (SESGG) [DoD & IC]– DoD Metadata WG

• DoD Metadata Registry (DoD MDR)• DoD Discovery Metadata Spec (DDMS)

– Service Registry & Governance WG (SR&G WG)– Content Discovery IPT

• Ucore v2.0 [DoD, IC, DHS, DoJ]

• Automated Metadata Population Service (AMPS) [AF lead]

• XML Cross Domain Services [DISA-NSA]

• Controlled Unclassified Information (CUI) – [Federal]

UNCLASSIFIED 40

UCore V2.0 Vision, Scope and Governance

Development and

Configuration Management

(DoD, DoJ Lead)

Policy Implementation

(IC Lead)

Outreach and

Communications(DHS Lead)

Executive Steering Council (ESC)

(Rotating Chair between DoD, IC, DoJ, DHS and State/Local

Representative)DoD CIO Initial Chair

Business Case

Design / Build

PilotTest and Eval

Config Mgmt

http://www.ucore.gov

UNCLASSIFIED 41

Automated Metadata Population Service (AMPS) Pilot Outcomes

• Annotators for DDMS elements– Creator– Description– Date– Format– Geospatial– Identifier– Security– Subject– Title– Type

• Annotators for Asset Types– Microsoft Office– PDF– Position-Location Indicators– Information Exchange

Schemas– HTML

InformationAsset

COIVocabularies

AMPS

ID: http://www.jfcom.mil/cac_v1.docTitle: Cyber Awareness CampaignCreator: Jim SmithSubject: cyber warfare; information assurance; non-kinetic effect; security; cyber threat; infrastructureDescription: A campaign to increase awareness of cyber threats to the DoD information infrastructure.Security: U//FOUOType: Cyber COIDate: 2008-03-31Format: Microsoft WordGeospatial: 35.8, -75.3

UNCLASSIFIED 42

Differentiating Types of Metadata

• Structural & Semantic Metadata– “Rules governing a chunk” - Name, description,

data constraints, and relationships of tags used in information resources to delimit one chunk of data from another chunk

– Artifacts where structural metadata is described: XML schemas, RDBMS structures, taxonomies

– Register in DoD Metadata Registry, use submission pkg

• Resource Metadata– “Advertisement” - Terms to aid in the recall and

retrieval of artifacts– Artifacts that we collect resource metadata on:

PPT, DOC, GIF, JPG, MPG, RDBMS– Register in “Data Catalog”, use DoD Discovery

Metadata Specification (DDMS)