semtech2011e - niem ontologies and vocabularies - topquadrant

7/27/2019 SEMTECH2011E - NIEM Ontologies and Vocabularies - TopQuadrant

1/65

Click to edit Master title style

Copyright 2011 TopQuadrant Inc 1

NIEM Ontologies and Vocabularies

Transforming NIEM to RDF/OWL andQuerying NIEM-compliant Instance Data

using SPARQL and SPIN

Ralph Hodgson, CTO, TopQuadrant

Gokhan Soydan, Semantic Solution Developer, TopQuadrant

SemTech 2011 East, Thursday, December 1, 2011, 3:00 PM - 3:50 PM

Level: Technical Intermediate

Location: Auditorium


2/65



at s t ere to ta a out, sthere enough time?

Business and Technical

Motivations

Approaches to model-based

Information Exchange using

controlled vocabularies

Expressing NIEM as OWL

Models and Vocabularies

The Power of RDF/OWL and

SPARQL

Next Possibilities

Reusable Message Building blocks

Composable Message Schemas

Controllable Vocabularies

Linked Data

Information Insight

UML XML Schema

UN/CEFACT CCTSOWL XML Schemas

OWL and Turtle/JSON-LD

XSD to OWL Transformation

U.S. DOJ Logical Entity Exchange

Specification 3.1 (LEXS) XML Instance Messages to RDF Conversion

SPARQL inferencing over LEXS Messages

Demonstration

NIEM as LOD

Take Away


3/65


First lets remindourselves on:

why information is

exchanged

Sculpture by M. Chava Evans (Baltimore, MD)

Sculpture, Studio 33, Torpedo Factory,

Alexandria, VA


4/65


technical motivations

Sculptures in the National Gallery, East Building, Washington DC, Nov 25, 2011

XML OWL

UML?

OWL as a specificationlanguage for information

models and controlled

vocabularies


5/65


But life in the XML Ecology isnt easyfrom hierarchies

to Graphs

from Graphs

to hierarchiesmore at http://topquadrantblog.blogspot.com/2011/09/living-in-xml-and-owl-world.html

b k h h
http://topquadrantblog.blogspot.com/2011/09/living-in-xml-and-owl-world.htmlhttp://topquadrantblog.blogspot.com/2011/09/living-in-xml-and-owl-world.htmlhttp://topquadrantblog.blogspot.com/2011/09/living-in-xml-and-owl-world.html


6/65



Some breakthroughs:Co-existence of OWL and XSD/XML

+TopBraidTransformers

Convert XSD to RDF/OWL

XSD

RDF/OWL

TopBraid

Transformers


XSD

RDF/OWL

TopBraid T ransformers


XSD

RDF/OWL

Semantic XML

Convert XML to RDF/OWL

XML

RDF/OWL++

Make OWL Schemas from NIEM and LEXS XSD Schemas1

2 Use the OWL Schemas to make RDF from LEXS XML Messages


7/65



SPARQL Rules (SPIN)

Convert RDF/OWL to XML

XML

RDF/OWL

SPARQL Web Pages

(SWP)

Convert HTML to PDF

PDF

HTML

ReportingHubSemantic Processing

SPARQL Rules (SPIN)


XML

RDF/OWL

SPARQL Web Pages (SWP)

Convert RDF/OWL to HTMLHTML

RDF/OWL


8/65



Generating XML Schemas and ControlledVocabularies from OWL Models

GRDDL XSLT

Generator

XSLT Processor

Going fromXML to OWL

ref: XML SchemaPlus http://www.xspl.us


9/65



Different Reasons to Connect the Dots

1) 360 Degrees View

2) Transitive Connections

3) Information DiscoveryC

More about the same thing

A

B What is linked to a thing of

interest

A Find things that share common

attributes or relationships

l


10/65



Personal Motivations:August 1, 2009Data Independence Day

www.oegov.org


11/65 Copyright 2011 TopQuadrant Inc 11

Current practices for Living in the XML

Ecology raise many challenges:

X

X

1. Vocabulary Alignment

2. Governance of core models

3. Extensibility and tailoring of

models to local needs

4. Resilience to change


12/65



Some ways XML Message Schemas have been,

or are being, made using UML (1 of 5)

1 The Weather Data Model

ref: WXXM 1.1 Primer, 1.1 10 February 2010, https://wiki.ucar.edu/display/NNEWD/WXXM

Take Away

No URIs No inherent aggregation properties Special programs Complex queries


13/65





2 CIM Models in the SmartGrid

ref: EPRI CIM and 61850 Harmonization 2009 Project Report, Nov 17, 2009,

http://cimug.ucaiug.org/Meetings/Charlotte2009/Presentations/CIM%20and%2061850%20Harmonization%20102909.pdf

Take Away


14/65





3 Harmonizing Spatial Data NEN 3610:2011 and GML

ref: http://www.nen.nl/web/Normshop/Norm/NEN-36102011-nl.htm

Configuration(XML)

GMLApplicationSchema

(XML Schema)

ShapeChange(Java, Servlet)

UMLApplicationSchema(XMI)

Configuration(XML)

GMLApplicationSchema

(XML Schema)

ShapeChange(Java program)

UMLmodel

EncodingRules

Guidelines

/

Take Away



15/65



Some of the ways XML Message Schemas have

been, or are being, made using UML (4 of 5)

4 UN/CEFACT Standards for Message Exchange

/

Source: 16th UN/CEFACT PLENARY

http://www.unece.org/fileadmin/DAM/cefact/cf_plenary/plenary10/UNCEFACT%2016TH%20PLENARY_full_rev5.ppt

Take Away



16/65


17/65 Copyright 2011 TopQuadrant Inc 17

TopQuadrant has faced the

OWL co-existence with UMLand XML challenges on anumber of projects

SmartGrid Semantic

Harmonization and

Interoperability

NASA Telemetry and

Command, Simulationand Data Architecture

Models and

VocabulariesThe Netherlands MoJ

Ontology-Driven MetadataWorkbench Message

Builder

EPIM Reporting Hub for

the Norwegian Oil and

Gas Fields

Th N th l d M J O t l D i


18/65



The Netherlands MoJ Ontology-Driven

Metadata Workbench Message Builder

Business Needs

Accurate and rapid Information Sharing

between Organizations

Agility in response to Legislation

Changes

Data Quality is guaranteed

Reduced Costs of Message Schema

Development

Technical Benefits

Direct and flexible Reuse of Data

Components

Full Automation of XML Schema

creation

Semantic Consistency is preserved and

confirmed

Linked Data / traceability

Version Managementref:http://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-

workbench-composing-xml-message-schemas-owl-models.htm

Take Away

h h l d l
http://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htmhttp://www.enterprisedatajournal.com/article/netherlands-ministry-justice-metadata-workbench-composing-xml-message-schemas-owl-models.htm


19/65



The Netherlands MoJ Ontology-Driven

Approach to Message Design using

UN/CEFACTSolution: Ontology-Based Metadata Workbench:Transform Domain Models into UN/CEFACT CCTS compliant representation and allow Business

Analysts to assemble business documents for electronic messages from Component Parts.

Take Away

i f h


20/65



Rich

Ontologies

CCTS Ontologies

Core Component

Overlay

Creation of XML Message Schemas

Contexts

Domains

Business

DocumentOntologies

CCTS

MetaModel

CCTS

Document

SPIN Transformation rules

CCTS XML

SchemaPlus

CCTS XML

Schema

XSP

MetaModelXSLT Script

Business

Component

Overlay

Rich Ontologies areexpressive models of

domains. These include

LKIF and detailed situations

of law and legal document

and procedures.

CCTS-Compliant XML Schemas

are generated from the XSP

Document

CCTS Document Editor XSP Generation XSD Generation

Users create CCTS documents

from BIEs and Core

Components

Projects

Acronyms

BIE Business Information Entity

CCTS UN/CEFACT Core Component Technical

Specifications

LKIF Legal Knowledge Interchange Format

SPIN SPARQL Inferencing Notation

XSLT XSL Transformations (XSLT) Version 2.0

XSP XML SchemaPlus

Take Away


21/65



NASA Constellation Program

CxP 70160 ANX10

Infrastructure

Specification

CxP 70160 ANX11

Application Programming

Interface Specification

CxP 70160 ANX14

Policy and Security

Model

Constellation Program Data Architecture

and Interoperability through the use of OWL

Ontologies with strategies for co-existence

with XML and other data formats.

Take Away


22/65

l k d l l

b


23/65



ReportingHub Vision

Need:

Reporting to authorities and

partners on the NCS in a costefficient and secure manner

Outcome:

Improved Information

Integration and ExchangeFaster and better decisions

Enablers:

A Field Specific Asset Model based on the Common Asset Model

ISO 15926, PCA RDL and NPD Facts

SPARQL as a way to query the data in a triple store and reason

about data using appropriate inference engine(s)

Web Services for hiding the complexity of SPARQL Queries

Machine driven creation of new data relationships without

restructuring the data modelSPARQL Rules (SPIN)Convert XML to RDF/OWL

XML

RDF/OWL

1500 named users, and

100 concurrent users


Convert HTML to PDF

PDF

HTML


Convert RDF/OWL to PDF

PDF

RDF/OWL

Take Away

Cli k di i l l

ReportingHub


24/65



SPARQL Rules (SPIN)

Convert RDF/OWL to XML

XML

RDF/OWL

SPARQL Web Pages

(SWP)

Convert HTML to PDF

PDF

HTML

ReportingHubSemantic Processing

SPARQL Rules (SPIN)


XML

RDF/OWL


Convert RDF/OWL to HTML

HTML

RDF/OWL

Take Away


25/65


The NIEM/LEXS Experiment

Cli k di M i l l

Th NIEM/LEXS E i


26/65



The NIEM/LEXS Experiment

From NIEM/LEXS XSD

Schemas and Instance Data

To OWL Models and RDF Triples

NIEM/LEXS

RDF/OWL Stack

VAEM, VOAG, VOID, DC

LEXS Rules

LEXS Instances

DTYPE

NIEM Vocabs and

Datatypes

NIEM Ontologies

LEXS Ontology


27/65


28/65


29/65


What we will show you today

Generation of OWL Models from

XML Schemas

Auto-conversion of LEXS-based XMLmessages to RDF

An experiment with fake (generated)

Incidents data to show how multiplemessages can be aggregated

Some SPARQL Queries and SPIN rules

at work


30/65


31/65


32/65


DEMO of XSD to OWL and XML to OWLTransformations


33/65

NIEM Person (Proto) OWL Model


34/65

Copyright 2011 TopQuadrant Inc. Slide 34

NIEM Person (Proto) OWL Model

Note: to address the reusability required in the MoJ work, NIEM

Person was re-factored into individual Details classes.

Take Away

Refactoring of NIEM Person into an OWL Model


35/65


Refactoring of NIEM Person into an OWL Model

with reusable Concepts (person:Details)

Depending on the context of use, concepts describing different

details about a person can be selected for the UBL Business

Documents and Messages.

Take Away

Refactoring of the NIEM Person into an OWL Model


36/65


Refactoring of the NIEM Person into an OWL Model

with reusable Concepts (person:AppearanceDetails)

A persons Appearance Details will be needed for

criminal investigations.

Take Away

NIEM JXDM Complex Type Example


37/65


NIEM JXDM Complex Type ExampleTake Away


38/65


DOJ Logical Entity Exchange Specification(LEXS)


What is the DOJ LEXS?


39/65



What is the DOJ LEXS?

LEXS provides a flexible, NIEM-based framework used

for the creation of NIEM-conformant IEPDs for

information sharing, both for publishing information

and for system-to-system federated searches.

source: http://it.ojp.gov/default.aspx?area=implementationAssistance&page=1017&standard=486

LEXS is a family of NIEM-conformant

IEPDs that define flexible structures to

support a variety of applications.

Any application that participates in

OneDOJ, is a part of LEISP, or supports

law enforcement information sharing

must participate in LEXS exchanges.

If additional structures beyond the

base LEXS are required, LEXS should

be extended by using NIEM (Option 2).


40/65


Using SPARQL to count the properties on the


41/65



Using SPARQL to count the properties on theLEXS/NIEM OWL Models

SELECT ?class (COUNT (DISTINCT ?p) AS ?properties)

WHERE {

?class a owl:Class .

OPTIONAL {

?class rdfs:subClassOf ?r .

?r a owl:Restriction .

?r owl:onProperty ?p .}

}

GROUP BY ?class

ORDERBYDESC( ?properties )


digest:EntityAssociationType really stands


42/65



digest:EntityAssociationType really standsout with 194 Properties

Is this a refactoring opportunity?


Some NIEM Controlled Vocabularies


43/65



Some NIEM Controlled VocabulariesFBI


44/65


The digest:EntityActivity Class


45/65



The digest:EntityActivity Class

OWL Class with properties

Inheritance

Association

A digest:EntityActivity is both a digest:Entityand a digest:EntityActivityType

Multiple Inheritance is common

Note that the proto-OWL ontology

respects the XML Schemas use ofwrapped data types. An optimization can

unfold these to direct data types

Association


A digest:EntityActivity Instance


46/65



A digest:EntityActivity Instance

1997-03-12

000000000003

Incident1997-03-12T00:01:00.0ZOn 3/12/1997 at 12:01 a.m., Mr. Donald R. Duck (Witness 1) saw a white male break the glass of his neighbor's (Jacob Joe) front

door. Mr. Duck placed a 911 call on his cell phone to report the incident. Within minutes, police arrive at the residence (1 NW Brockway Avenue) to find the subjectransacking the house. Detective Bond was the responding and arresting officer. The subject was taken to the Santa Fe Police Department and placed under arrest. An arrestreport was filed on 3/12/1997.

Class

Instance

Take Away


Transforming LEXS Instance Data to RDF


47/65



burglary-incident-w-arrest-basic-lexs.xml

Transforming LEXS Instance Data to RDF

Semantic XML

ConvertXML to RDF/OWL

XML

RDF/OWL ++

burglary-incident-w-arrest-basic-lexs (RDF)Automatic

Conversion from

LEXS XML to RDF

TopBraids Semantic XML Engine

uses sxml:tag annotations on theauto-generated NIEM/LEXS OWL

Ontologies to control the

transformations.


Useful QA Check on the Semantic XML1 Triples


48/65



Q p

SELECT *

WHERE {

?subject composite:child ?object .

NOT EXISTS { ?object a sxml:Comment }

NOT EXISTS { ?object a ?type .

?type sxml:element "xi:include" }

}

0 is good!

QA Check

1

Semantic XML is a composite pattern model:?anElement composite:child ?anotherElement

?anElement composite:child ?anAttribute


Example SPARQL Query for the sample Burglary Incident


49/65



p Q Q y p g y

SELECT ?s ?fn1 ?ssn1 ?fbiID1v ?fpID1v ?fpj1v

WHERE {

?s rdf:type digest:EntityPerson .

?s digest:personRef ?p1 .?p1 core:personNameRef ?pnR1 .

?pnR1 core:personFullNameRef ?pfnR1 .

?pfnR1 dtype:value ?fn1 .

?p1 core:personSSNIdentificationRef ?pSSNR1 .

?pSSNR1 core:identificationIDRef ?pSSN1 .

?pSSN1 dtype:value ?ssn1 .

?p1 digest:personAugmentationRef ?p1a .

?p1a jxdm:personFBIIdentificationRef ?fbiID1 .

?fbiID1 core:identificationIDRef ?fbicID1 .

?fbicID1 dtype:value ?fbiID1v .

?p1a jxdm:personStateFingerprintIdentificationRef ?fp1 .

?fp1 core:identificationIDRef ?fpcID1 .

?fpcID1 dtype:value ?fpID1v .

?fp1 core:identificationJurisdictionRef ?fpj1 .

?fpj1 dtype:value ?fpj1v .}

Find all people involved in

an incident for which we

have full names, SSNs,

FBI IDs, finger prints and

the state of jurisdiction


50/65


51/65


Generating Random Instance Data using


52/65



Generating Random Instance Data usingthe RDF Instance Graph Seed

burglary-incident-w-arrest-basic-lexs (RDF)

Automatic Cloner Using

Deep Random Graph Copier

1000 GraphClones

1 Seed Graph


53/65


SPINMap was used to transform the


54/65



pFake People to NIEM/LEXS People

RDF/OWLInstances


55/65


56/65


Using SPIN to classify Person Instances


57/65



g y


Using SPIN to transform Person


58/65

c to ed t aste t t e sty e


data type properties to direct attributes

Witness Class

Victim Class

Operator Class

Officer Class

Arrestee Class

Male Class

Female Class

Dispatcher Class

Person Class

SPIN Rule on Person Class

Sub-class Relationships


59/65


A Query over the Incidents Data (2 of 2)


60/65

y


Q y ( )

Not surprising to confuse the police as

suspects if you see this going on


So why Integrate Data using RDF/OWL?


61/65

y


So why Integrate Data using RDF/OWL?

Ontology-Driven

Data Refineries

Frictionless

Data


62/65


63/65


64/65


65/65