philadelphia, may 2–4, 2005 effective use of spatial databases for enterprise data integration dr...

Philadelphia, May 2–4, 2005

www.locationintelligence.net

Effective use of spatial databases for

Enterprise data integrationDr Paul WatsonLaser-Scan



Overview• Motivation - review key drivers for Data Integration• Obstacles to Data Integration• Data Quality Management Principles

– Data Stewardship– Knowledge Management– Enterprise metadata– Rules based processing

• Methdology – Data Quality Improvement cycle• Spatial Business Rules• Spatial Knowledge Management – Radius Studio

– Rules browser/ rules repository– Conformance check/ report– Reconciliation– Certification



Agile Principle Business Needs Information NeedsEnrich theCustomer

Adapt products &Services

Integrate, extend

MasterChange

ReactForecast

TransformModel, test, extrapolate

MobiliseResources

Plan, EmpowerBuild Virtual teams

Analyse, Build knowledgeCollaboration platforms

Co-operate toCompete

Focus, partner Transform, publish, exchange

Drivers for Data Integration

Agile Competitors and Virtual Organisations - Goldman, Nagel and Preiss



Obstacles to Data Integration

• Proprietary Data Silos - data cul de sac

• Monolithic Information Systems – embedded logic

• Built-in, Private Data Models – structure/lifecycle

• Unknown/Unproven Data Quality – KR/KM



Enablers For Enterprise Information Architectures

• Electronic Data Interchange • Straight-through Processing• Service Oriented Architectures• Rules-based Processing• Knowledge Representation Standards• Data & service integration is the goal



Technological Building Blocks

• Open data access standards

• Extensible, interoperable platforms

• Metadata publication/retrieval

• Data stewardship



Data Quality ImpactOne third of companies have been forcedto delay or scrap new systems because of faulty data,and a full 75% have experienced significant problemsresulting from data quality issues - PwC

55-70% of CRM and 70% of Data Warehouseproject failures are due to data quality issues - Gartner, Meta

US business loses $600 billion each yeardue to data quality problems - Data Warehousing Institute

When is data “fit for purpose”?



Principles, Tools & Method for DQ Improvement• Knowledge Management

– Describe the domain/problem independently (from data and systems)

• Rules-based paradigm– Decouple the problem definition from problem

discovery & resolution (what to do about it)• Data Quality Improvement Cycle

– Employ a process of continuous monitoring & improvement - sustainable interoperability™



Knowledge Management

• Where is knowledge now?– Embedded in data– Hidden in point applications– Inside people’s heads

• Store the knowledge/expertise of the organisation where everyone can contribute to it and share it - as enterprise metadata in a database



Benefits of a knowledge management approach

• Explicit/unambiguous – not embedded• Inclusive - accommodates domain experts• Non-technical – not just for developers• Open, distributable – location/applications• Auditable – regulatory/governance issues• Evolutionary – incremental acquisition, knowledge

is refined/grows over time • Structured – machine-readable



Knowledge Representation

•RDF•OWL•SWRL



Data Quality as Knowledge• Express data requirements as rules (e.g. SWRL)• Data quality rules – enterprise metadata - DB• Rules metadata can be shared (interpreted and

enforced) by many different applications• Rules can be used to measure Data Quality - %

conforming data instances• Rules guide data reconciliation - prioritise• Rules can be used to measure quality improvement

reliably• “Fit for Purpose” = satisfies the DQ rules



Rules-based Processing Paradigm

• Fact – Pattern – Action• Given some facts, if they meet any of the patterns/rules,

perform the defined action– Declarative – rule separated from processing– Pluggable actions – reporting/ reconciliation

Data Rule A

Report Action A

Reconcile Action A





make baselineassessment

refine rulemetadata

check dataconformance

perform datareconciliation

datacertification

datapublication

define rulemetadata

define qualitymission

Data Quality Improvement Cycle

0

10

20

30

40

50

60

70

80

90

Rule A Rule B

SP

0

10

20

30

40

50

60

70

80

90

Rule A Rule B

01020304050

60708090

100

Rule A Rule B

☺

e.g. address cleaning



Enterprise Integration Gapfor Location

• Spatial data - complex, brittle and surprising• Maintenance - manual, expensive, error-prone• Adherence to a spatial model is often business critical

(e.g. land & property management, utilities)• Tools - significant bespoke development, inflexible, built

by developers not domain experts• Location data - key for BI, data quality chasm, mining• Rudimentary IT standards for spatial data• Spatial semantics never explicit – KR/KM



Spatial Knowledge Management

• Same methodology– Spatial business rules as metadata– Conformance checking/reporting– Data Reconciliation – rules driven– Certification/publication– Only detailed data operations change

• Business Rule - Percentage: 0 ≤ x ≤ 100• Location Rule Spatial (a,b) : coveredBy, contains, withinDistance

etc.• Empower the domain expert – short “dev.” cycles, stay agile



SpatialData Quality

RulesIntra-featureConstraints

physicalgeometric

DataSpecifications

Ad hocRules

Inter-featureConstraints

proximity topologicaldirectional

Spatial Rules Discovery



Spatial Rule Types



SpatialRulesAuthoring

curbline

building



Spatial Rule Builder



Add Rule Clauses

building

zoning



Spatial & Non-spatial Conditions

Fire stationbuilding

Street centreline

“There’s no such thing as a spatial rule”



<ruleml:imp> <ruleml:_body> <swrlx:classAtom> <owlx:Class owlx:name="Road"/> <ruleml:var>r</ruleml:var> </swrlx:classAtom> <swrlx:individualPropertyAtom swrlx:property="hasGeom"> <ruleml:var>r</ruleml:var> <ruleml:var>g</ruleml:var> </swrlx:individualPropertyAtom> <swrlx:datavaluedPropertyAtom swrlx:property="hasName"> <ruleml:var>r</ruleml:var> <ruleml:var>n</ruleml:var> </swrlx:datavaluedPropertyAtom> <swrlx:classAtom> <owlx:IntersectionOf> <owlx:Class owlx:name="Geometry"/> <owlx:ObjectRestriction owlx:property="hasFeature"> <owlx:SomeValuesFrom> <owlx:IntersectionOf> <owlx:Class owlx:name="RoadSegment"/> <owlx:DataRestriction owlx:property="hasName"> <owlx:hasValue>n</owlx:hasValue> </owlx:DataRestriction> </owlx:IntersectionOf> </owlx:SomeValuesFrom> </owlx:ObjectRestriction> </owlx:IntersectionOf> <ruleml:var>j</ruleml:var> </swrlx:classAtom>

<swrlx:individualPropertyAtom swrlx:property="hasAggregate"> <ruleml:var>j</ruleml:var> <ruleml:var>a</ruleml:var> </swrlx:individualPropertyAtom> </ruleml:_body> <ruleml:_head> <swrlx:individualPropertyAtom swrlx:property="equals"> <ruleml:var>g</ruleml:var> <ruleml:var>a</ruleml:var> </swrlx:individualPropertyAtom> </ruleml:_head> </ruleml:imp>

High St.

High St.

Road

Road Segment

For each Road, the set of Road Segments having the same name as the Road must have the same aggregate geometry as the Road

Rul

es a

s K

now

ledg

e



Conformance Checking

SpatialRulesEngine

EnterpriseMessaging

EnterpriseSpatial DB

EnterpriseMetadata -Spatial Rules

Web ServicesClients

AsynchronousMessagingClient

BrowserClient

ReportingSolution



Data Reconciliation

SpatialRulesEngine



Web ServicesClients

UpdateA-B-A

CloneA-B-A’

Schema MapA-B-C



Warehouse Certification

SpatialRulesEngine



Web ServicesClients

0

10

20

30

40

50

60

70

80

90

Rule A Rule B

01020304050

60708090

100

Rule A Rule B

http://www.iso.org/iso/en/ISOOnline.frontpage



Summary

• Data quality issues in integration are best addressed using knowledge management/rules-based approaches

• Spatial data quality is no different to any other data quality – standard, interoperable rules are key

• Enterprise spatial rules engines form a secure base from which to develop open, distributable location enabled applications

philadelphia, may 2–4, 2005 effective use of spatial databases for enterprise data integration dr...

Documents