School of ComputingFACULTY OF ENGINEERING

The VISTA project:Integrating UK utility data

Beck, Boukhelifa, Cohn, Fu & Parker

Overview

The Utility Underworld

Massive network of buried services: gas, water, electricity, telephone, cable, sewage, drains … Need to know asset location for planning and maintenance Many databases, varying accuracy and provenance Context

~4M street openings p.a. Direct costs of £1B p.a. Indirect costs of £3B-£5B p.a. Safety!

A Congested View

Overview

Introduction - VISTA

Utility Data

Data Integration

Data Delivery and Visualization

Implementation Issues

Conclusions

VISTA Consortium

Visualising Integrated information on buried assets to reduce streetworks

VISTA

VISTA Project

4 year government funded project

23+ Utility partners, other universities (Nottingham and Leeds)

Aims to reduce cost of street works in the UK

• Motivation: Traffic Management Act

Facilitate sharing and exchange of knowledge about buried assets

• Digitise existing paper maps

• Integrate utility data

• Visualize the integrated data set

Tagline: Swift, safe, cost-effective streetworks

Objectives for Leeds University

Agree a core set of attributes

Provide a framework for integrating and accessing this data

Investigate presentation needs for different classes of user

Design appropriate presentation techniques

The Vision (details)

Overview

Introduction - VISTA

Utility Data

Data Integration

Data Delivery and Visualization

Implementation Issues

Conclusions

Utility Data

Heterogeneous in nature Modern data is predominantly, but not exclusively, digital (GIS: vector)

• Available in paper / raster / vector

No common format or standard for data

Captured over the past 200 years

Variation in data quality

• Only 50% of buried infrastructure location known accurately (Marvin and Slater (1997)).

Relative vs. Absolute positioning

School of ComputingFACULTY OF ENGINEERING

The current picture

Current Practice

Utility packs Combined service drawings are rare Reliance on Ordnance Survey backdrop for visual integration of asset

information User preference for n+1 maps

Users would like combined service drawings

Preparation Manual process Informal design No standard format or symbology

Vista aims to automate map production

Overview

Introduction - VISTA

Utility Data

Data Integration

Data Delivery and Visualization

Implementation Issues

Conclusions

Utility Data: Problem Domain

Heterogeneous in Practice

Different ways of storing asset data

Paper – CAD – GIS

Raster to Vector conversion

• Employed spatial grammar techniques through genetic algorithms

Different ways of storing digital asset data

Different syntactic models

Global Schema based integration

• During prototype phase use ETL software (FME from Safe)

• Will recommend that OGC interoperable sources are implemented

• A mixed model is inevitable in the short term

Different ways of structuring digital asset data

Different syntactic and schematic models

Integration based on a common utility data model (global schema)

• Resolving schematic heterogeneity

Utility Data: Problem Domain

Different ways of describing asset data

Semantic inconsistency

Ontology/Global Thesauri employed at the data level

• Resolving semantic heterogeneity

• When the same asset type is given different names by different companies

• When different asset types are given the same name by different companies

Different ways of sharing and representing asset data

Paper – CAD – GIS

• Different symbols and conventions

• Uncertainty

User/domain tailored visualisations

Integration constraints

Require low operational impact

Organisations are unlikely to change their internal data model

Organisations must retain full data autonomy

Aim: Swift, safe, cost-effective streetworks

Objectives

Agree a core set of attributes

Provide a framework for integrating and accessing this data

Investigate presentation needs for different classes of user

Design appropriate presentation techniques

The Vision (details)

Agree a core set of attributes

Currently 29 Global Schema fields grouped into 11 types

Keep It Simple – flat file approach

Semantically transparent field-names

Distinguished between core and non-core data

Asset x 10 fields (5 core)

Condition x 1 field (0 core)

Confidence x 1 field (0 core)

Date x 1 field (0 core)

Detection System x 1 field (0 core)

Dimension x 5 fields (5 core)

Domain x 4 fields (2 core)

GIS x 2 fields (1 core)

Location x 3 fields (1 core)

Rehabilitation work x 1 field (0 core)

Risk x 1 field (0 core)

A framework for integrationOverview

A framework for integrationSyntactic (format) integration

Essentially resolving the differences in GIS format between each utility dataset. Ideally will be done using a syntactically interoperable approach (OGC WFS).

Currently use FME middleware to convert the source data into the target format (ORACLE). This could be scripted to deal with data refreshing from different locations.

At implementation, there is likely to be a hybrid approach (middleware/WFS) in the short/medium term.

A framework for integrationSchematic (structure) integration

To generate the metadata that describes the relationship between the source data and the global schema.

These may be direct 1-1 mappings or they could represent transformations: Scaling numerical data

Calculated values (conversion of ‘depth of invert’ to ‘depth of cover’)

Compound data needs splitting

Atomised data needs compounding

Furthermore, as utility data can be sparsely populated some error checking may be required.

Rule generation is a complex, iterative, process.

A framework for integrationSchematic (structure) integration

Metadata rules generated in RadiusStudio from 1Spatial

Simple user interface.

Can do complex mapping and error checking.

RadiusStudio is a web service.

We will be researching into techniques to deploy this metadata in XSLT.

XSLT could be used to store all metadata natively.

However, is difficult to edit, design and share.

Domain knowledge is essential so the rules must be understood by the end-users.

A framework for integrationSemantic (term) integration

Generation of a cross domain global thesaurusThesaurus: a tree of terms linked together by hierarchical, and associative or equivalence relationships.

• Can be converted into an OWL ontology

• Developed within MultiTes Pro

Articulated in OracleReconciling semantic heterogeneities

A framework for integrationOverview

Integrated Data

The approach allows data from different utility domains to be successfully integrated Network data Furniture data

This provides greater flexibility for data presentation Other attributes can be used during data visualisation

Global schema continually under development Requirements for telecoms 3d features Changes are easy to implement

• because of the way the metadata is collected, stored and shared

Overview

Introduction - VISTA

Utility Data

Data Integration

Data Delivery and Visualization

Implementation Issues

Conclusions

Developing a utility web service

Pilot Projects

Infrastructure: Traditional Web GIS VISTA: ORACLE (datastore)

• Source utility data – materialized

• Schematically and semantically integrated VISTA: GeoServer (WFS delivery)

• Consumes data from Oracle

• Delivers OGC compliant WFS

• Via a secure link between Leeds and Developer servers

• Integrated data

• Materialized views of integrated data (faster delivery) Developer Service

• Consumes VISTA WFS

• Developer controls how the data is rendered

• Can be made fit for multiple different end-users (Planning, on site, etc.)

• Renders integrated data to ‘accredited’ users

Pilot Projects

East Midlands x 2 Utilities:

• Anglian Water• Central Networks• National Grid• Severn Trent• United Utilities• Yorkshire Water

Portal Developer• Jacobs• Anglian Water

Scotland Data partners

• Perth and Kinross Council• Scottish Water• Transport Scotland

Portal Developer• Symology

School of ComputingFACULTY OF ENGINEERING

Bespoke Visualization

Bespoke utility visualization

Uncertainty Visualization Include aspects of uncertainty onto the display

Incorporating Aesthetics Reduce clutter and visual complexity

Ontology-based Visualization Use utility ontology to drive the visualization

End user requirements are crucial

Uncertainty visualization EXAMPLES

A. future environmental setting -> colour

D. longitude / magnitude -> glyphs

B. contours -> line style

H. local uncertainty -> volume

C. regional uncertainty -> quadtree

E. noise -> overlaid grid

G. kind of object -> blur

F. shape -> texture

2D SVG map of utility Data

Visualization PrototypeUSE OF BLUR

Gas

SewerWater

OS

Visualization PrototypeUSE OF BACKGROUND COLOUR

Unified 3-colour scheme• green: certain• yellow: probable• red: uncertain

Aesthetics and Clutter

Close proximity between assets

Line crossings and busy junctions

Missing 3D information

Label overlap

Backdrop information

Aesthetics and Complexity

Bends (b)

Crosses (c)

Angles (m)

Orthogonality (o)

Symmetry (s)

[Purchase, 98]

Aesthetics – A graph example

Detect cluttered areas

Detect

Then simplify

Aesthetics – Is it possible to improve this!

•Promote area of interest•Declutter•Graph the non-essential areas

•Retain context

School of ComputingFACULTY OF ENGINEERING

Ontology Driven VisualizationTo be developed

Ontology-based visualization

Ontology<CONCEPT>

<DESCRIPTOR>Hydrant</DESCRIPTOR>

<NT level="1">Combined Hydrant </NT>

<NT level="1">Fire Hydrant </NT>

<NT level="1">Washout Hydrant</NT>

<BT level="1">Water Furnishing and Fixture

<BT level="2">Water Asset</BT>

</BT>

<SN>Device for extracting large volumes of water from the

network. Used to flush out the network, provide water for

fire fighting or provide a temporary connection.</SN>

<SC>Water</SC>

</CONCEPT>

<CONCEPT>

<NON-DESCRIPTOR>Impounding Reservoir</NON-DESCRIPTOR>

<USE>Raw Water Storage</USE>

<SN>….</SN>

<SC>Water</SC>

</CONCEPT>

Overview

Introduction - VISTA

Utility Data

Data Integration

Data Delivery and Visualization

Implementation Issues

Conclusions

Implementation issuesOperational Impact

Require low operational impact

Organisations are unlikely to change their internal data model

Organisations must retain full data autonomy

Changes, such as WFS, should have additional business benefits

Implementation issuesData Currency

Data Currency

Data currency should fit the purpose of the end use

• Back office planning

• Field requirements

Recognise that there is always lag in the system

Implementation issuesOther issues

Other issues

Response time

• Virtual vs. materialised

Scale of integration

• Localised?

• National?

Data Security

• Requires unpicking

• Always better than sharing data on CD!

Implementation issuesImpact on architecture

Virtual or Materialised:

The utility industry needs to decide

What are the applications?

What are there requirements?

• Utility requirements are significantly different to disaster/emergency response

Overview

Introduction - VISTA

Utility Data

Data Integration

Data Delivery and Visualization

Implementation Issues

Conclusions

Conclusions

VISTA has ‘proved the concept’ for dynamic integration of heterogeneous utility data in the UK

Developed a cross domain utility thesaurus

Developed a number of visualization mechanisms

Recognised a number of implementation issues

Further Work

Develop a rich cross domain ontology

Links to FGDC, the Common Information Model (IEC 61970-301 etc)

Ontology driven integration process

Ontology driven visualization process

Once stabilised the whole system should be modelled in UML

Any Questions?

www.mappingtheunderworld.ac.uk

www.comp.leeds.ac.uk/mtu

www.vistadtiproject.org

e-mail: arb@comp.leeds.ac.uk