inspire roadmap and architecture: lessons learned...act luxembourg - inspire roadmap and...

1 ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned

INSPIRE roadmap and architecture: lessons learned INSPIRE 2017

Stijn Goedertier – GIM Thierry Meessen – GIM Jeff Konnen – ACT Luxembourg Patrick Weber – ACT Luxembourg

2

Administration du cadastre et de la topographie (ACT) Luxembourg

ACT Luxembourg - INSPIRE roadmap and architecture: lessons learned

• Responsible for:

• National cadastral plan

• Differential GPS network

• Aerial photo’s and maps

• Open data portal: data.public.lu

• Geoportal: geoportail.lu

• INSPIRE: catalog.inspire.geoportail.lu

• …

http://data.public.lu/

https://geoportail.lu/en/

http://catalog.inspire.geoportail.lu/

3

#1 Make a plan


• ACT is responsible for the INSPIRE data

harmonisation for the whole of Luxembourg

• 253 datasets from 22 government entities

• Made a 2016-2021 roadmap

• Tasks, roles & responsibilities, estimated timing and

effort

4

#2 Review the application architecture


Data harmonisation FME Desktop / FME Server 2017

Metadata & discovery service GeoNetwork

View service GeoServer

Download service GeoNetwork (ATOM/OpenSearch)

Data store Postgres/PostGIS

File store AWS S3 (via uData)

Validation ETF-WebApp, GDI-DE Testsuite, INSPIRE validator

Components were replaced by their open-source

counterparts (where feasible)

5

#3 Keep things as simple as possible


• e.g. Metadata: one language only: English

• e.g. Metadata: only mandatory metadata elements

• e.g. Open Data: no access restriction means no

authentication and authorisation

• e.g. Service capability document: only minimal

extension (scenario 1)

6

#4 Go for a file download service


• Minimally a predefined dataset download service: i.e.

a file download service ATOM/OpenSearch or WFS2.0

stored queries)

• The user must download the entire dataset as a file

• Requires no effort from data publisher (when

ATOM/OpenSearch generated from metadata)

• Where practicable a direct access download service

(i.e. a WFS2.0 query service)

• The user can query and get the feature(s) of interest

• Implementations like GeoServer App-Schema have

many limitations

• QoS (availability, response, throughput) is more

difficult to guarantee

7



predefined dataset download service:

• Implementation: GeoNetwork was enhanced to

support local atom feeds generated automatically on

the basis of the service and dataset metadata

records.

• Conformance testing: Implementation was tested

with the GDI-DE test suite.

• Benefits:

• No additional effort required from data

publisher (just correct metadata records)

• QoS easy to guarantee

8



direct access download service:

• Implementation: GeoServer Application Schema.

• Conformity: good results with ETF-Webapp

• Drawbacks:

▶ Requires transformation of source database

▶ Not possible to distinguish multiple datasets

(only feature types) (multiple virtual WMS not

possible with App-Schema)

▶ QoS more difficult to guarantee

9

#5 Speed up deployment with Docker


• Dockerfiles / docker images for:

• FME Server

• PostgreSQL,

• GeoServer,

• GeoNetwork,

• ETF-Webapp

• NGINX

• INSPIRE dashboard.

• Building images and starting-up containers in concert

using Docker-compose or Rancher

10

#6 Automate conformance testing


• Metadata: INSPIRE Geoportal Validator

• Data: ETF-Webapp (ARE3NA ETS repository)

• Network Services: GDI-DE TestSuite

11

#7 Automate the workflow


FME

GeoServer

View (WMS) and download (WFS) services

ReST API

GeoNetwork

Data Catalog

CSW-T

API

uData

Data Portal

uData

API

INSPIRE

Validation service

ReST

API

1. Harmonise dataset

2. Upload harmonised dataset

3. Configure view service

4. Update metadata records

5. Test conformance of data, metadata, and services

12

#7 Automate the workflow - 1. Harmonise data with FME


• XML Schema: write GML according to 34 built-in

schemas… or your own

• supports INSPIRE complexities: multiple

geometries, nested features, object referencing,

repeating elements, nillable elements, grids …

How to map features into the right schema?

13



What is the SchemaMapper?

restructures the existing schema (data model) of

incoming features to a new schema, based on mappings

defined in an external lookup table.

very useful

▶ when the mappings are potentially complex,

▶ when they need to be maintained by someone

unfamiliar with FME.

14


• source_feature_type target_feature_type: mappings of

features types.

• source_attr_name target_attribute_name: mappings of

attributes.

• Attribute filters: new attribute-value combinations

(filter_target_attribute_name – filter_target_attribute_value),

conditional to the presence of attribute-value combinations in the

source feature (filter_source_attribute_name –

filter_source_attribute_value).

source_feature_type target_feature_type source_attr_name target_attr_name

UNESCO_vieux_quartiers_fortifications_LUX_2zones ProtectedSite

UNESCO_vieux_quartiers_fortifications_LUX_2zones




UNESCO_vieux_quartiers_fortifications_LUX_2zones Id inspireID.Identifier.localId


UNESCO_vieux_quartiers_fortifications_LUX_2zones Lien_Link1

UNESCO_vieux_quartiers_fortifications_LUX_2zones Lien_Link2 gml_descriptionReference.xlink_href

UNESCO_vieux_quartiers_fortifications_LUX_2zones Nom_Name siteName{0}.GeographicalName.spelling{0}.SpellingOfName.text



UNESCO_vieux_quartiers_fortifications_LUX_2zones Type_Typ



UNESCO_vieux_quartiers_fortifications_LUX_2zones Ville_Stad

UNESCO_vieux_quartiers_fortifications_LUX_2zones Zone

UNESCO_vieux_quartiers_fortifications_LUX_2zones Zone

15



A simple workspace looks as follows

(sometimes more operations required)

16

#7 Automate the workflow - 2. Upload dataset with FME


Upload datasets to a data portal using an Application

Programming Interface (API)

https://www.data.gouv.fr/en/apidoc/#!/dataset

s/create_resource

https://www.data.gouv.fr/en/apidoc/




17

#7 Automate the workflow - 2. Upload dataset with FME


import requests

url = 'https://data.public.lu/api/1/datasets/57c07be'

headers = {'X-API-KEY': 'eyJhb'}

data = {

'title': 'Dataset title',

'description': 'Dataset abstract',

'tags': 'keywords',

'private': 'true',

'spatial': {'geom': 'null', 'granularity': 'other',

'zones': []}

}

response =

requests.put(url,headers=headers,json=data)

print(response.text)

18

#7 Automate the workflow - 3. Configure view service with FME


Use the GeoServer ReST API to create workspaces, stores,

layers, styles, and WMS services.

http://docs.geoserver.org/latest/en/user/rest/api/index.html



19

#7 Automate the workflow - 3. Configure view service with FME


HTTPCaller: general-purpose ReST client

20

#7 Automate the workflow - 4. Update metadata records


Data and service metadata records are managed with

GeoNetwork.

Standards: ISO19139, CSW, INSPIRE TG metadata &

discovery service

21

#7 Automate the workflow - 4. Update metadata records


• Use Xquery update statements to update the

metadata record

• Use CSW-T API to post the updated metadata record

to GeoNetwork

• Also for service metadata (e.g.

<srv:coupledResource> <srv:operatesOn>)

22

#7 Automate the workflow - 5. Test conformance


▶ ETF-webapp validator: FME HTTPCaller used to execute tests

23

Conclusion: lessons learned


• #1 Make a plan

• #2 Review the application architecture

• #3 Keep things as simple as possible

• #4 Go for a file download service

• #5 Speed up deployment with Docker

• #6 Automate conformance testing

• #7 Automate the publication workflow

inspire roadmap and architecture: lessons learned...act luxembourg - inspire roadmap and...

Documents