1

Reporting Summary Information of Spatial Datasets and Non-Compliance Issues Using FME Workspaces

Martin de ZuviriaGIS Architect

Mapping & Charting Establishment

April, 2016

Abstract

An overview of two groups of FME workspaces implemented at the Mapping and Charting Establishment (MCE) that include the generation of reports in Excel format is presented here.

The first group includes data validation and data compliance assessments. An example showing Self Validation of Spatial Data Input from DND Bases using FME Server is presented.

The second group, implemented using FME Desktop, includes the creation of summary reports for some key datasets distributed by MCE. Two examples of FME workspaces are presented here: the first one showing reports created for NRCan CanVec plus charts, and the second one for Open Street Map (OSM) data delivered in FGDB format for custom AOIs.

3

Self Validation of Spatial Data Input from DND Bases

Summary

The FME workspace presented here is part of a joint project between the Mapping and Charting Establishment (MCE) and the Assistant Deputy Minister for Infrastructure and Environment (ADM(IE)). This project involves managing DND real property and other spatial data provided by DND Bases across Canada at MCE through a unique, integrated and standardized Real Property Spatial Data Warehouse (RPSDW) containing a SQL Server database.

Data provided by DND Bases must meet the standards defined and documented by ADMIE, in terms of data format accepted (ArcGIS FGDB), schema and attribute data types, domains and accepted values for each feature class. An FME workspace was created to perform this data compliance assessment. This ETL workspace was published to FME Server to make it available to DND GeoTechs from DND Bases through the DWAN. This ETL was created together with a user’s guide, that allows GeoTechs to perform a self validation of the DND real property and other spatial data before these data is sent to MCE to be loaded into the RPSDW SQL Server database. This on-site QA will save time and resources and will prevent loading of non-compliant data to the SQL Server database.

The FME workspace presented here outputs all errors, warnings and non-compliance issues to a destination FGDB and a detailed report of all these issues to an Excel file.

ADMIE Standards: Feature Classes, attributes and properties defined and documented

Overview of the Self-Validation Workspace

Errors and Warnings are reported to a FGDB with the same input schema and format: A single feature may be output several times

showing only one error at a time (one error – one record) in a new ‘TypeError’ field and a timestamp is also added

Issues found are reported to an Excel file containing a summary sheet (below) and two additional sheets for every feature class

where errors have been detected

Issues found for every feature class are reported in two separate sheets: one containing a bar-type chart (see below) to allow a

fast view and another one showing the extent and shape of features with errors (next slide)

Issues found for every feature class are reported in two separate sheets: one containing a bar-type chart (previous slide) to allow

a fast view and another one showing the extent and shape of features with errors (see below)

OpenStreetMap is the worldwide project that creates, daily updates and distributes free geographic data for the world (OSM XML format)

OpenStreetMap represents physical features on the ground (e.g.roads or buildings) using tags attached to its basic data structures (its nodes, ways, and relations). Each tag describes a geographic attribute of the feature being shown by that specific node, way or relation. OpenStreetMap's free tagging system allows the map to include an unlimited number of attributes describing each feature. The community agrees on certain key and value combinations for the most commonly used tags, which act as informal standards.

However, users create new tags to improve the style of the map or to support analyses that rely on previously unmapped attributes of the features. Short descriptions of tags that relate to particular topics or interests can be found using the feature pages.

For more info, please visit www.openstreetmap.org

Open Street Map

11

OSM XML to ArcGIS Workflow Note: this draft workflow is currently being tested for a selected CountrySteps

1. Define AOIAOI could be a city, country, continent or the world

2. Define Source Select source for OSM data (e.g. Planet OSM for the world or Geofabrik for a country , continent or city)

3. Download Download data in OSM XML compressed format (*.osm.bz2)

4. Decompress Decompress downloaded data (Windows RAR may be used)

5. Run FME WorkspaceRun on Desktop OSM FME Workspace (transforms OSM XML to FGDB and Reports to an Excel file summary information of all feature’s timestamps, individual contributors and failed -non OGC compliant – features)

6. Open MXD Open Map Document template with symbology and scale dependencies, repair data sources, rename it and save it to folder where output FGDB is located

7. Analyze & Compare Open the Excel summary, analyze content and, if applicable, compare update ofThe same AOI from previous reports

Deliver data Workflows are been tested using FME Desktop 2016.0 and ArcGIS 10.3.1

Tool

Web

Web

Web

Windows

FME

ArcGIS

Excel

8. Compress Compress output FGDB, Excel report and MXD fileWindows

Steps 1,2,3: Select an AOI and Export data from the Web (OSM XML format)

Step 5: Run on FME the OSM tools transforms OSM XML to FGDB (Tool1) and Reports to an Excel file summary (Tool 2) statistics of all feature’s timestamps, individual contributors and failed -non OGC

compliant – features)

Step 5: The OSM workspace transforms OSM XML to FGDB and Reports to an Excel file the summary statistics of all feature’s

timestamps, individual contributors and failed -non OGC compliant – features)

Step 6: Open FGDB with template ArcGIS .MXD , analyze results (including warning and errors reported in Log file) and rename *.MXD

Step 6: Open FGDB with template ArcGIS .MXD, analyze results and rename *.MXD

Step 7: Open Excel output and analyze results (selected results for a test AOI are illustrated below). If satisfied with all results,

compress files and deliver to User(s) (steps 8 and 9)

Year NrFeatures2015 29726962014 32366232013 22023132012 28962362011 19353812010 1988922009 1043012008 526282007 110672006 14

Month NrFeatures1 2498652 2251783 1631454 2977385 4919396 3986897 5493788 2989129 245497

10 52355

UniqueUser NrFeatures   

jsimoes (uid = 451289) 1168479

topolusitania (uid = 380552) 779378

ViriatoLusitano_import (uid = 2330803) 603570

rtafav2 (uid = 602999) 564034

Lobinho (uid = 571263) 512478

Patrick Bous (uid = 223763) 473572

Maria Inês (uid = 1751259) 422466

oha (uid = 8204) 251820

TaedeT (uid = 57577) 241793

biscas (uid = 882362) 186720

bri g (uid = 20163) 172049

ecosdamontanha (uid = 1786621) 160102

Wolfram Sobotta (uid = 74983) 156806

UniqueUser NrFeaturesFail   topolusitania (uid = 380552) 329

TaedeT (uid = 57577) 226

chipsterxx (uid = 412095) 212

rtafav2 (uid = 602999) 121

bitormarques (uid = 702584) 102

Rui Oliveira (uid = 445220) 93

ch_stgt (uid = 834112) 89

oha (uid = 8204) 82

jpaf84 (uid = 458277) 59

biscas (uid = 882362) 54

Patrick Bous (uid = 223763) 54

Barrocas (uid = 1131231) 50

dcp (uid = 44221) 50

CanVec+ is a digital cartographic reference product produced by Natural Resources Canada. CanVec+ originates from the best data sources covering Canadian territory and offers quality topographical information in vector format that comply with international geomatics standards.

CanVec+ is a key data source for many GeoTechs at DND and for other people as well. New versions of charts are published on a periodic basis, but the user has to be aware that this does not mean that all individual data layers in a chart are updated in the same way. For example, a chart may have been updated in 2014 but it may contain some power lines that were last validated long ago. Since every feature in CanVec+ has a ‘Validity date’ (date of the data source used to create, revise or confirm an object), a report was created for each CanVec+ 2014 chart grouping the features of each feature class by their validation year. In addition to these individual reports, a unique table (FGDB format) containing all individual reports was created. FME Workbench was used for this task.

NRCan CanVec plus digital charts

Report of an individual CanVec+ 2014 chart (partial view)

FGDB table containing the reports of all individual CanVec+ 2014 charts

FGDB table containing the reports of all individual CanVec+ 2014 charts sorted decreasingly by ‘Validation_Year’

FGDB table containing the reports of all individual CanVec+ 2014 charts sorted increasingly by ‘Validation_Year’