fme extensive usage inside the mapping production system of natural resources canada
DESCRIPTION
Natural Resources Canada (NRCan) uses FME software since 1998. Initially, it was used exclusively as a translation tools. As NRCan gained experience with the FME, it began to integrate it within its cartographic production processes. Today, FME is a centerpiece for validation, integration and dissemination of spatial data from NRCan. This presentation will describe the infrastructure (feature catalog, constraints management, database, etc.) surrounding the use of FME at NRCan and it will also present some interesting validation and spatial data management solutions.TRANSCRIPT
![Page 1: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/1.jpg)
1
FME within NRCan
Marie-Eve Martin Jean-Marc Prévost
![Page 2: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/2.jpg)
2
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 3: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/3.jpg)
3
Agenda
! Context ! Who are we? ! Where do we come from? ! Where are we now? ! Where are we going?
! System architecture ! Programming standards, Quality assurance, testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 4: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/4.jpg)
4
Natural Resources Canada Mission and Outcomes
![Page 5: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/5.jpg)
5
NRC
an
Earth Science Sector
Geoma4cs Canada
CCMEO / CCCOT
Energy Sector
Canadian Forest Service
Direc4on & Coordina4on
Minerals & Metal Sector
Science & Policy Integra4on
Energy Technology & Programs Sector
Corporate Management & Services Sector
NRCan structure overview
CCMEO Canada Centre for Mapping and Earth Observa5on
CCCOT Centre Canadien de Cartographie et d’ Observa5on de la Terre
CCMEO total number of employees: 319 227 employees for Mapping 92 employees for Earth Observa5on
Sherbrooke
OGawa
![Page 6: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/6.jpg)
6
Agenda
! Context ! Who are we? ! Where do we come from? ! Where are we now? ! Where are we going?
! System architecture ! Programming standards, Quality assurance, testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 7: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/7.jpg)
7
FME history at CCMEO
NTDB specs
vectoriza4on File management
CanVec, CanTopo, GeoBase, etc. era Database management
FGP 2015 2010 2005 2000 1995 1990 1985
Transla4ons -‐ Workbench
-‐ Data manipula4on & valida4on
Custom transformers
-‐ Programming standards
-‐ Tes4ng methodology
≈700
![Page 8: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/8.jpg)
8
Agenda
! Context ! Who are we? ! Where do we come from? ! Where are we now? ! Where are we going?
! System architecture ! Programming standards, Quality assurance, testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 9: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/9.jpg)
9
Distribu?on Management
System architecture at CCMEO
Aquisi?on
Cond
i4on
ning
Partners
Data acquisition
Others
Data Access QA
![Page 10: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/10.jpg)
10
What do we produce ?
! Main products : ! Canvec ! Toporama ! CDED ! 200 collections of various nature
! Statistics for last year : ! 8 500 000 downloads ! 3 000 mail and phone requests
![Page 11: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/11.jpg)
11
Where to find our products ?
! Data availability : ! Geogratis : geogratis.gc.ca ! Geobase : www.geobase.ca ! WMS services
! Licence : Open Government Licence
![Page 12: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/12.jpg)
12
Data extraction tool http://geogratis.gc.ca/site/eng/extraction
![Page 13: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/13.jpg)
13
Agenda
! Context ! Who are we? ! Where do we come from? ! Where are we now? ! Where are we going?
! System architecture ! Programming standards, Quality assurance, testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 14: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/14.jpg)
14
![Page 15: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/15.jpg)
15
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 16: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/16.jpg)
16
System architecture
FME, Python, .EXE, SQL, etc.
Windows servers Produc4on: 64 processes (32bits) Distribu4on: 56 processes (64bits)
Oracle 11g, PostgreSQL 9.2 ArcSDE 10.1, PostGIS 2.0
400 TB
Instruc?on files
![Page 17: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/17.jpg)
17
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 18: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/18.jpg)
18
Worbenches – Last saved build
Programs s4ll in opera4on !
![Page 19: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/19.jpg)
19
Taking control
! 2007 problematic: ! Mix of FME versions on computers ! Everyone wanted the lasted build ! Increasing number of workbenches ! Workbenches were transferred to support team
! Solution : ! FME upgrades management ! Programming standards
![Page 20: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/20.jpg)
20
FME upgrades management
! Upgrade rules: ! Everyone use one common build ! Upgrade only to official builds ! At least one upgrade per year
(idealy no more than two)
![Page 21: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/21.jpg)
21
Programming standards
![Page 22: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/22.jpg)
22
Programming standards
! Documentation (.docx and annotations) ! Parameter naming standards ! Known bad practices review ! Unit tests ! Peer review
![Page 23: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/23.jpg)
23
Programming standards Examples of bad practices
! Missing Rounders ! Missing Terminators
![Page 24: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/24.jpg)
24
Programming standards Unit tests
! Each workbench is managed in a source code control environment
! Each workbench is saved with the bat file, sources and expected results files necessary to test it
! Batch testing of all workbenches before every upgrade.
![Page 25: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/25.jpg)
25
Programming standards Unit tests
! Fail rate is used to plan upgrades
Source Destination Fail rate
2013 SP1 2013 SP4 2%
2013 SP4 2014 SP1 10%
![Page 26: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/26.jpg)
26
Programming standards Peer review
! FME is easy to learn without taking classes ! FME always offer various way of doing
something
! Peer review allows: ! Improved shared knowledge ! Programming uniformity ! Identifying repetitive patterns
![Page 27: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/27.jpg)
27
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 28: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/28.jpg)
28
Validation processes
! At CCMEO, FME is used for : ! Feature translation ! Feature creation ! Feature validation
Catalog
![Page 29: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/29.jpg)
29
Distribu?on Management
Validation processes
Aquisi?on
Cond
i4on
ning
Partners
Data acquisition
Others
Data Access QA
![Page 30: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/30.jpg)
30
Validation Catalog
! ISO-19110 compliant
XML HTML Database
Users
![Page 31: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/31.jpg)
31
Catalog based validations
! Spatial relations validation ! Domain attribute validation ! Proximity validation ! Minimal dimension validation ! Segmentation validation ! Data clipping validation
![Page 32: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/32.jpg)
32
Catalog based validations Example
Spatial relations validation ! Based on Egenhofer-Clementini DE-9IM masks
![Page 33: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/33.jpg)
33
Catalog based validations Examples
! Spatial relations
! Attribute value
![Page 34: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/34.jpg)
34
Catalog validation within FME
ISO 19110 XML
List friendly format
Generic reader
Generic writer
Valida4on Error
forma4ng Valida4on rules
extrac4on Valida4on
Error forma4ng
Valida4on rules extrac4on
One custo
m transfor
mer
per valida
5on conc
ept!
![Page 35: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/35.jpg)
35
Catalog validation within FME
! Real program example:
![Page 36: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/36.jpg)
36
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 37: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/37.jpg)
37
Utility customs - Examples
! TerminatorRecorder ! Record all the errors to a .ffs file before causing the workbench to fail
! TerminatorMinMax ! Cause the workbench to fail if
the number of feature is not as expected
![Page 38: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/38.jpg)
38
! Matcher_CCMEO ! Match features if
start/end points are within tolerance
! SnapToNeatline ! Snap features within tolerance without
collapsing
Utility customs - Examples
![Page 39: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/39.jpg)
39
Utility customs - Examples
! SchemaMapper_CCMEO ! Similar to SchemaMapper but : ! Allow mapping with a variable number of
conditions ! Conditions may be ranges of values ! Conditions may use any operator (>, <, != , etc) ! Conditions are validated and produce report if
badly formatted
TYPE=[11:13];PRECISION<=20;NAME=Lake
![Page 40: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/40.jpg)
40
Validation customs Minimal dimension
Voronoi
Side lines
Centerline
Area Width Length
![Page 41: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/41.jpg)
41
Validation customs Proximity
What are we looking for?
Valida4on Process
Op4misa4on Process
![Page 42: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/42.jpg)
42
Agenda
! Context ! System architecture ! Programming standards, Quality assurance,
testing methods, etc. ! Validations processes and dependencies ! FME developments at NRCan ! Safe Software collaboration
![Page 43: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/43.jpg)
43
Safe Software collaboration
! Motivations for collaboration ! Simplifies our workbenches ! Faster processing ! Maintenance under Safe’s responsibility
![Page 44: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/44.jpg)
44
Safe Software collaboration Generalization
Hydro network
AngularityCalculator
SherbendGeneralizer
StreamOrderCalculator
StreamPriorityCalculator
NetworkFlowOrientor
BoundingBoxAccumulator
![Page 45: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/45.jpg)
45
Safe Software collaboration
VoronoiDiagrammer
![Page 47: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/47.jpg)
47
Amalgamator
![Page 48: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/48.jpg)
48
ConvexityFilter
![Page 49: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/49.jpg)
49
Generalizer Wang The Wang algorithm will itera5vely combine, eliminate and exaggerate bends un5l the input line feature has no bend that is smaller than the given tolerance value.
McMaster / McMaster Weighted Distance The McMaster algorithm calculates a new loca5on for each point by first taking the average value of the x and y coordinates of the point and a number of neighboring points. It then slides the averaged point towards the original point according to a specified displacement value. The overall effect is that each point will be pulled towards its neighboring points.
NURBfit (B-‐Spline) The NURBfit algorithm will fit lines using B-‐Spline curves of given polynomial degree. The resul5ng lines will follow these curves with given segment length. The higher the degree, the smoother the line. An example of usage is smoothing contour lines in order to remove sparks and simulate the work of a cartographic cra[sman.
Inflec?on Points The Inflec5on algorithm will calculate the loca5on of the inflec5on points along a line and return the list of these points. Inflec5on points are measures of the sinuosity of a line.
![Page 50: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/50.jpg)
50
SherbendGeneralizer SherbendGeneralizer
![Page 51: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/51.jpg)
51
NetworkFlowOrientor NetworkFlowOrientor
![Page 52: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/52.jpg)
52
StreamPriorityCalculator
StreamPriorityCalculator
![Page 53: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/53.jpg)
53
Densifier
![Page 54: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/54.jpg)
54
Chopper
![Page 55: FME Extensive Usage Inside the Mapping Production System of Natural Resources Canada](https://reader034.vdocuments.net/reader034/viewer/2022042714/55796d83d8b42a3a5c8b4eac/html5/thumbnails/55.jpg)
55
Chopper use case scenario
chopper
dissolve
chopper chopper
dissolve
dissolve
Data extraction
Group by metadata No group by