curing migration flu or: how i learned to stop worrying and love the lgim
TRANSCRIPT
![Page 1: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/1.jpg)
Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM Carl Von Stetten GIS Analyst, Central Contra Costa Sanitary District
![Page 2: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/2.jpg)
The Migration Flu comes in cycles
Currently on 3rd mapping system migration FME integral to moving between GIS systems
![Page 3: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/3.jpg)
The long journey…
For decades, CentralSan has been trying to get from hardcopy map books…
Picture courtesy http://esri.com
…to a fully integrated digital environment.
![Page 4: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/4.jpg)
In the old days…
From at least the late ‘70s on, this was our GIS:
![Page 5: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/5.jpg)
CentralSan goes digital!
Transition to digital mapping began in 1989 Platform was Intergraph’s MGE
System running on Bentley MicroStation Ran on UNIX-based CLIX
workstations CAD files with database
attachments for attributes
![Page 6: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/6.jpg)
The flue began with a fever…
MGE was far from perfect UNIX environment: MGE/MicroStation + RIS +
Informix Three different software companies Different release cycles Brittle version compatibility
Windows environment: MGE/MicroStation + RIS + MS SQL Server Same problems, just traded database platforms
![Page 7: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/7.jpg)
…followed by the chills…
Data itself wasn’t always reliable MicroStation file corruption
Entire software product line (Axiom International) developed just to repair corrupt files
Broken database linkages Orphaned database records
Data split into hundreds of tiles/basemaps 739 sewer map tiles / 24 recycled water tiles 137 assessor map base files
![Page 8: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/8.jpg)
But the hardcopy product got better…
![Page 9: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/9.jpg)
The seamless GIS begins
In late ‘90s/early ‘00s began developing web-based GIS application California CAD Solutions (CalCAD) helped deploy
Autodesk MapGuide FME was used to convert Microstation tiles into a
seamless set of Mapguide .SDF files and attribute database tables
![Page 10: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/10.jpg)
Benefits realized
Seamless map environment enabled integration with external systems Computerized Maintenance Management System
(CMMS) data from Sussex/Accela Parcel and permit data from Sungard ERP system Pipeline CCTV inspection data from WinCan
![Page 11: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/11.jpg)
…oh, the aches…
MGE problems a continuous pain in the @#! 1. Run the FME processes 2. Discover corrupted files/broken database links “Holes” in seamless map
3. Repair the files/reenter database data 4. Rerun FME 5. Discover new corrupt files/broken links 6. Rinse and repeat…
![Page 12: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/12.jpg)
Curing the flu
In 2005, chose Intergraph GeoMedia 5.2 with Public Works Manager (PWM) extension to replace MGE
Designed a custom schema to contain all the different feature classes from MGE/MicroStation files
Every element had to have an MGE “linkage” to be able to migrate into GeoMedia
![Page 13: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/13.jpg)
FME to the rescue!
Wrote FME workbenches to prep data for migration: Find MicroStation elements without MGE “linkages” Find elements with incomplete or invalid attributes
If Intergraph hadn’t been contracted to perform the data migration, we would have used FME
Wrote new FME workbenches to convert GeoMedia data to MapGuide / SQL Server for web GIS Still using mostly same FME workbenches 9 years
later
![Page 14: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/14.jpg)
And the flu symptoms return…
Deployed GeoMedia/PWM 5.2 Intergraph wrote significant amount of custom
PWM code to automate workflows Bugs in display and storage of arcs
GeoMedia/PWM 6.0 released shortly after implementation completed API completely changed Rewrite of custom code approx. $50K – no way! Could only use for printing/publishing (fixed arcs
bug)
![Page 15: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/15.jpg)
If only there was a flu shot…
GeoMedia 6.1 released a few years later Another API change Cost to catch up custom code rose to $100K
GeoMedia 5.2 won’t run reliably on Windows 7 Resorted to maintaining Windows XP virtual
machines for compatibility
![Page 16: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/16.jpg)
Another hardcopy improvement though…
![Page 17: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/17.jpg)
Migration time again…
Began looking for a replacement to GeoMedia in 2011
Esri ArcGIS chosen, planning begun Introduced to ArcGIS Local Government
Information Model (LGIM) Standardized data model Supported by many 3rd party systems (CMMS,
CCTV, hydraulic modeling, ERP, etc.)
![Page 18: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/18.jpg)
…and FME rescues us again!
Splitting data migration workloads with Esri Both CentralSan and Esri are using FME
Successfully writing to both Esri file geodatabase
and Microsoft SQL Server enterprise geodatabase formats
About 60% complete with FME workbench development and testing
![Page 19: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/19.jpg)
What does the final cure look like?
Eliminate hardcopy maps entirely Focused applications for each
department/workgroup Mobile solutions for map viewing and field data
collection GIS will be authoritative inventory source for
assets Tight integration with CMMS and CCTV systems
![Page 20: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/20.jpg)
Now for some tips…
Three things we picked up along the way
![Page 21: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/21.jpg)
Tip #1 – Keep DRY
In programming, DRY is a principle of code organization Don’t Repeat Yourself Move code that will repeat over and over again into
a subroutine or stand-alone module Applies equally well to FME If you will reuse sequences of transformers
multiple times, put them in custom transformers Export custom transformers to share and reuse in
different workbenches
![Page 22: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/22.jpg)
Tip #1 – Keep DRY
![Page 23: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/23.jpg)
Tip #2 - SchemaMapper
Replaces multiple AttributeCreator, AttributeMapper, FeatureTypeFilter transformers
Can remap feature types too Uses CSV file, Excel file (.xls/.xlsx), or database
table to store mappings/values Benefits: Reduce transformer count/complexity External table can be edited/maintained by
someone other than FME workbench author
![Page 24: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/24.jpg)
Tip #2 - SchemaMapper
![Page 25: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/25.jpg)
Tip #3 – Version Control
Changes to workbenches can be tracked in FME:
![Page 26: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/26.jpg)
Tip #3 – Version Control
Workbench files are text files, so they can be effectively managed in various version control systems Examples: Git, Mercurial, SVN, Team Foundation
Provides ability to track history, roll back revisions if needed
Using on-premise or hosted version control services provides a backup & recovery solution Examples: Github, BitBucket
![Page 27: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/27.jpg)
Tip #3 – Version Control
![Page 28: Curing Migration Flu or: How I Learned to Stop Worrying and Love the LGIM](https://reader031.vdocuments.net/reader031/viewer/2022020208/55a6af4d1a28ab475c8b45cf/html5/thumbnails/28.jpg)
Thank You!
Questions?
For more information: Carl Von Stetten ([email protected])
Web: http://centralsan.org Twitter: @cfvonner