GIS Database Management in a Near Real-Time World

Mary Buffington, GISP, Project Manager

Eastern Shore Regional Shore Cooperative (ESRGC)

Doug Adams, GISP, GIS Program Manager

Baltimore County Department of Public Works

Maryland Society of Surveyors53rd Annual Fall Conference &

Exhibition

Jedi Surveying

•Mason Dixon Boundary - 1767• Survey of the Coast – 1807

• U.S. Coast and Geodetic Survey – 1878• National Geodetic Service - 1970

•National Spatial Reference System• Horizontal Datums

• North American Datum of 1927 (NAD 27)• North American Datum of 1983 (NAD 83)• North American Datum of 1983 HARN (NAD 83 HARN)• North American Terrestrial Reference Frame of 2022 (NATRF2022)

• Vertical Datums• National Geodetic Vertical Datum of 1929 (NGVD 29)• North American Vertical Datum of 1988 (NAVD 88)• North American-Pacific Geopotential Datum of 2022 (NAPGD2022)

Defining the Boundaries of Practice …

•Common Interest and concerns• They create and use map-related information• They serve the public• They are concerned for the health, safety and welfare of the

public• They share interest in technologies for location-based data• They provide answers to questions that start with: “Where?”

https://www.urisa.org/clientuploads/directory/Documents/Advocacy/DefiningBoundaries_WhitePaperDec2018.pdf

Issues Observed by Surveyors Concerning Use of GIS

•Using GIS maps and data to determine precise locations on the ground•Using GIS maps and data without being aware of their quality

• Accuracy• Currency• Data Source• Method of Compilation

This information is referred to as metadata.

Surveyors’ Expertise Includes

•Measuring and recording the location of boundaries, fixed works and terrain•Assessing and mitigating the locational errors in maps•Producing AUTHORITATIVE (source) data/documents for the location of boundaries, monuments, reference points, fixed works, elevation, and the shape of the earth• Interpreting cadastral law and practices

GIS Professionals’ Expertise Includes:

•Creating geospatial databases and software applications•Analyzing the location and characteristics of mapped features•Compiling and combining geographic data into new, map-based visualizations•Developing new spatial data from analysis of multiple sources•Managing GIS systems and operations

Derivative

Derivative 1

Derivative 1.1

Derivative 1.2

Derivative 2

Derivative 2.1

Derivative 2.2

Property Line Survey

PlatCadastral

Map

Plats Feature Class

Subdivision Feature Class

Zoning Map

Master Water & Sewer Plan

Property Feature Class

… and the list goes on. Most of my organizations non-asset data can be linked back to property data in some way.

Survey Control

Utilities

Sewer

Water

Storm

Transportation

Street Centerlines

Edge of Pavement

Changing Technology

•Manual vs. Analytical vs. Softcopy photogrammetry• Tablet digitizing vs. Heads-up digitizing• Film camera vs Digital camera•Airplane vs. Drone• ...and it’s going to continue to change

Geospatial Resources

•Plats.Net (http://www.plats.net/pages/index.aspx)•Maryland Land Records (https://mdlandrec.net/main/index.cfm)•Map Service Center (https://msc.fema.gov/portal/home)

National Geospatial Data Act 2018Codifies• Federal Geographic Data Committee (DOI) - 1990• National Spatial Data Infrastructure - 1994• National Geospatial Advisory Committee - 2008

• Data Themes• Geospatial Data Standards

• Geospatial Data Portal – GeoPlatform 2011 (https://www.geoplatform.gov/)

2M

• Organizational tool to store and manage data• Data organization using feature datasets

• Set coordinate system of the data• Groups similar data together - sewer, water, stormwater, etc.

• Allows tabular rules for data integrity• Domains - set of valid values for data

• Pipe material: CMP, PVC, etc.• Pipe size: 8 inch, 12 inch, etc.• Set default

• Subtypes - classify different types of features• Building type - primary, secondary• Road - paved, gravel, dirt• Set default

Geodatabase Benefits

• Allows spatial rules for data relationships• Topology - defines and enforces data integrity rules

• “Gravity Mains must not intersect”• “Gravity Mains must not have dangles”• “Manholes must not overlap”• “Parcels must not have gaps”

• Geometric Network - model common networks and infrastructure in the real world• Find disconnected network elements• Find loops or circuits• Calculate the shortest path upstream from one point to

another• Google maps!

Geodatabase Benefits

• Relationship Class• Objects can relate to other objects (buildings related to

parcels)• Objects can relate to tables (parcels can be associated with

owner information)• Manage the associations between objects in one class to

objects in another class• Parcel data related to Real Property Data

Geodatabase Benefits

Geodatabase Design

• Features as points, lines, polygons•Continuous surfaces - rasters, TIN...•Map graphics - text labels (annotation)•How do these items relate?•Model the user’s view

• Identify the functions that support the organization’s goals and objectives

• Identify the data required to support the functions• Organize the data into logical sets of features

Database Types• Enterprise

• Data stored in SQL Server, Oracle, etc.• Recommended for data stored and managed in relational database• Storage size limited to DBMS limits

• File Geodatabase• Recommended for data stored and managed in file system folder• Size limited to 2 TB per dataset (can be raised to 256 TB)

• Personal Geodatabase• Original format for gdb stored and managed in Access data files• File limit size of 2 GB• Concurrent use eventually degrades for large numbers of readers

File Types

• File Geodatabase (FGDB)• One editor• Many Users

•Personal Geodatabase (PGDB)• One editor• One User

• Shapefile - very prevalent...very limiting• Triangular Irregular Network (TIN)•GRID

Lidar Data Storage

Quality LevelPoints Per

MeterLAZ

(sq mi)

LAS(2.5X)(sq mi)

LAS(10X)

(sq mi)

Baltimore County

(682 sq mi)

1 8+ 0.38 GB 0.95 GB 3.8 GB 648–2591 GB

2 2-8 0.16 GB 0.4 GB 1.6 GB 273-1091 GB

3 1-2 0.05 GB 0.125 GB 0.5 GB 85-341 GB

In comparison, a days worth of data from a single CORS station is:• 1 second interval – 25 MB• 5 second interval - 5 MB

Geodatabase Features

•Points, lines, polygons• Feature type is dependent on scale

• River - line vs polygon• City - point vs polygon

• Topology• Within a layer and between layers• Reinforces geographic relationship between features

Importing data to a Geodatabase

• Shapefiles• CAD(ish)• Other geodatabase feature classes/features• Tables (coming soon in Baltimore County…)• Raster data

• Raster datasets• Raster catalog • Mosaic dataset

• ...the list goes on • If you can import it - you can export it!

Attribute Table

•Provide information about the feature• Set of data elements arranged in rows and columns

• Row = one record/feature• Column = Field

• Joins and Relates• “Join” geographic features and tables based on a common field

• ACCTID...• One to one - One owner owns one parcel

• Relate - establishes a connection between records in two tables• One to many - one parcel has many permits

Data Types- Raster/Vector

•Vector• Points, lines, polygons• Is the feature represented on a map?• Is the shape of a feature important?• Will the feature have different representations at different scales?

•Raster• Cells with values, continuous• DEM, imagery…• LiDAR not meant to be survey grade

Attachments

Advantages• Stored in geodatabase• Security• Enable deduplication•Any application supported format

• Image• Video• Word• Excel

https://bc-gis.maps.arcgis.com/apps/webappviewer/index.html?id=66c733159a024a9cbb20134f93610b4e

Related Objects•Attachments - deduplication!• External Data (joins, relates, etc.)• Hyperlinks

Data Models

• Local Government Information Model (LGIM)• Transportation•Water Utilities• ElectricUse as a guide - easily customizable

http://appsforms.esri.com/products/download/index.cfm?fuseaction=download.all

2D

Baltimore CountyDPWData Themes

Vertical Data Models

Sewer Treatment

Plants

Sewer Pump

Stations

Sewer Grinder Pumps

Water Treatment

Plants

Water Wells

Dams Levees

Theme Documentation - Spreadsheet

• Attribute Listing• Logical Model• Physical Model• Domains• Relationship Classes• Data Quality• Metadata• Business Rules• Schema Modifications• Topology

Theme Documentation – WordOverview• Purpose• Description• Stakeholders

• Program Manager• Support Staff

• Update Frequency• Date of Last Update• Supported Program(s)• Asset Group• Requirements• Data Locations• Views• Applications• Business Rules• Tools• Source Materials

Overview (con’t)• Geodatabase Development• Schema• Layer Files• Definition Queries• Map Documents• Queries• Supporting Documents

Map Production• Page Size• Marginalia• Symbology• Special Notes• Description

Geodatabase Development• Compilation Rules• Assumptions• Remaining Tasks• Compilation Statistics• Questions• Compilation Issues

Maintenance• Maintenance Workflows• Metadata Updates

Maintenance Record

File Management – Theme Based• Applications – Documentation and specifications for application used to support the creating, maintenance

and quality control of this data.• Documentation – This folder is where the Geodatabase Design (xls) and Geodatabase Documentation (docx)

documents should be stored. Any schema change documents should also be located in this folder. • Expressions – This folder should contain any expressions used in map production, definition queries, etc. saved

from ArcMap.• Historic – This folder should contain any documentation created prior to the current documentation standard.

Folders can be created as required to organize the materials. The contents of this folder can be compressed.• Layer Files – This folder should contain any layer files created for web applications or map production. The

layer files should be documented in the geodatabase documentation document.• Maps – This folder should contain any maps produced specifically from the related data. Subfolders can be

created for successive years as required.• Maintenance – This folder should include the maintenance record (if not included in the geodatabase

documentation document) and source materials.• Research – This folder should contain information on research performed related to the design, compilation or

maintenance of the geodatabase. The contents of this folder can also be compressed.• Scripts – This folder should contain any scripts used in the compilation, maintenance, presentation of the data.• Source Material – This folder should contain the source material used in the compilation of the data. The

folder should be organized to sufficiently ensure the content is clearly understood.• Views – The documentation and specification for materialized or dynamic views created specifically for this

data. 2M

https://tidesandcurrents.noaa.gov/map/index.html

https://mdfloodmaps.net/map/

https://coast.noaa.gov/slr/#/layer/fld/0/-8514647.371503906/4764471.237285254/14/satellite/none/0.8/2050/interHigh/midAccretion

https://waterdata.usgs.gov/nwis/rt

https://earthquake.usgs.gov/

Latest earthquakes (past 24 hours, M2.5+)

Data Services•Map - view

• https://bcgis.baltimorecountymd.gov/arcgis/services• https://geodata.md.gov/imap/services• https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHL/MapServ

er• https://idpgis.ncep.noaa.gov/arcgis/services• https://hydro.nationalmap.gov/arcgis/rest/services/nhd/MapServer

• Feature - view, query, download...• https://geodata.md.gov/imap/rest/services/Environment/MD_Protect

edLands/FeatureServer• LiDAR

• https://lidar.geodata.md.gov/imap/rest/services

https://viewer.nationalmap.gov/services/

https://opendata.maryland.gov/

2D

I - IdentificationII - Data QualityIII - Spatial Data OrganizationIV - Spatial ReferenceV - Entity and AttributesVI - DistributionVII - Metadata Reference

… and then there’s Feature Level Metadata

Metadata - FGDC

Data OrganizationNSDI Core/Expanded

Core• Geodetic Control• Hydrology• Elevation• Cadastral• Orthoimagery• Transportation• Governmental Units

Extended• Biota• Climate & Weather• Geology• Land Use-Land Cover• Real Property• Soils• Utilities

Data Organization - ISO 19000 Metadata Theme Categories

• Farming• Biota• Boundaries• Climate/Meteorology/Atmosphere• Economy• Elevation• Environment• Geoscientific Information• Health

• Imagery/Base Maps/ Earth Cover• Intelligence Military• Inland Waters• Location• Oceans• Planning Cadastral• Society• Structure• Transportation• Utilities/Communication

Codified Layers

•Chesapeake Bay Critical Area•Commercial Revitalization Districts•Growth Tiers• Enterprise Zones•Master Land Use Plans•Master Water Sewer Plans•Metropolitan District• Zoning• Zoning Variance

Open Data Portals

Data.gov - https://www.data.gov/

HIFLD Open - https://services1.arcgis.com/Hp6G80Pky0om7QvQ/ArcGIS/rest/services

Md iMAP - https://data.imap.maryland.gov/

ReferencesBolstad, Paul. 2019. GIS Fundamentals, 6th Ed. XanEdu

DeMers, Michael N., 2009. Fundamentals of Geographical Information Systems, 4th Edition. John Wiley & Sons

Harmon, John E. and Anderson, Steven J., 2003. The Design and Implementation of Geographic Information Systems, John Wiley & Sons. ISBN 978-0-471-20488-6

Robinson, Arthur, et al., 1995. Elements of Cartography, 6th Edition. John Wiley & Sons, Inc.

Tomlinson, Roger. 2013. Thinking About GIS: Geographic Information System Planning for Managers, 5th Ed. Esri Press

Wolf, Paul, et al., 2014. Elements of Photogrammetry with Application in GIS, 4th Edition. McGraw-Hill.

Zeiler, Michael, 2010. Modeling Our World, Second Edition: The ESRI Guide to Geodatabase Concepts, Esri Press

Zeiler, Michael, et al. 2004. Designing Geodatabases Case Studies in GIS Data Modeling, Esri Press

QuestionsMary Buffington, mebuffington@salisbury.edu, 410.548.3844Douglas Adams, dadams@baltimorecountymd.gov, 410.887.3451