eeos 381 -spatial databases and gis...

Lecture 6

Introduction to Enterprise Geodatabases and ArcSDE

EEOS 381 - Spatial Databases and

GIS Applications

EEOS 381 - Spring 2015: Lecture 6 2

What is ArcSDE?What is ArcSDE?

Technology for multiuser geodatabasesintegrated into Esri’s ArcGIS for Desktop and ArcGIS for Server products

– “Spatial Database Engine”

– Manages spatial data within an enterprise RDBMS

Gateway (“middleware”) between GIS clients and an enterprise RDBMS

Enables you to store, access, and manage spatial data within an RDBMS such as

– Oracle, Microsoft SQL Server and SQL Server Express, IBM DB2, IBM Informix (Commercial)

– PostgreSQL (Open Source)



Typically installed on a server (for Enterprise GDBs) for Windows, Unix and Linux

Desktop and Workgroup GDBs can be installed on a server or local PC

Multiuser GDBs

have all the

functionality of

file and personal

GDBs, including

all the

“application

logic”



Critical when you need to manage long transactions and versioned-based workflows such as

–Support for multiuser editing

–Distributed editing

–Managing historical archives


Notes to Keep in MindNotes to Keep in Mind

RDBMS physically stores relational tables

– vector, raster, tabular data (expanding functionality of RDBMS), metadata, …

ArcSDE “interprets” the contents for use in GIS software and clients

– ArcGIS, ArcIMS, ArcView 3.x (via Database Access extension) and ArcInfo Workstation, non-Esri and OpenSource too (like MassGIS’OLIVER)


ArcSDE as Part of ArcGISArcSDE as Part of ArcGIS

“3-tier architecture”

Clients

ArcSDETechnology

RDBMS

Prior to ArcGIS 9.2, Esri sold ArcSDE as a core produ ct. At ArcGIS 9.2, ArcSDE is no longer a separate product. It is now

“ArcSDE Technology”.


Benefits of ArcSDEBenefits of ArcSDE

Centralized spatial database– Large volume of data

– Seamless, non-tiled layers

– Rapid spatial searching

– Storage and integration of vector, raster, tabular data

– Database and application portability

Concurrent access for MANY users– Versioning/Multi-user editing

– Clients connect over TCP/IP network (e.g. WEB)

– “Direct connect” also available



Use of robust RDBMS environment– Security, backup and recovery

– Scalability, Indexing

– Referential and data integrity

– Can access GIS attribute data at the db level via

SQL*Plus, OSQL, or in MS Access with ODBC

connection, etc.

– Link to data in existing RDBMS

– Supports long transactions and Views

– SDE administrators may be assisted by system

administrators and RDBMS administrators



Spatial queries directly at SQL prompt if RDBMS supports spatially-enabled data types– Oracle Spatial, DB2 Spatial Extender, Informix Spatial

DataBlade, SQL Server 2008 spatial data type

Use C, Java API for custom app. dev.

Fully integrated with ArcGIS

Can batch data import/export and other processes using Administrator Commands


Drawbacks of ArcSDEDrawbacks of ArcSDE

Requires separate purchase of RDBM$ (but PostgreSQL is free)

– Need a DBA or be trained in your RDBMS software for backups, tuning (require time, money, expertise)

Can take a while to master intricacies, work around RDBMS errors


Types of ArcSDE(ArcSDE GDBs are Scalable)

Types of ArcSDE(ArcSDE GDBs are Scalable)

EnterpriseGeodatabase

Workgroup Geodatabase

Desktop Geodatabase (known as Personal SDE at 9.2)

Application Scenario

Large-scale enterprise applications Small- to medium-sized departmental applications

Small teams or a single user who requires the functionality of a multiuser geodatabase

Supported RDBMS DB2, Informix, Oracle, PostgreSQL, SQL Server

SQL Server Express SQL Server Express

Management Interface

ArcCatalog, RDBMS, ArcSDEcommand line

ArcCatalog ArcCatalog

Storage Capacity Depends on the server 4 GB 4 GB

Licensing Availability

ArcGIS Server Enterprise ArcGIS Server Workgroup ArcGIS Engine, ArcInfo, ArcEditor

Supported OS Platform

Any platform Windows Windows

Number of Concurrent Users

Unlimited editors and readers 10 editors and readers 1 editor and 3 readers

Network Application

Intranet and Internet Intranet and Internet Desktop and local network use

Differentiating Characteristics

Supports versioning and multiuser editing; Supports spatial types; integration with enterprise IT

Supports versioning and multiuser editing

Supports versioning

N E W @ 9 . 2


Accessing ArcSDE in ArcGISAccessing ArcSDE in ArcGIS

Enterprise

Desktop and Workgroup


Accessing Enterprise ArcSDEAccessing Enterprise ArcSDE

• Double-click• Enter the parameters

as shown at right.• Note: the Password is the

same as the User Name • To connect to your data

(to edit), you would enter your Oracle username and password

• Click OK.

These parameters get saved

to a .sde file in your profile

(on the C: drive). You can

copy this file to H: for use

on any PC. You can also use the [email protected]

connection in S:\ge381_s15\

The password for DATAREADER

is DATAREADER1


ArcSDE Feature ClassesArcSDE Feature Classes

GDB “container” for vector data

– One geometry type (point, line, polygon, annotation, etc.)

per feature class, and attributes

– Owned by RDBMS users (i.e. Oracle accounts, like

MGISDATA)

Stored as series of tables in the RDBMS

– Joined at query time so that the client (i.e. ArcCatalog) sees

one object (layer) in the geodatabase

– One row is one feature (single or multi-part)

Spatial column identifies geometry

– “SHAPE” field

– in RDBMS, actual coordinates stored in BLOB (binary long

object) or long raw formats

shape + attributes = feature



ArcSDE manages integrity of tables (i.e. when a layer is renamed, deleted, edited)

– users should never alter these tables’schemas

• i.e. add columns to layers in ArcCatalog or ArcMap only !!

– You CAN use SQL*Plus or MS Access to edit business table attributes (and to add rows to non-feature class tables – but be careful!



Types of storage

– ST_GEOMETRY

• Base (business) table with attributes and spatial column, and spatial index table

• Default at version 10

– SDELOB (binary geometry type)

• Business (attributes), F (spatial), and S (spatial index) tables

– Each type is linked to a configuration keyword as defined in the SDE.DBTUNE table. Choose the keyword when creating data.

For more details see:http://help.arcgis.com/en/arcgisdesktop/10.0/help/i ndex.html#/Feature_class_storage/002n0000007m000000 /


Feature class tables in the RDBMS (SDELOB storage)


Business table– stores attributes

– name in RDBMS is feature class name (e.g.

TOWNS_POLY), as seen in ArcGIS. Its actual

appearance depends on the RDBMS:

• Oracle: OWNER.TABLENAME

– Ex: MGISDATA.TOWNS_POLY

• SQL Server: DATABASE.OWNER.TABLENAME

– Ex: SDE.MGISDATA.TOWNS_POLY


Feature table (“F” table)– stores geometry in binary column (X,Y,Z,M

values), plus envelope (bounding rectangle)

for each feature, area & length

– named F<layer_id> in RDBMS (layer_id is

primary key in SDE.LAYERS table)

– one-to-one relationship between business and

F table, via unique feature identifier

(“OBJECTID”, always indexed)



*** With ST_GEOMETRY storage the business table stores the geometry, and there is no “F” table


Spatial index table (“S” table)– stores grid tiles and envelope (X/Y extent of

each tile)

– named S<layer_id> in RDBMS

– one-to-many relationship between F and S

table

– used during spatial searches (pan/zoom) for

optimum data retrieval



*** With ST_GEOMETRY there is also a spatial index table for each feature class


“Adds” table (“A” table)– stores added features during ArcMap

versioning edit session

“Deletes” table (“D” table)– stores deleted features during ArcMap

versioning edit session

aka Delta (change) tables – created only when a feature class is Registered as Versioned

Named A<registration_ID> and D<registration_ID>(registration_ID is primary key in SDE.TABLE_REGISTRY table)

Feature class tables in the RDBMSFeature class tables in the RDBMS


Column indexes may be built on attributes in feature class business table

Improves performance when using

joins, relates, and relationship

classes.

Attribute IndexesAttribute Indexes


Spatial IndexesSpatial Indexes

Clients use spatial filter to reduce query results and speed up data retrieval

– Ex.: Only fetch visible features for display

ArcSDE uses spatial index to reduce I/O

– Eliminates need for full table scan

– Based on standard feature of RDBMS

Format of the spatial index is determined by the RDBMS being used

– Grid tile or R-tree (Oracle Spatial) structure



ArcSDE for Oracle uses grid tile structure

– Spatial index built by applying a

grid to the feature class

– ArcSDE records which features

fall within each grid cell in an

index table (“S” table), based on

feature envelope

– A feature that falls within many

cells is listed in each

– Grid cells with no data are not

listed in table

– A layer may have 1 to 3 index

grids, though usually one is

sufficient

– When you create a vector

feature class, ArcGIS

automatically determines the

optimum grid cell size (in the

units of the data, e.g. feet,

meters) and the number of

grids.

The feature class is overlaid by grid cells to create the spatial index.



Manage in Indexes tab in the Feature Class Properties box

(you must be the data owner)

Can also use:


Loading Data into ArcSDELoading Data into ArcSDE

Use ArcToolbox Conversion Tools > To Geodatabase

Right-click spatial database connection and choose Import

– Supported data types or XML docs

Right-click existing SDE layer and choose Load > Load Data…

Create new empty FC with desired schema (can import from a

table) and Load in features from any data source

Right-click non-SDE layers and choose Export > To

Geodatabase

Use SDE commands

– shp2sde , cov2sde , tbl2sde (for tables)

Use special tools/dialogs for labels and annotation

In all cases, you must specify a configuration keyword (your

SDE administrator will tell you if you should use DEFAULTS or

another keyword


After Loading Data into ArcSDEAfter Loading Data into ArcSDE

Right-click and

– Assign privileges to data – way to control access

• Enter db user or role

• Check SELECT, UPDATE, INSERT, DELETE

– Analyze

• Updates DBMS statistics

• Check/clear components to analyze

Rename layers?

– Loaded data named the same as existing data will have “_1” in new name

Take a look at the data!!!


Raster Data in the RDBMSRaster Data in the RDBMS

>> ArcSDE/GDB raster layers are known as “raster datasets”



Benefits

– RDBMS security, data management and retrieval

– Common data format

– Supports large seamless images (mosaics), raster catalogs, and mosaic datasets

– Compression

• LZ77 lossless or lossy

• JPEG and JPEG 2000 lossy

– Pyramids stored in database for improved display performance



Loading

– ArcCatalog / ArcToolbox (all supported formats)

– Use sderaster command (for 1 to 3-band TIFF,

BSQ, or ArcSDE raster)

– Then:

• Build Pyramids

• Calculate Stats

• AnalyzePyramids

enable fast display at any

scale



SDE.RASTER_COLUMNStable …

– Stores information on every raster dataset in the database, one entry per dataset

– Each layer has unique ID -RASTERCOLUMN_ID - assigned by ArcSDE

For more details see:http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Raster_

storage_types_in_an_ArcSDE_geodatabase/002n0000006v000000/



… and a series of many tables– Raster layer Business table

(OWNER.IMAGE_NAME – as seen in ArcGIS)• Link between supporting tables and RASTER_COLUMNS table

– Raster layer Block Table(SDE_BLK_<RASTERCOLUMN_ID>)

• Stores actual pixel data

– Raster layer Metadata Table(SDE_RAS_<RASTERCOLUMN_ID>)

• contains raster description

– Raster layer Band Table(SDE_BND_<RASTERCOLUMN_ID>)

• Information about each raster band

– Raster layer Auxiliary table(SDE_AUX_<RASTERCOLUMN_ID>)

• Stores colormaps and statistics


ArcSDE Uses Client/Server ModelArcSDE Uses Client/Server Model

All data stored in RDBMS accessed over a TCP/IP network

All data is retrieved through SQL (Structured Query Language) queries

– SELECT statements fetch data (see pages 259-260 in textbook)

Query filters limit the rows returned

– Attribute and Spatial

Clients display data

Two connection methods– Application Server– Direct Connect


ArcSDE Application Server ConnectionArcSDE Application Server Connection

Traditional ArcSDEconnection method– Available for all supported

RDBMS

GIS client requires no additional software

Server requires ArcSDEinstance– giomgr spawns one

dedicated gsrvr process per connection, using RDBMS authentication

Performance considerations– Decreases client load

– Decreases network load

– Increases server load ArcSDEgiomgr DataData

ArcSDEgsrvr

ArcGIS

RDBMSserver

• Link between application and database• Processes client requests• Sends data from client to server• Updates & maintains ArcSDE metadata• Active until user disconnects

• Listens for, monitors, and maintains connections, spawns gsrvrprocesses

5151

TCP/IP network


ArcSDE Direct ConnectArcSDE Direct Connect

Client connects directly to RDBMS

Client does its normal job AND performs function of the gsrvr , using a software

library called a Direct Connect driver

No ArcSDE instance used

No gsrvr or giomgr processes required

on server

Performance considerations

– Increases client load

– Increases network load

– Decreases server load (most appropriate for robust clients on hearty computers)

– More stable than App Server connection

ArcGIS client

(with Direct Connect Driver)

RDBMS

For more details start with:http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/What_is_a_direct_connection_to_a_geodatabase_in_Oracle/002n00000035000000/

This is the default at

10.1


Services FileServices File

Stores port number (5151) used by ArcSDE

for communication between client and RDBMS

Add the following line:

esri_sde 5151/tcp

to the SERVICES file inC:\WINDOWS\SYSTEM32\DRIVERS\ETC

or C:\WINNT\system32\drivers\etc

or other similar location.


TNSNAMES.ORA FileTNSNAMES.ORA File

Text file that stores information that defines Oracle

database addresses for establishing connections to

them.

This file normally resides in the

%ORACLE_HOME%\NETWORK\ADMIN folder.

Required for direct connection to SDE from ArcGIS (and

to use SQL*Plus)

Syntax:

<addressname> = (DESCRIPTION =

(ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(Host = <hostname>)(Port = <port>))

)(CONNECT_DATA =

(SERVICE_NAME = <sid>))

)


SDE.DBTUNE TableSDE.DBTUNE Table

RDBMS table (owned by SDE user) that stores storage tuning parameters in an enterprise ArcSDE geodatabase

Based on keywords– “DEFAULTS” used for all feature classes loaded without keyword

specified

– Different keywords can be set up by the SDE administrator for vector data, raster data, topology, and for different users and editors

– Control where data are stored - i.e. which tablespace (Oracle) or file group (SQL Server) for feature tables, indexes.

– If you want users to be able to write to (i.e., store data in) certain parts of a database, the SDE administrator would set up a keyword (like ##STUDENTS), and in ArcGIS dialogs that have a dropdown or textbox to specify a configuration keyword, the userwould enter STUDENTS.


ArcSDE RepositoryArcSDE Repository

Administrative information

– Feature class name

– Unique feature class ID, etc. …

AKA “ArcSDE system tables”

sde.raster_columns

sde.table_registry

sde.layersLAYER_IDTABLE_NAME...

A few of the many tables owned by the sde user

See system tables diagram at:http://help.arcgis.com/en/arcgisdesktop/10.0/help/0 02n/pdf/sdesystables_diagram.pdf

Full details at:http://help.arcgis.com/en/arcgisdesktop/10.0/help/i ndex.html#/ArcSDE_geodatabase_system_tables/002n0000008m000000/


ArcSDE Administration ToolsArcSDE Administration Tools

Command-line utilities for managing the ArcSDEserver

– Executed at operating system prompt

– May be used remotely

– Command syntax found in the ArcSDE Developer Help

• See http://help.arcgis.com/en/geodatabase/10.0/admin_cm ds/support_files/admincmdref.htm

– Benefits

• Easy to format

• Easy to debug

• Reusable

• Record of work (can write output to log files)

• Unattended execution (can batch)


ArcSDE Administration ToolsArcSDE Administration Tools

Example: the sdemon command

– Purpose: ArcSDE server monitor, used to check connection statistics. The sytax is:sdemon -o info -I <{users | config | stats | locks }> [-I <instance>]

– Required parameter sdemon -o info

– Optional parameter [ ]

– Substitute a value < >

– Choose a value from list for the <instance>

argument:-I <{users | config | stats | locks }>

Others: sderaster , cov2sde , shp2sde , sdeexport , sdeimport , sdelayer , sdetable ,

...


VersioningVersioning

ArcSDE Geodatabase mechanism to allow multiple users to simultaneously edit the same database, including the same layers and features, without explicitly applying locks to prohibit other users from modifying the same data

Part of “multi-user editing” capability of SDE geodatabases

No need to extract or make copies of data for editing



What is a “version”?:– “In databases, an alternative state of the database

that has an owner, a description, a permission (private, protected, or public), and a parent version. Versions are not affected by changes occurring in other versions of the database.”

- From ESRI GIS Dictionary

– You can think of a version as your own “doorway” to the database, allowing you to make changes that don’t affect the database as a whole.

– Also see ESRI Help:• http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/What_

is_a_version/00270000000q000000/

• http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/A_quick_tour_of_versioning/00270000000r000000/



Use versioning to:

– Manage alternative engineering designs

– Solve complex what-if scenarios without impacting the production database

– Create point-in-time representations of the database

– Conflict resolution dialog allows DBA to choose proper version



Versions are not copies of the database or layers

– Only store edits made to database in A and D tables

Versions are named– default version (always present) is SDE.DEFAULT

– syntax: OWNER.VERSION_NAME

– AKA a named “state” of the database

Maintain parent-child relationships (see chapter

7 in Modeling Our World and chapter 10 in Building A Geodatabase)



Isolates user's work across multiple edit sessions, allowing the user to edit without locking features in the production version or immediately impacting other users.

Supports undo/redo capability

Multiple users can access a version

A user may access multiple versions– What the user can do is based on version

permissions

Contains access to all the datasets in the geodatabase



Versions have permissions:

– Public - any user may view the version and modify features

– Protected - Any user may view version, only the owner may edit

– Private - Only the owner may view the version and modify features

Notes:

– Version owner can rename, delete, change access

– SDE user can see and delete all versions regardless of their permissions



How tables react to edits:

– Insert (add) feature

• record (OBJECTID) added to A table

–Delete feature

• record (OBJECTID) added to D table

–Update feature

• records (OBJECTIDs) added to A and D tables


Versioning Process Step 1 – Data OwnerVersioning Process Step 1 – Data Owner

In ArcCatalog:

1.Register data with geodatabase, if necessary

2.Register data as Versioned (makes data editable)

• A & D tables added to feature class:

– A<registration_ID> stores adds

– D<registration_ID> stores deletes

3.Grant SELECT, UPDATE, INSERT, DELETE privileges to all editors

To register an object as versioned, right-click it in ArcCatalog and choose ‘Register As Versioned…’. You must be connected as the owner of the dataset.


Versioning Process Step 2 - EditorVersioning Process Step 2 - Editor

1. Create version, set permission (ArcCatalog)

2. Change version in your spatial database connection and add feature class (or feature dataset) to ArcMap

3. Make and save edits in ArcMap

4. From Versioning Toolbar in ArcMap:

– Reconcile with parent (other edits in parent seen by child)

– Post changes to parent (changes in child seen by parent)

• parent owner needs to do this if parent version is not ‘public’

– parent makes version of child and posts


SDE.DEFAULT(PROTECTED)

SCOTT.OS_DBA(Protected)

MTRUST.MYEDITS(protected)

JKERRY.MYEDITS(protected)

SCOTT.MTRUST_QA(protected)

SDE.READY_FOR_DEFAULT

Layer edited and saved

Versioning Process –Editing

Versioning Process –Editing


Versioning Process Step 3 – SDE userVersioning Process Step 3 – SDE user

1. Compressing the Database– Done as SDE user, with Compress button added to a toolbar from

Customize box in ArcCatalog, or with ArcToolbox tool

– Moves records in A & D tables into base (business, F, & S) tables

– Removes all redundant rows and states not referenced by a version

– Improves performance

– Two types:

• Full compress: When editing is done and all reconciling and posting is complete, after deleting all versions

• Partial compress: done anytime, but locks may prevent deletion of certain states

2. Analyze data after compression

Note: “Compact” is procedure for PGDBs


SDE.DEFAULT(PROTECTED)

SCOTT.OS_DBA(Protected)

JKERRY.MYEDITS(protected)

SCOTT.MTRUST_QA(protected)

SDE.READY_FOR_DEFAULT

MTRUST.MYEDITS(protected) Layer edited

and saved

Versioning Process -Compressing

Versioning Process -Compressing

Compress Geodatabase Start the process all over again


Non-versioned EditingNon-versioned Editing

Editing of source data directly

– Features and attributes

The last saved edit is final

No ability to undo/redo changes

– The only way to “undo” back to last edit is to stop editing without saving changes

The object (feature class or table) does not have to be registered as versioned)

To “turn on” non-versioned editing, in ArcMap, go to the Editor toolbar, click on the Editor menu dropdown, and choose Options…. Then, in the Versioning tab, uncheck the ‘Edit a version…”checkbox.


GDB ReplicationGDB Replication

Allows you to create copies of data across two or more

geodatabases such that changes to the data may be

synchronized.

Requires ArcInfo or ArcEditor

For more information, start with “Understanding

distributed data” in ArcGIS Help for full details:http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Unde

rstanding_distributed_data/002700000020000000/


GDB ArchivingGDB Archiving

Provides the functionality to record and access changes

made to all or a subset of data in a versioned

geodatabase.

The mechanism for capturing, managing, and analyzing

data change (“historical versions”).

See “Geodatabase archiving” for more details:http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/Geod

atabase_archiving/002700000045000000/


ViewsViews

A “virtual table” in a database that displays certain information in the database, based on a view definition

Not a table itself; the definition is stored and the view appears as a table when accessed by a user

Advantages

– Can limit rows (use “where clause”) and can include only certain fields

• Simplicity and security

– Can assign alias (more intuitive) field names

– Can include joins to other tables

When SHAPE field is included in an SDE view, the view appears as an ArcSDE Feature Class (“Spatial view”)

Use the sdetable –o create_view admin command

See: http://help.arcgis.com/en/arcgisdesktop/10.0/help/i ndex.html#/What_is_a_view/002n00000027000000/

eeos 381 -spatial databases and gis...

Documents