data management in the multiuser geodatabase · pdf file• all loaded data is converted to...

Data Management in the Multiuser Data Management in the Multiuser GeodatabaseGeodatabase

Emily ApseyEmily ApseyESRI InstructorESRI Instructor

Agenda Agenda

•• Geodatabase architectureGeodatabase architecture

•• Vector data storage architectureVector data storage architecture–– CClient Server Modellient Server Model–– Vector data loading workflowsVector data loading workflows–– Data Maintenance StrategiesData Maintenance Strategies

•• Raster data storage architectureRaster data storage architecture–– Raster data loading workflowsRaster data loading workflows

•• Personal for Microsoft AccessPersonal for Microsoft Access•• FileFile•• ArcSDE technology at three levels:ArcSDE technology at three levels:

–– DesktopDesktop–– WorkgroupWorkgroup–– EnterpriseEnterprise

PersonalAccess

File

ArcGIS Desktop

ArcGIS Server

FilePersonal(Access)

WorkgroupEnterprise

Desktop

Geodatabase options Geodatabase options

CapacityNumber of users

ArcGIS Server licensingArcGIS Server licensing

•• Three editions of ArcGIS Server functionalityThree editions of ArcGIS Server functionality

•• Functionality scaled by capacityFunctionality scaled by capacity–– Workgroup or enterpriseWorkgroup or enterprise

•• ArcSDE technology bundled with all editionsArcSDE technology bundled with all editions

AdvancedAdvanced

StandardStandard

BasicBasic

Spatial Analysis& Editing

Data Management

Visualization& Mapping

WorkgroupEnterprise

Geodatabase properties and behaviorGeodatabase properties and behavior

•• Properties stored in a relational database (DBMS)Properties stored in a relational database (DBMS)–– Feature geometry and attributes stored in tablesFeature geometry and attributes stored in tables–– Rules and relationships stored in system tables Rules and relationships stored in system tables

•• Behavior implemented Behavior implemented by client by client ((ArcObjects)ArcObjects)–– ArcGIS software operates according to propertiesArcGIS software operates according to properties–– Geodatabase integrity enforced by clientGeodatabase integrity enforced by client

ArcObjects interprets

properties as

behavior

Introducing multitier architecture Introducing multitier architecture

•• Application tier manages geodatabase integrityApplication tier manages geodatabase integrity–– ArcObjectsArcObjects–– ArcSDE ArcSDE

•• Storage and retrieval tierStorage and retrieval tier–– DBMSDBMS

DBMSDBMS

ArcSDE ArcSDE

ArcObjectsArcObjects

Technology Application access

ArcSDE commands, API accessArcSDE commands, API access

SQLSQL

ArcGIS Desktop, Server, EngineArcGIS Desktop, Server, Engine

Multitier architecture: ArcObjectsMultitier architecture: ArcObjects

•• Building blocks of ArcGISBuilding blocks of ArcGIS

–– Libraries of software componentsLibraries of software components–– Developer kits available for Developer kits available for

Desktop, Server, and Engine Desktop, Server, and Engine

•• ArcObjects communicateArcObjects communicatewith ArcSDE technologywith ArcSDE technologyand the DBMSand the DBMS

–– Enforce geodatabase behaviorEnforce geodatabase behavior

Multitier architecture: ArcSDEMultitier architecture: ArcSDE

•• ArcSDE installation includes aArcSDE installation includes a repositoryrepository–– Stores ArcSDE settings and properties of geodatabase Stores ArcSDE settings and properties of geodatabase

elementselements

•• Data dictionary Data dictionary –– System tables owned System tables owned

by ArcSDE administratorby ArcSDE administrator•• SDE tablesSDE tables•• GDB tablesGDB tables

•• SQL code SQL code –– Called by application tierCalled by application tier–– Manages ArcSDE repositoryManages ArcSDE repository

and user tablesand user tables *Only SQL Server prefixes tables with SDE_

Multitier architecture: DBMSMultitier architecture: DBMS

•• Handles storageHandles storage–– ArcSDE repositoryArcSDE repository–– Tables comprising spatial and nonTables comprising spatial and non--spatial dataspatial data

•• Provides mechanism forProvides mechanism for–– SecuritySecurity–– Backup and recoveryBackup and recovery–– Concurrency and scalabilityConcurrency and scalability

•• SQL used to interact directly with DBMSSQL used to interact directly with DBMS

SELECT * FROM ADDRESSES

Agenda Agenda


•• Vector data storage architectureVector data storage architecture–– Client Server ModelClient Server Model–– Vector data loading workflowsVector data loading workflows–– Data Maintenance StrategiesData Maintenance Strategies


Geometry storageGeometry storage

•• SDEBINARYSDEBINARY–– Available for SQL Server, and Available for SQL Server, and

OracleOracle–– Only option currently availableOnly option currently available

for workgroupfor workgroup

•• ST_GEOMETRYST_GEOMETRY–– ESRI Spatial Type ESRI Spatial Type –– DB2, Informix, Oracle, and DB2, Informix, Oracle, and

PostgreSQLPostgreSQL

•• Oracle Spatial Oracle Spatial –– SDO_GEOMETRYSDO_GEOMETRY

•• SQL Server SpatialSQL Server Spatial–– GEOGRAPHYGEOGRAPHY–– GEOMETRYGEOMETRY

•• PostgreSQL PostGISPostgreSQL PostGIS–– PG_GEOMETRYPG_GEOMETRY

SDEBINARY feature classSDEBINARY feature class

•• Three tablesThree tables–– Attributes: Business (B) tableAttributes: Business (B) table–– Geometry: Feature (F) tableGeometry: Feature (F) table–– Spatial index entries: Spatial Spatial index entries: Spatial

index (S) tableindex (S) table

•• Joined on feature identifier valueJoined on feature identifier value•• F and S tables include layer identifier in nameF and S tables include layer identifier in name

–– Unique value assigned by repository table when feature class is Unique value assigned by repository table when feature class is createdcreated

RoadsRoadsRoads

S7S7S7

F7F7F7

7 8

12

FeatureID

Main Street

SP_FID GX GY …1 7 12 …1 8 12 …

FID … AREA LEN POINTS

1 … 0 926.45

SHAPE OBJECTID NAME …1 1 Main St. …

Layers tableLayers table•• Stores name and properties of each feature classStores name and properties of each feature class

7 … ROADS SHAPE 21233677 … 3500 …

SDE_LAYERSSDE_LAYERS

F7F7

S7S7

layer_id … table_name spatial_column eflags … gsize1 …

ROADSROADS

sp_fid gx gy eminx eminy emaxx emaxy

fid … eminx eminy emaxx emaxy … area len points

OBJECTID SHAPE ROAD_ID ROAD_TYPE NAME …

Spatial referencesSpatial references•• Feature classes can share same spatial referenceFeature classes can share same spatial reference

–– SRID column links data with spatial reference SRID column links data with spatial reference

SRID … FALSEX FALSEY XYUNITS … SRTEXT

1 … –87154275 –88756663 1600 … PROJCS["NAD_19…

XY domain and resolution Projection

SDE_SPATIAL_REFERENCESSDE_SPATIAL_REFERENCES

LAYER_ID … OWNER TABLE_NAME SPATIAL_COLUMN SRIDSDE_LAYERSSDE_LAYERS

1 … GISADMIN ROADS SHAPE 12 … GISADMIN PARCELS SHAPE 1

Table registryTable registry•• Row for every table in the geodatabaseRow for every table in the geodatabase

•• Registration_id assigned to versioning delta tablesRegistration_id assigned to versioning delta tables–– Example: The delta tables for Roads would be A43 and D43Example: The delta tables for Roads would be A43 and D43

Client server modelClient server model

•• All data accessed over a TCP/IP networkAll data accessed over a TCP/IP network•• All data retrieved through SQL queriesAll data retrieved through SQL queries

–– ArcSDE technology translates ArcSDE technology translates –– Spatial and attribute filters limit rows returnedSpatial and attribute filters limit rows returned

•• ArcSDE technology performs spatial filteringArcSDE technology performs spatial filtering

Request

Geodatabase client ArcSDE

SQL query

Database server

technology

A

Query filtersQuery filters•• Limit which rows are fetched from tableLimit which rows are fetched from table

•• An An index index can aid query filtercan aid query filter’’s effectivenesss effectiveness

•• Attribute filterAttribute filter

–– Select rows by value in a field Select rows by value in a field

–– Processed by DBMSProcessed by DBMS

•• Spatial filterSpatial filter–– Select features by geometrySelect features by geometry

–– Processed by DBMS and ArcSDEProcessed by DBMS and ArcSDE

WHERE CITY_NAME = 'Victoria'

Attribute indexesAttribute indexes

•• Organize table based on Organize table based on values of one or more values of one or more columnscolumns

•• CanCan provide faster access provide faster access –– Typically more efficient Typically more efficient

than searching entire tablethan searching entire table

–– Should have high degreeShould have high degreeof uniquenessof uniqueness

–– Can carry a cost when editingCan carry a cost when editing

•• Index must be updatedIndex must be updated

A

Query: where x = 4

<=2>2

<=4>4

<=6>6

Adding an attribute indexAdding an attribute index

•• Data owner can create index(es) on business tableData owner can create index(es) on business table

•• Must have exclusive access to geodataset or tableMust have exclusive access to geodataset or table

•• ToolsTools–– Feature class properties Feature class properties

–– Geoprocessing tool: Geoprocessing tool: Data Management > IndexesData Management > Indexes

–– ArcSDE command line: ArcSDE command line: sdetable sdetable ––o create_indexo create_index

–– Enterprise DBMS toolsEnterprise DBMS tools

Support for Unique and Ascending options varies by DBMS. Reference ArcGIS Help for more information.

Considerations for index creationConsiderations for index creation

•• Table is often queried by one or more columns Table is often queried by one or more columns

•• Column or columns are not frequently editedColumn or columns are not frequently edited–– Can drop and recreate index(es) for bulk editsCan drop and recreate index(es) for bulk edits

•• Multicolumn indexesMulticolumn indexes–– May be faster than individual field indexesMay be faster than individual field indexes

–– Build index on columns frequently queried togetherBuild index on columns frequently queried together

Spatial index queriesSpatial index queries

•• Clients use Clients use spatial filter spatial filter to reduce query resultsto reduce query results

•• ArcSDE server uses ArcSDE server uses spatial index spatial index to reduce I/Oto reduce I/O–– Eliminates need for full table scanEliminates need for full table scan

–– Same principle as DBMS column indexSame principle as DBMS column index

Spatial filter

Choosing a spatial grid index sizeChoosing a spatial grid index size

•• Use Calculate Default Spatial Grid Index (GP tool)Use Calculate Default Spatial Grid Index (GP tool)–– Reports recommended size based onReports recommended size based on

•• Spatial referenceSpatial reference•• Average feature sizeAverage feature size•• Number of featuresNumber of features

•• Each feature class will varyEach feature class will vary

•• Size shown is tile edge Size shown is tile edge in data coordinatesin data coordinates

Spatial index valuesSpatial index values

•• Convert existing dataConvert existing data–– GP environment setting usedGP environment setting used–– Else, default calculatedElse, default calculated

•• Create new feature classCreate new feature class–– Spatial grid is zero until data is loadedSpatial grid is zero until data is loaded

•• Dynamically generated for preview purposesDynamically generated for preview purposes

Altering spatial grid indexAltering spatial grid index

•• Must be feature class ownerMust be feature class owner•• ToolsTools

–– Feature class properties in ArcCatalogFeature class properties in ArcCatalog

–– ArcSDE command ArcSDE command sdelayer sdelayer ––o updateo update……--g <grid1,grid2..>g <grid1,grid2..>

Provide your own values

Uses default GP tool

After loading dataAfter loading data

•• Create DBMS statistics Create DBMS statistics –– AnalyzeAnalyze in ArcCatalogin ArcCatalog

•• Tools and options vary by DBMSTools and options vary by DBMS–– Critical to DBMS performanceCritical to DBMS performance–– Confer with DBA about frequencyConfer with DBA about frequency

•• Grant privilegesGrant privileges

•• EnsureEnsure metadatametadata is completeis complete

•• OptionallyOptionally–– Implement behaviorImplement behavior–– Build attribute indexesBuild attribute indexes–– Register as versionedRegister as versioned

Maintaining performance overviewMaintaining performance overview

•• Compress the geodatabase (versioned editing)Compress the geodatabase (versioned editing)–– Removes Removes unnecessaryunnecessary recordsrecords

•• Maintain statisticsMaintain statistics–– Database needs current statistics to efficiently return dataDatabase needs current statistics to efficiently return data

•• Monitor indexesMonitor indexes–– Fragmentation leads to increased I/OFragmentation leads to increased I/O

•• Ensure users employ best practicesEnsure users employ best practices–– Client actions affect performanceClient actions affect performance

Maintaining DBMS statisticsMaintaining DBMS statistics

•• ArcSDE does ArcSDE does notnot maintain statistics; DBA responsiblemaintain statistics; DBA responsible–– Describe data for database Describe data for database optimizeroptimizer–– Critical for maintaining performanceCritical for maintaining performance

•• Keep upKeep up--toto--datedate statisticsstatistics–– Depends on editing Depends on editing activityactivity–– BeforeBefore and and afterafter database compressdatabase compress–– It is better to have It is better to have ““overover--reportedreported”” (or none) then (or none) then ““underunder--

reportedreported”” statisticsstatistics

Which tables need DBMS statistics updated?Which tables need DBMS statistics updated?

•• Business tablesBusiness tables–– Fairly static (unless using move to base)Fairly static (unless using move to base)–– Therefore, typically do not need to update statistics frequentlyTherefore, typically do not need to update statistics frequently

•• AA-- and and DD--tables (can) require frequent updatestables (can) require frequent updates

•• FF-- and and SS--tables (can) require frequent updatestables (can) require frequent updates

•• STATE_LINEAGESSTATE_LINEAGES needs to have good statisticsneeds to have good statistics–– Update frequentlyUpdate frequently

•• Other versioned repository tables that are frequently queried:Other versioned repository tables that are frequently queried:–– StatesStates–– Mvtables_modifiedMvtables_modified

•• Raster tables are generally static Raster tables are generally static

Maintaining indexesMaintaining indexes

•• Insert/delete causes Insert/delete causes entries to be scatteredentries to be scattered–– Insertions introduce nonInsertions introduce non--contiguous entriescontiguous entries

–– Deletions cause skewed indexDeletions cause skewed index

•• Can lead to increased I/OCan lead to increased I/O

•• ArcSDE does ArcSDE does not not coalesce or rebuild after compresscoalesce or rebuild after compress–– Maintain index statisticsMaintain index statistics

•• For versioned databasesFor versioned databases–– Periodically rebuild delta table indexes Periodically rebuild delta table indexes

•• Use Use –– DBMS statementsDBMS statements

–– ArcSDE command lineArcSDE command line

Agenda Agenda




ArcSDE raster organizationArcSDE raster organization

•• Organize raster data for usageOrganize raster data for usage–– Raster datasetsRaster datasets

•• Can be grouped into Can be grouped into catalogscatalogs–– Feature class can have an Feature class can have an attributeattribute of type rasterof type raster

•• All loaded data is converted to All loaded data is converted to ArcSDEArcSDE raster formatraster format–– All data stored in vector and raster ArcSDE tablesAll data stored in vector and raster ArcSDE tables

3712a.bsq 3712b.bsq

3712c.bsq 3712d.bsq

WILSON.IMAGE_3712

Raster dataset schema overviewRaster dataset schema overview

•• At At least five tablesleast five tables::–– Business Business table table –– Pixel data Pixel data –– BLKBLK tabletable–– Deletion flag Deletion flag –– RASRAS tabletable–– Band metadata Band metadata –– BNDBND tabletable–– Statistics and color maps Statistics and color maps –– AUX AUX tabletable–– Raster attributes (optional) Raster attributes (optional) –– VATVAT tabletable–– Vector footprint:Vector footprint:

•• Spatial type Spatial type –– stored in business tablestored in business table•• ArcSDE Binary ArcSDE Binary –– stored in F and S tablesstored in F and S tables

Ortho

OBJECTID NAME SHAPE RASTER1 IMAGE24 1 1

S tableF table

Vector tables

Raster table (RAS)

Band table (BND)

Block table (BLK)

Auxiliary table (AUX)

Supporting raster tables

Attribute table (VAT)

ArcSDE Binary

Geodatabase rastersGeodatabase rasters––what makes them differentwhat makes them different

•• Raster catalogs and raster attributes exist only in the Raster catalogs and raster attributes exist only in the geodatabasegeodatabase

•• TilingTiling–– Data is tiled to reduce I/OData is tiled to reduce I/O

•• PyramidsPyramids–– Very large reduced resolution pyramids speed up displayVery large reduced resolution pyramids speed up display

•• CompressionCompression–– Data is decompressed on the clientData is decompressed on the client

•• NodataNodata–– nodata tiles are not stored, but manufactured on the client nodata tiles are not stored, but manufactured on the client

Geodatabase Raster StorageGeodatabase Raster Storage——Tiling Tiling •• The geodatabase The geodatabase subdividessubdivides raster data into tilesraster data into tiles

•• Allows for piecewise access to data for improved Allows for piecewise access to data for improved display performancedisplay performance

•• Default suggested tile size is 128 x 128Default suggested tile size is 128 x 1280 1 2

0

1

2

TilingTiling

Geodatabase Raster StorageGeodatabase Raster Storage——Pyramids Pyramids

•• Reduced resolution levels Reduced resolution levels of the sourceof the source

•• Greatly improves the Greatly improves the display performance display performance because:because:

–– Clients request data resolutionClients request data resolutionto match display resolutionto match display resolution

–– Number of pixels fetched as the Number of pixels fetched as the user zooms remains staticuser zooms remains static

•• Each level is tiled for Each level is tiled for storage in blocksstorage in blocks

2x2 pixelblock2x2 pixelblock

1 meter

1st (2 m)

2nd (4 m)

3rd (8 m)

Pyramid overheadPyramid overhead

•• Pixel dimensions double at each pyramid levelPixel dimensions double at each pyramid level–– Results in Results in ¼¼ as many pixelsas many pixels–– Tiled and stored in BLK tableTiled and stored in BLK table

•• Theoretical overhead is Theoretical overhead is 33%33%•• ArcSDE computes optimal number of pyramidsArcSDE computes optimal number of pyramids

–– Based on width and height of raster in pixelsBased on width and height of raster in pixels

Pyramid level

Storageoverhead

1 2 3

33%

25.00%

31.25%32.81%

14

+116

+1

64… = 1

3= 33%

Compression optionsCompression optionsLoss-less Lossy

Type: LZ77 JPEG, JPEG 2000

Best for: Data analyzed by computerData with sharp edges

Data analyzed visuallySmooth, continuous images

Applies to: All data types JPEG – 8 bit unsignedJP2 – 8 and 16 bit unsigned

Amount ofcompression:

ModerateHigher for morehomogenous data

HighUser controlscompression/quality ratio

Examples: Analytical surfacesAnalytical satellite/aerial imagesScanned linework andcolor-mapped images

Digital photos of assetsImagery for basemappingcartographic outputScanned true-color maps

Tips for loading multiple rastersTips for loading multiple rasters

•• Use a sample load with appropriate setting and propertiesUse a sample load with appropriate setting and properties•• Stage large data load over timeStage large data load over time•• Enterprise usersEnterprise users–– Critical to confer with DBACritical to confer with DBA

•• DBMS must be prepared for large data loadDBMS must be prepared for large data load–– Use direct connect from machine with fast processorUse direct connect from machine with fast processor•• AnalyzeAnalyze periodicallyperiodically–– Simultaneous selects and inserts into same table for pyramid Simultaneous selects and inserts into same table for pyramid

creationcreation–– Based on percentage change of newly added dataBased on percentage change of newly added data•• Generate raster statistics after load is complete, if applicableGenerate raster statistics after load is complete, if applicable–– Different from SQL Server statisticsDifferent from SQL Server statistics–– Used for client displayUsed for client display–– Manage through ArcGISManage through ArcGIS

Raster improvements at 9.2 and 9.3Raster improvements at 9.2 and 9.3

•• Compression algorithms Compression algorithms were threaded at 9.2.were threaded at 9.2.–– Both the encoding and decoding algorithms spawn threads to Both the encoding and decoding algorithms spawn threads to

compress and decompress raster datacompress and decompress raster data

•• Improved Improved bufferingbuffering and and streamingstreaming between ArcSDE between ArcSDE client and server at 9.2client and server at 9.2

–– Controlled by the RASTERBUFSIZEControlled by the RASTERBUFSIZE

•• PyramidPyramid construction makes more efficient use of construction makes more efficient use of memory and I/O at 9.2memory and I/O at 9.2

•• At 9.3, detect number of CPUs and start more writer At 9.3, detect number of CPUs and start more writer threadsthreads

–– Faster mosaics, faster pyramid construction Faster mosaics, faster pyramid construction

Conclusion Conclusion




Other FedUC Sessions and ActivitiesOther FedUC Sessions and Activities

•• Technical Sessions this afternoonTechnical Sessions this afternoon–– Managing Distributed Data with Geodatabase Replication Managing Distributed Data with Geodatabase Replication –– Multiuser Geodatabase Editing Workflows Multiuser Geodatabase Editing Workflows

•• HandsHands--On Learning Lab, ESRI Registration areaOn Learning Lab, ESRI Registration area–– Basics of Geodatabase ModelBasics of Geodatabase Model–– Workflow Management with JTXWorkflow Management with JTX–– Using the GIS Data ReViewerUsing the GIS Data ReViewer

Thank youThank youfor attending!for attending!

[email protected]@esri.com

data management in the multiuser geodatabase · pdf file• all loaded data is converted to...

Documents