siva ravada - nloug · •gps locations for each train collected throughout the day •each...
TRANSCRIPT
Siva Ravada Director of Development
Oracle Spatial Technologies
<Insert Picture Here>
Agenda
• Spatial on Exadata
• Introduction to Exadata
• Spatial operations on Exadata
• Customer Examples
• 3D Data Support
• OBIEE and MapViewer
• How maps are integrated into OBIEE
• Oracle Spatial Technology for Cloud
• Q & A
What Is the Oracle Exadata Database
Machine?
• Oracle SUN hardware uniquely engineered to work together with Oracle database software
• Key features: • Database Grid – Up to 128 Intel cores connected by 40 Gb/second
InfiniBand fabric, for massive parallel query processing.
• Raw Disk – Up to 336 TB of uncompressed storage (high performance or high capacity)
• Memory – Up to 2 TB
• Exadata Hybrid Columnar Compression (EHCC) – Query and archive modes available. 10x-30x compression.
• Storage Servers – Up to 14 storage servers (168 Intel cores) that can perform massive parallel smart scans. Smart scans offloads SQL predicate filtering to the raw data blocks. Results in much less data transferred, and dramatically improved performance.
• Storage flash cache – Up to 5.3 TB with I/O resource management
4
12 Xeon cores
2.93 GHz
12 Xeon cores
2.93 GHz
2 TB storage
12 Xeon cores, 2.26 GHz
2 TB storage
12 Xeon cores, 2.26 GHz
2 TB storage
12 Xeon cores, 2.26 GHz
RAC, OLAP
Partitioning
Compression
Exadata DB Machine X2-2 Quarter Rack
Exadata X2-2 Quarter Rack Diagram
Oracle Spatial Focused on Parallelizing
Spatial Computations
• US Rail • Bulk nearest neighbor queries to find closest track,
and project reported train positions onto tracks
• Validate home appraisals for a Government Sponsored Enterprise (GSE) • Find all the parcels touching parcels to validate
appraisals
• Satellite Imagery Provider • Find all the useful portions of cloud covered
imagery • 850,000 strip images
• 58,000,000 cloud cover geometries
Spatial on Exadata: Overview
• Parallelizing spatial queries against partitioned tables
• Parallelizing massive spatial computations with Create Table As
Select (CTAS)
• Parallelizing spatial operators
• SDO_JOIN and parallelizing spatial functions
Parallelizing Spatial Queries Against
Partitioned Tables
Parallel Spatial Query Against
Partitioned Tables
• Partition pruning occurs first
• If a spatial operator’s query window spans multiple
partitions, partitions are spatially searched in
parallel
• True for all spatial operators
Parallel Spatial Query Against
Partitioned Tables – Example
• Example:
• A re-insurance company maintains portfolios for
hundreds of insurance companies
• Which companies will be (or are) affected by the
projected path of a hurricane
• 36 million rows, 64 partitions, each with about 571,000
rows
• 50 seconds serial
• 1.28 seconds parallel (on a ½ rack Exadata V1
database machine)….. 39 times faster
Parallelizing Spatial Operators
US Rail Application
• Requirement • GPS locations for each train collected throughout the day
• Each location has other attributes (time, speed, and more)
• GPS locations have a degree of error, so they don’t always fall on a track
• Bulk nearest neighbor queries to find closest track, and project reported train positions onto tracks
• This information is used for: • Tracking trains
• Analysis for maintenance, ensure engineers are within parameters, etc…
Parallel and CTAS With Spatial Operators
• Spatial operators can parallelize with CTAS
when multiple candidates feed the second argument
• For example, a GPS records 45,158,800 train positions
For each train position:
• Find the closest track to the train (with SDO_NN)
• Then calculate the position on the track closest to the train
• See SQL on next slide…
Linear Scalabiltiy to process 45,158,800
Train Positions (One SQL statement)
ALTER SESSION FORCE PARALLEL ddl PARALLEL 72;
ALTER SESSION FORCE PARALLEL query PARALLEL 72;
CREATE TABLE results NOLOGGING PARALLEL
AS SELECT /*+ ordered index (b tracks_spatial_idx) */
a.locomotive_id,
sdo_lrs.find_measure (b.track_geom, a.locomotive_pos) measure
FROM locomotives a,
tracks b
WHERE sdo_nn (b.track_geom, a.locomotive_pos, 'sdo_num_res=1') = 'TRUE';
• X2-2 Half Rack, 34.75 hours serially vs. 41.1 minutes parallel
• X2-2 Half Rack (48 cores) over 47x faster … Linear scalability
Summary of Rail Application Results
(Performance and Compression)
• Results compare serial and parallel “Exadata” runs
• Serial runs on other hardware are slower than Exadata
• On Exadata:
• 34.75 hours serially vs. 41.1 minutes in parallel
• X2-2 Half Rack (48 database cores) - 47x faster
• X2-2 Full Rack (96 database cores) about 94x faster
• EHCC (Exadata Hybrid Columnar Compression)
• Can compress point geometries
• 12.03 GB vs 1.125 GB EHCC Archive High Compression
• 10.69X compression
The SDO_JOIN Operation
Parallelizing Spatial Functions
• Validate home appraisals for a Government
Sponsored Enterprise (GSE)
• Find all the parcels touching parcels to validate
appraisals
• Satellite Imagery Provider
• Find all the useful portions of cloud covered
imagery
• 850,000 strip images
• 58,000,000 cloud cover geometries
SDO_JOIN – Spatial Cross Product
SELECT /*+ ordered */ b.risk_zone_id,
c.parcel_id
FROM TABLE (SDO_JOIN ('RISK_ZONES', 'GEOM',
' PARCELS', 'GEOM',
'mask=anyinteract')) a,
risk_zones b,
parcels c
WHERE a.rowid1 = b.rowid
AND a.rowid2 = c.rowid;
• Effective way to compare all geometries in one layer
to all geometries in another (or most to most)
• Leverages spatial index for both spatial layers
• Can be orders of magnitude faster
SDO_JOIN – Strategy For Parallelization
CREATE TABLE result1 NOLOGGING AS
SELECT a.rowid1 AS risk_zones_rowid,
a.rowid2 AS parcels_rowid
FROM TABLE ( SDO_JOIN ('RISK_ZONES', 'GEOM',
' PARCELS', 'GEOM') );
• First SDO_JOIN, primary filter only
SDO_JOIN – Strategy For Parallelization
ALTER SESSION FORCE PARALLEL ddl PARALLEL 72;
ALTER SESSION FORCE PARALLEL query PARALLEL 72;
CREATE TABLE result2 NOLOGGING PARALLEL AS
SELECT /*+ ordered use_nl (a,b) use_nl (a,c) */
sdo_geom.relate (b.geom, 'DETERMINE', c.geom, .05) relation,
b.risk_zone_id, c.parcel_id
FROM result1 a,
risk_zones b, parcels c
WHERE a.risk_zones_rowid = b.rowid
AND a.parcels_rowid = c.rowid;
• Then parallelize spatial function
• For this example, call sdo_geom.relate on each pair.
Strategy – First SDO_JOIN, then
SDO_GEOM.RELATE in Parallel
•SDO_JOIN –
• Very effectively utilizes the spatial index of high risk zones
and parcel polygons
• Returns rowid pairs that likely intersect
•SDO_GEOM.RELATE with DETERMINE mask, in
parallel:
• Given two polygons, returns their
reationship
• Intersection is expensive
Strategy – First SDO_JOIN, then
SDO_GEOM.RELATE in Parallel
• Validate home appraisals for a Government Sponsored
Enterprise (GSE)
• Find all the parcels touching parcels to validate appraisals
• Processed 2,018,429 parcels serially in 38.25 minutes
• Exadata ½ RAC (48 cores) – 45x faster - 50 seconds
• X2-2 Full RAC (96 cores) about 90x faster
• X2-8 (128 cores) even faster
• Satellite imagery requirement from previous slide
• 45 days to 4 days (non-Exadata with hardware constraints)
• From 4 days to hours or minutes on Exadata
<Insert Picture Here>
Oracle Spatial
3D Data Model
Oracle Confidential
Simulation & Virtual Worlds
3D Urban Modeling Energy Exploration
Architecture & Engineering Simulation, Gaming, VR
3D Functionality in Oracle Spatial 11g
• SDO_GEOMETRY (3D)
• SDO_TIN
• SDO_POINT_CLOUD
3D
CO
OR
DIN
AT
E S
YS
TE
MS
Types Building Models,..
Surface
Modeling
Scene,
Object Modeling
Efficient
Storage
Query
Analysis
Data Types SDO_GEOMETRY
• Points
• Lines
• Simple Surfaces • All points of a surface lie in a 3D plane
• Composite surfaces • Has one or more connected simple surfaces
• Simple surfaces cannot cross
• Simple Solids
• Composed of closed surfaces
• Can have interior surfaces
• Composite Solids
• Consists of n connected simple solids
• Representation might be easier
• Multi-points, -lines, -surfaces, -solids
• Multi-solid is different from composite solid
Data Types
• SDO_PC
• Point clouds
• Used to capture as built
• SDO_TIN
• Used to model terrain
City GML and Visualization Models
• Schemas for modeling 3D data for different
information models
• City GML
• Building Information Model (BIM)
• Utilities to publish as CityGML
• Provide a standard Visualization Schema
Themes
• Combine
• Geometry
• Texture
• Materials,
color maps, …
• Attributes
• Declare schema in metadata
Sample Theme Geometry
create table TEXTURES (
id number,
texture blob,
CONSTRAINT textures_pk PRIMARY KEY(id));
create table road_tiles_zoom4 (
id number,
geom sdo_geometry,
texture_id number,
texture_coords sdo_ordinate_array,
gen_id number,
CONSTRAINT road_tiles4_pk PRIMARY KEY(id),
CONSTRAINT ROAD_TILES4_FK FOREIGN KEY(texture_id) REFERENCES TEXTURES(id)
deferrable initially deferred,
CONSTRAINT ROAD_TILES4_GEN_FK FOREIGN KEY(gen_id) REFERENCES
ROAD_TILES_ZOOM3(id)
deferrable initially deferred);
Sample Theme Texture
create table TEXTURES (
id number,
texture blob,
CONSTRAINT textures_pk PRIMARY KEY(id));
create table road_tiles_zoom4 (
id number,
geom sdo_geometry,
texture_id number,
texture_coords sdo_ordinate_array,
gen_id number,
CONSTRAINT road_tiles4_pk PRIMARY KEY(id),
CONSTRAINT ROAD_TILES4_FK FOREIGN KEY(texture_id) REFERENCES TEXTURES(id)
deferrable initially deferred,
CONSTRAINT ROAD_TILES4_GEN_FK FOREIGN KEY(gen_id) REFERENCES
ROAD_TILES_ZOOM3(id)
deferrable initially deferred);
Sample Theme Generalization
create table TEXTURES (
id number,
texture blob,
CONSTRAINT textures_pk PRIMARY KEY(id));
create table road_tiles_zoom4 (
id number,
geom sdo_geometry,
texture_id number,
texture_coords sdo_ordinate_array,
gen_id number,
CONSTRAINT road_tiles4_pk PRIMARY KEY(id),
CONSTRAINT ROAD_TILES4_FK FOREIGN KEY(texture_id) REFERENCES TEXTURES(id)
deferrable initially deferred,
CONSTRAINT ROAD_TILES4_GEN_FK FOREIGN KEY(gen_id) REFERENCES
ROAD_TILES_ZOOM3(id)
deferrable initially deferred);
Sample Theme Attributes (BIM, CityGML, …)
create table TEXTURES (
id number,
texture blob,
CONSTRAINT textures_pk PRIMARY KEY(id));
create table road_tiles_zoom4 (
id number,
geom sdo_geometry,
texture_id number,
texture_coords sdo_ordinate_array,
gen_id number,
… Attribute columns: name, address, population, etc …
CONSTRAINT road_tiles4_pk PRIMARY KEY(id),
CONSTRAINT ROAD_TILES4_FK FOREIGN KEY(texture_id) REFERENCES TEXTURES(id)
deferrable initially deferred,
CONSTRAINT ROAD_TILES4_GEN_FK FOREIGN KEY(gen_id) REFERENCES
ROAD_TILES_ZOOM3(id)
deferrable initially deferred);
Metadata (Theme) insert into USER_SDO_3DTHEMES (
name,
description,
base_table,
theme_column,
style_column,
theme_type,
definition)
values (
'Road Tile 4 Theme',
'Road Tile 4 Theme',
'ROAD_TILES_ZOOM4',
'GEOM',
' ',
'SDO_GEOMETRY',
'<Theme3d srid="4327">
<LOD>
<ThemeLOD>4</ThemeLOD>
<Generalization>
<ForeignKey>ROAD_TILES4_GEN_FK</ForeignKey>
<TargetTheme>Road Tile 3 Theme</TargetTheme>
<ActivationDistance uom_id="9001">3200000</ActivationDistance>
</Generalization>
</LOD>
<DefaultStyle>
<Texture>
<TextureBLOB>
<ForeignKey>ROAD_TILES4_FK</ForeignKey>
<ValueColumn>TEXTURE</ValueColumn>
</TextureBLOB>
<TextureCoordinates>
<ValueColumn>TEXTURE_COORDS</ValueColumn>
</TextureCoordinates>
</Texture>
</DefaultStyle>
<hidden_info>
<field column="name" name="Name"/>
<field column="address" name="Address"/>
</hidden_info>
</Theme3d>');
create table TEXTURES (
id number,
texture blob,
CONSTRAINT textures_pk PRIMARY KEY(id));
create table road_tiles_zoom4 (
id number,
geom sdo_geometry,
texture_id number,
texture_coords sdo_ordinate_array,
gen_id number,
CONSTRAINT road_tiles4_pk PRIMARY KEY(id),
CONSTRAINT ROAD_TILES4_FK FOREIGN KEY(texture_id) REFERENCES TEXTURES(id)
deferrable initially deferred,
CONSTRAINT ROAD_TILES4_GEN_FK FOREIGN KEY(gen_id) REFERENCES ROAD_TILES_ZOOM3(id)
deferrable initially deferred);
Metadata (Scene)
insert into USER_SDO_SCENES (
name,
description,
definition)
values (
'Road Tile 3 Scene',
'Road Tile 3 Scene',
'<scene srid="4327">
<theme display="true" pickable="false">Road Tile 2 Theme</theme>
<theme display="true" pickable="false">Berlin Theme LOD1</theme>
<theme display="true" pickable="false">Ground Theme</theme>
<theme display="false" pickable="true">Pickable Features Theme</theme>
</scene>');
Metadata (Viewframe)
insert into user_sdo_viewframes (
name,
description,
scene_name,
definition)
values (
'Sat Tile 3 Viewpoint',
'Sat Tile 3 Viewpoint',
'Sat Tile 3 Scene',
'<ViewFrame3d>
<SceneName>Sat Tile 3 Scene</SceneName>
<ViewPoint3d srid="4327">
<Eye x="-75" y="45" z="12000000"/>
<Center x="0" y="0" z="-6378137"/>
<Up x="0" y="1" z="0"/>
</ViewPoint3d>
<DefaultStyle>String</DefaultStyle>
</ViewFrame3d>');
OBIEE and MapViewer
<Insert Picture Here>
Oracle Business
Intelligence Enterprise
Edition 11g
Part of Oracle Fusion Middleware
Infrastructure & Management
Database
Middleware
Applications
When are Map views useful
• Visualizing data related to geographic locations
• Showing or detecting spatial relationships and
patterns
• Showing lots of data in a relatively small area
• Drilling down from a (map) overview to a detailed
report, chart, or graph
Why Spatial Map Visualizations? Ideal For Master-Detail Analysis
© 2010, NAVTEQ
Spatial Visualization: Map
views in OBIEE 11g
Oracle BI Enterprise Edition &
Location Integration Map views: Thematic Map-based Visualizations
• Thematic Mapping in OBIEE 11g
• First-class view in Answers Point-and-click interface
Creation of Map Views
Interactions are UI-driven
State management
Dashboard Prompts
Print & Export
Agents
Integration with
Presentation Server stack
Oracle Fusion Middleware
MapViewer
Support for thematic overlays
Master-detail linking
Map Metadata Management
Via Presentation Server Administration
Creating Map views
• (At Least) One Column related to location (e.g.
country code, state name)
• (At Least) One Measure
• Default map format created
• Provided…
• Edit Map View to add / edit / delete formats
Map View Formats
• Color Fill (choropleth)
• Percentile, Value,
Continuous binning
• Dashboard user run-time
slider
• Graphs – Bar, Pie
• Adjustable graph size
• Series by second dimension
• Bubble (variable sized)
• Min-Max size specification
• Color specification
• Variable Shape
• Circle, Triangle, Diamond
• Customizable
• Image
• Imported via MapViewer
• More can be added from
MapBuilder
• Custom Point Layer
• Uses Lat / Long
• Does not require a Layer
Def
OBIEE 11g Map view feature
• New view type in 11g
• Supports common use cases
• And interactions: drilling, master-detail linking
Master-Detail linking: Country at a glance on
the map, details for a state in charts
A geographic dimension usually has a
well known hierarchy, e.g. country,
region, state, county
Spatial Technology for the Cloud
• Spatial components are built to run on premise and in the cloud
• Applications built on these components only need to change the URL to switch from on premise to cloud implementations
• Examples • J2EE Geocoder
• Routing Engine
• MapViewer
• Web services (WLS, CS-W, WMS)
• Network Data Model
• More components being built in this web services model
• 3D data server
• City GML server
Find out more...
oracle.com/database/spatial.html
oracle.com/technology/products/spatial
oracle.com/technology/products/spatial/htdocs/pro_oracle_spatial.html
A Q &