rtree spatial indexing with mongodb - mongodc

WHY WE CHOSE MONGODB TO PUT BIG-DATA ‘ON THE MAP’

JUNE 2012

@nknize+Nicholas Knize

“The 3D UDOP allows near real time visibility of all SOUTHCOM Directorates information in one location…this capability allows for unprecedented situational awareness and information sharing”

-Gen. Doug Frasier

TST PRODUCTS

ACCOMPLISHING THE IMPOSSIBLE

• Expose enterprise data in a geo-temporal user defined environment

• Provide a flexible and scalable spatial indexing framework for heterogeneous data

• Visualize spatially referenced data on 3D globe & 2D maps• Manage real-time data feeds and mobile messaging • View data over geo-rectified imagery with 3D terrain• Support mission planning and simulation• Provide real-time collaboration and sharing

ISPATIAL OVERVIEW


• Horizontally scalable – Large volume / elastic

• Vertically scalable – Heterogeneous data types (“Data Stack”)

• Smartly Distributed – Reduce the distance bits must travel

• Fault Tolerant – Replication Strategy and Consistency model

• High Availability – Node recovery

• Fast – Reads or writes (can’t always have both)BIG DATA STORAGE CHARACTERISTICS


Desired Data Store Characteristic for ‘Big Data’

• Cassandra– Nice Bring Your Own Index (BYOI) design– … but Java, Java, Java… Memory management can be an issue– Adding new nodes can be a pain (Token Changes, nodetool)– Key-Value store…good for simple data models

• Hbase– Nice BigTable model– Theory grounded heavily in C.A.P, inflexible trade-offs– Complicated setup and maintenance

• CouchDB– Provides some GeoSpatial functionality (Currently being rewritten)– HEAVILY dependent on Map-Reduce model (complicated design)– Erlang based – poor multi-threaded heap management

NOSQL OPTIONS


Subset of Evaluated NoSQL Options

Why MongoDB for Thermopylae?• Documents based on JSON – A GEOJSON match made in heaven!

• C++ - No Garbage Collection Overhead! Efficient memory management design reduces disk swapping and paging

• Disk storage is memory mapped, enabling fast swapping when necessary

• Built in auto-failover with replica sets and fast recovery with journaling

• Tunable Consistency – Consistency defined at application layer

• Schema Flexible – friendly properties of SQL enable easy port

• Provided initial spatial indexing support – Point based limited!WHY TST LIKES MONGODB


MONGODB SPATIAL INDEXER


... The Spatial Indexer wasn’t quite right

• MongoDB (like nearly all relational DBs) uses a b-Tree – Data structure for storing sorted data in log time– Great for indexing numerical and text documents (1D attribute data)– Cannot store multi-dimension (>2D) data – NOT COMPLEX GEOMETRY

FRIENDLY

DIMENSIONALITY REDUCTION


How does MongoDB solve the dimensionality problem?

• Space Filling (Z) Curve – A continuous line that

intersects every point in a two-dimensional plane

• Use Geohash to represent lat/lon values– Interleave the bits of a

lat/long pair– Base32 encode the result

GEOHASH BTREE ISSUES


• Neighbors aren’t so close!– Neighboring points on the

Geoid may end up on opposite ends of the plane

– Impacts search efficiency

• What about Geometry?– Doesn’t support > 2D– Mongo uses Multi-

Location documents which really just indexes multiple points that link back to a single document

Issues with the Geohash b-Tree approach

Case 3:

Case 4:

Multi-Location Document (aka. Polygon) Search Polygon

Case 1:

Case 2:

Success!

Success!

Fail!

Fail!

Mongo Multi-location Document Clipping Issues($within search doesn’t always work w/ multi-location)

MULTI-LOCATION CLIPPING


• Constrain the system to single point searches– Multi-dimension support will be exponentially complex (won’t scale)

• Interpolate points along the edge of the shape– Multi-dimension support will be exponentially complex (won’t scale)

• Customize the spatial indexer– Selected approach

SOLUTIONS TO GEOHASH PROBLEM


Potential Solutions

CUSTOM TUNED SPATIAL INDEXER


Thermopylae Custom Tuned MongoDB for Geo

TST Leverage’s Guttman’s 1984 Research in R/R* Trees• R-Trees organize any-dimensional data by representing

the data as a minimum bounding box. • Each node bounds it’s children. A node can have many

objects in it (max: m min: ceil(m/2) )• Splits and merges optimized by minimizing overlaps• The leaves point to the actual objects (stored on disk

probably)• Height balanced – search is always O(log n)

Spatial Indexing at Scale with R-Trees

RTREE THEORY


Spatial data represented as minimum bounding rectangles (2-dimension), cubes (3-dimension), hexadecant (4-dimension)

Index represented as: <I, DiskLoc> where:

I = (I0, I1, … In) : n = number of dimensionsEach I is a set in the form of [min,max] describing MBR range along a

dimension

R*-Tree Spatial Index Example• Sample insertion result for 4th order

tree• Objectives:

1. Minimize area2. Minimize overlaps3. Minimize margins4. Maximize inner node utilization

a b cd e f g h i j k l

m n o p

R*-TREE INDEX OBJECTIVES


Insert

• Similar to insertion into B+-tree but may insert into any leaf; leaf splits in case capacity exceeded.– Which leaf to insert into?– How to split a node?

R*-TREE INSERT EXAMPLE


Insert—Leaf Selection

• Follow a path from root to leaf.• At each node move into subtree whose MBR area

increases least with addition of new rectangle.

mn

o p

Modified from Dr. Sahni (UoF Advanced Data Structures)


• Insert into m.

m



• Insert into n.

n



• Insert into o.

o



• Insert into p.

p


mn

o p

aa

a

x


m n o p

Query• Start at root• Find all overlapping MBRs• Search subtrees recursively


Query

• Search m.

mn

o p

a

a

x x


m n o p

a

aa

b

cd

e

g


R*-Tree Leverages B-Tree Base Data Structures (buckets)

R*-TREE MONGODB IMPLEMENTATION


Geo-Sharding – (in work)Scalable Distributed R* Tree (SD-r*Tree)

“Balanced” binary tree, with nodes distributed on a set of servers:

• Each internal node has exactly two children

• Each leaf node stores a subset of the indexed dataset

• At each node, the height of the subtrees differ by at most one

• mongos “routing” node maintains binary tree

GEO-SHARDING


d0 d1

r1d0Data Node Spatial

Coverage

a a

b

c

cb d0

r1

a

b

c

c

b

d2d1

ed

d

r2

e

SD-r*Tree Data Structure Illustration

• di = Data Node (Chunk)• ri = Coverage Node

Leveraged work from Litwin, Mouza, Rigaux 2007

SD-r*Tree DATA STRUCTURE


SD-r*Tree Structure Distribution

d0

r1

a

b

c

c

b

d2d1

ed

d

r2

e

r2

d1 d2

d0

r1

GeoShard 2 GeoShard 3

GeoShard 1

mongos

SD-r*TREE STRUCTURE DISTRIBUTION


GeoSharding Alternative – 3D / 4D Hilbert Scanning Order

GEO-SHARDING ALTERNATIVE


Next Steps: Beyond 4-Dimensions - X-Tree(Berchtold, Keim, Kriegel – 1996)

Normal Internal Nodes Supernodes Data Nodes

• Avoid MBR overlaps

• Avoid node splits (main cause for high overlap)

• Introduce new node structure: Supernodes – Large Directory nodes of variable size

BEYOND 4-DIMENSIONS


X-TREE PERFORMANCE


X-Tree Performance Results(Berchtold, Keim, Kriegel – 1996)

T-Sciences Custom Tuned Spatial Indexer

• Optimized Spatial Search – Finds intersecting MBR and recurses into those nodes

• Optimized Spatial Inserts – Uses the Hilbert Value of MBR centroid to guide search – 28% reduction in number of nodes touched

• Optimize Deletes – Leverages R* split/merge approach for rebalancing tree when nodes become over/under-full

• Low maintenance – Leverages MongoDB’s automatic data compaction and partitioning

CONCLUSION


Example Use Case – OSINT (Foursquare Data)

• Sample Foursquare data set mashed with Government Intel Data (poly reports)

• 100 million Geo Document test (3D points and polys)

• 4 server replica set

• ~350ms query response

• ~300% improvement over PostGIS

EXAMPLE


Community Support

• Thermopylae contributes fixes to the codebase– http://github.com/mongodb

• TST will work with 10gen to fold into the baseline

• Active developer collaboration– IRC: #mongodb freenode.net

FIND US


http://github.com/mongodb

THANK YOUQuestions?

Nicholas [email protected]

THANK YOU


Backup

Thermopylae Sciences & Technology – Who are we?

• Advanced technology w/ 160+ employees• Core customers in national security, venues and

events, military and police, and city planning• Partnered with Google and imagery providers• Long term relationship focused – TS/SCI Staff TST + 10gen + Google = Game-changing approach

WHO ARE THESE GUYS?


ENTERPRISEPARTNER

Key Customers - Government• US Dept of State Bureau of Diplomatic Security

– Build and support 30 TB Google Earth Globe with multi-terabytes of individual globes sent to embassies throughout the world. Integrated Google Earth and iSpatial framework.

• US Army Intelligence Security Command– Provide expertise in managing technology integration – prime

contractor providing operations, intelligence, and IT support worldwide. Partners include IBM, Lockheed Martin, Google, MIT, Carnegie Mellon. Integrated Google Earth and iSpatial framework.

• US Southern Command– Coordinate Intelligence management systems spatial data collection,

indexing, and distribution. Integrated Google Earth, iSpatial, and iHarvest.

– Index large volume imagery and expose it for different services (Air Force, Navy, Army, Marines, Coast Guard)

GOVERNMENT CUSTOMERS


COMMERCIAL CUSTOMERS


Key Customers - Commercial

ClevelandCavaliers

USGIF Las VegasMotor Speedway

BaltimoreGrand Prix

iSpatial framework serves thousands of mobile devices

rtree spatial indexing with mongodb - mongodc

Technology

sorted data

referenced data

representingthe data

dimensional data

enterprise data

d attribute data

rtreesspatial data

realtime data feeds