2010 09-neo4j-deutsche-telekom
DESCRIPTION
In the Telecommunications sector, there are a lot of complex data sets and problems that are well suited for graph models and the use of graph databases like Neo4j. This talk gives just some ideas on where Neo4j currently is used within the TelCo sector. If you recognize problem areas that you have, don't hesitate to contact me at peter at neotechnology dot com, we are eager to learn more and help!TRANSCRIPT
NOSQL data models
Bigtable clones
Key-value stores
Document databases
Graph databases
Data complexity
Dat
a si
ze
The Neo4j model: Property GraphCore abstractions:
NodesRelationships between nodesProperties on both
name = “Emil”age = 29sex = “yes”
type = KNOWStime = 4 years
type = carvendor = “SAAB”model = “95 Aero”
11 22
33
The NoSQL landscapeNeo4j introExamples for TelCos
CDRRoutingSocial graphsMaster Data ManagementSpatial and LBSNetwork topology analysisNeo4j and Android
Agenda
Most widely deployed graph db in the worldACID, persistent, embedded/serverRobust: 24/7 production since 2003Mature: lots of production deploymentsScalable: High Availability, Master failoverCommunity: ecosystem of tools, bindings, frameworksProduct: OSGi, Spatial, RDF, languages
Available under AGPLv3 and as commercial productBut the fi rst one is free! For ALL use-cases.
Linkshttp://neo4j.org http://lists.neo4j.org
Neo4j?
Neo4j – Nodes, Relationships, Properties
Nodes have different propertiesMatrix characters: People vs. Programs
Build structure as you goWho loves Neo?
Building a node spaceGraphDatabaseService graphDb = ... // Get factory
// Create Thomas 'Neo' AndersonNode mrAnderson = graphDb.createNode();mrAnderson.setProperty( "name", "Thomas Anderson" );mrAnderson.setProperty( "age", 29 );
// Create MorpheusNode morpheus = graphDb.createNode();morpheus.setProperty( "name", "Morpheus" );morpheus.setProperty( "rank", "Captain" );morpheus.setProperty( "occupation", "Total bad ass" );
// Create a relationship representing that they know each othermrAnderson.createRelationshipTo( morpheus, RelTypes.KNOWS );// ...create Trinity, Cypher, Agent Smith, Architect similarly
Building a node spaceGraphDatabaseService graphDb = ... // Get factoryTransaction tx = graphdb.beginTx();
// Create Thomas 'Neo' AndersonNode mrAnderson = graphDb.createNode();mrAnderson.setProperty( "name", "Thomas Anderson" );mrAnderson.setProperty( "age", 29 );
// Create MorpheusNode morpheus = graphDb.createNode();morpheus.setProperty( "name", "Morpheus" );morpheus.setProperty( "rank", "Captain" );morpheus.setProperty( "occupation", "Total bad ass" );
// Create a relationship representing that they know each othermrAnderson.createRelationshipTo( morpheus, RelTypes.KNOWS );// ...create Trinity, Cypher, Agent Smith, Architect similarlytx.commit();
Code (2): Traversing a node space
// Instantiate a traverser that returns Mr Anderson's friendsTraverser friendsTraverser = mrAnderson.traverse(
Traverser.Order.BREADTH_FIRST,StopEvaluator.END_OF_GRAPH,ReturnableEvaluator.ALL_BUT_START_NODE,RelTypes.KNOWS,Direction.OUTGOING );
// Traverse the node space and print out the resultSystem.out.println( "Mr Anderson's friends:" );for ( Node friend : friendsTraverser ){
System.out.printf( "At depth %d => %s%n",friendsTraverser.currentPosition().getDepth(),friend.getProperty( "name" ) );
}
Rubygem install neo4j
require ”rubygems”require 'neo4j'
class Person include Neo4j::NodeMixin property :name, :age, :occupation index :name has_n :friendsend
Neo4j::Transactoin.run do neo = Person.new :name=>'Neo', :age=>29 morpheus = Person.new :name=>'Morpheus', :occupation=>'badass' neo.friends << morpheusend
neo.friends.each {|p|...}
Spatial and social data
name = “Omni Hotel”lat = 3492848long = 283823423
11
length = 7 miles
name = ...lat, long = ...
name = “Swedland”lat = 23410349long = 2342348852
33
1313
ROAD ROOOAD
name = “The Tavern”lat = 1295238237long = 234823492
length = 3 miles
name = ...
77
22
ROAD
ROADR
OAD
name = ...
4242
ROAD
Financial data – fraud detection
name = “Mr Godfather”karma = veeeery-lowcash = more-than-you
11
amount = $1000
name = “Emil”cash = always-too-li'l
title = “ATM @ Wall St”id = 230918484233cash_left = 384204
33
1313
TRANSFER WITHDRAW
name = “The Tavern”lat = 1295238237long = 234823492
amount = $1000
name = ...
77
22
OWNS
DEPOSITTR
ANSF
ER
name = ...
4242
WITHDRAW
CDR analysis
name = “Mr Godfather”karma = veeeery-lowcash = more-than-you
11
time = 2min
name = “Emil”cash = always-too-li'l
Name= customer_support
33
1313
CALLED CALLED
name = “Bob”lat = 1295238237long = 234823492
time =13min
name = Alice
77
22
CALLED
CALLEDFA
MIL
Y
name = flat_business
4242
DATA_P
LAN
Call Data Records (CDR)Forming a graphLocation basedPossible uses:
Find clusters (better plans)Build social connectionsFind influencers
Routing
Routing with Neo4j and A*
Routing
Connecting and Splitting
Routing
Social graphs
RecommendationsLocation based servicesInfluencersShortest path
Recommendations and big graphsGlobal heuristics
Page rankLocal recommendations
Shortest pathsHammock functionsRandom walksDijkstra, A*, Shooting star etc
Grammar based Random Walks
Pic from gremlin.tinkerpop.com / Marko Rodriguez
Impact, Dependency Analysis
Master Data Management
SpatialComplex data
Multiple indexing (domain, Spatial, temporal)Location entering many domainsGIS going mainstream, topologies explodeNo good systems out thereProprietary stacks rule (ESRI, Oracle)Open Government DataShapefiles suck.
Current challenges in SpatialDomain and Spatial interconnectionsUnstructured domain dataRoutingTopology handlingNo good OSS full GIS stack
Multiple indexes - GIS
OpenStreetMap
The OpenStreetMap datasetWiki for Spatial infoFreely available dataVery unstructured, free tagging
Points, Ways, Relations, Tags, ChangesetsChanges can be pushed back
Used for other purposesGreat coverage in interesting places (towns, disasters etc)
OpenStreetMap
OpenStreetMap
Web exposure of Spatial Data
Davide SavazziGeotools & GeoServerRoutinguDig
GSoC 2010 - uDig
Neo4j dynamic layers
Layer1
Layer2
Layer3
GeometryEncoder
DynamicQuery
DynamicStyles
DynamicMeta-Inf
Connected domain data Neo4j Spatial GIS and Spatial stacks
OpenStreetMap
OSM
Index (RTree)
DynamicLayers
Dynamic Layers
Network Topology analysis
Analytics of network coverage and frequenciesCell towersDrive dataInfrastructure
AnalyticsSpatial signal strengthAntenna placement and azimuthFrequency planningNetwork differences over timeReporting and charting
Small footprintRunning almost unmodifiedInteresting scenarios
Semantic homescreenConnected devicesLocal cachesDisconnected databases
Neo4j and Android
Connected devices
Emils PC Marcus
Emil
LeighJohan
“Link all pictures that any of my friends has taken of me into my PicturesOfMe folder and update this every hour”
Interesting scenariosSemantic homescreenConnected devicesLocal cachesDisconnected databases
How ego are you? (aka other impls?)Franz’ AllegroGraph (http://agraph.franz.com)
Proprietary, Lisp, RDF-oriented but real graphdb
Sones graphDB (http://sones.com)
.NET, Deutsche Telekom VC backed
Twitter's FlockDB (http://github.com/twitter/flockdb)
Twitter's (graph) database for large and shallow graphs
Google Pregel (http://bit.ly/dP9IP)
We are oh-so-secret
Objectivity's InfniteGraph (http://infinitegraph.com)
New, closed OODB with Graph Layer on top
API ReferencesWiki, Code, API references
http://wiki.neo4j.org/content/Neo4j_Spatialhttp://github.com/neo4j/neo4j-spatialhttp://components.neo4j.org/neo4j-spatialMailing list: [email protected]://neo4j.org/community/list/
Questions?
Image credit: lost again! Sorry :(
http://neotechnology.com