polyglot persistence with mongodb and neo4j
TRANSCRIPT
NoSQL Polyglot PersistenceUsing Neo4j and MongoDB together
William Lyon @lyonwj
Does Neo4j work with ______ ?
Languages, frameworks, databases…
Does Neo4j work with ______ ?
Languages
Does Neo4j work with ______ ?
Frameworks
Does Neo4j work with ______ ?
Frameworks
Does Neo4j work with ______ ?
Deployment
Does Neo4j work with ______ ?
Databases
Does Neo4j work with ______ ?
Languages, frameworks, databases…
Does Neo4j work with ?
Why?+
MongoDB
Features
• NoSQL database• Document datamodel• JSON-like documents (BSON)• Flexible data model• Horizontal scalability (sharding)• Complex queries
{ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics": [ "keynote", "spring" ], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}
MongoDB
Use Cases
• Product catalog• User profiles• Metadata• Content• Events• Analytics
{ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics": [ "keynote", "spring" ], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}
https://www.mongodb.com/use-cases/
MongoDB
Use Cases
• Product catalog• User profiles• Metadata• Content• Events• Analytics
{ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics": [ "keynote", "spring" ], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}
https://www.mongodb.com/use-cases/
Neo4j
Graph Database
• Property graph data model• Nodes and relationships
• Native graph processing• Cypher query language
Property Graph Model Components
Nodes • The objects in the graph • Can have name-value properties • Can be labeled
Relationships • Relate nodes by type and
direction • Can have name-value properties
CAR
DRIVES
name: “Dan” born: May 29, 1970
twitter: “@dan”name: “Ann”
born: Dec 5, 1975
since: Jan 10, 2011
brand: “Volvo” model: “V70”
LOVES
LOVES
LIVES WITH
OWNS
PERSON PERSON
Cypher: SQL for graphs
CREATE (:Person { name:“Dan”} ) -[:LOVES]-> (:Person { name:“Ann”} )
LOVES
Dan Ann
LABEL PROPERTY
NODE NODE
LABEL PROPERTY
NEO4J USE CASES
Real Time Recommendations
Master Data Management
Fraud Detection
Identity & Access Management
Graph Based Search
Network & IT-Operations
NEO4J USE CASES
Real Time Recommendations
Master Data Management
Fraud Detection
Identity & Access Management
Graph Based Search
Network & IT-Operations
GRAPH THINKING: Real Time Recommendations
VIEW
ED
VIEWED
BOUG
HT
VIEWED BOUGHT
BOUGHT
BO
UG
HT
BOUG
HT
NEO4J USE CASES
Real Time Recommendations
Master Data Management
Fraud Detection
Identity & Access Management
Graph Based Search
Network & IT-Operations
GRAPH THINKING: Master Data Management
MANAGES
MANAGES
LEADS
REGION
MANAGES
MANAGES
REGION
LEADS
LEADS
COLL
ABO
RATE
S
NEO4J USE CASES
Real Time Recommendations
Master Data Management
Fraud Detection
Identity & Access Management
Graph Based Search
Network & IT-Operations
GRAPH THINKING: Fraud Detection
OPENED_ACCOUNT
HAS IS_ISSUED
HAS
LIVES LIVES
IS_ISSUED
OPE
NED_
ACCO
UNT
GRAPH THINKING: Graph Based Search
NEO4J USE CASES
Real Time Recommendations
Master Data Management
Fraud Detection
Identity & Access Management
Graph Based Search
Network & IT-Operations
PUBLISH
INCLUDE
INCLUDE
CREATE
CAPT
URE
IN
INSO
URCE
USES
USES
IN
IN
USES
SOURCE SOURCE
NEO4J USE CASES
Real Time Recommendations
Master Data Management
Fraud Detection
Identity & Access Management
Graph Based Search
Network & IT-Operations
BROWSESCO
NNEC
TS
BRIDGES
ROUTES
POW
ERSROUTES
POWERSPOWERS
HOSTS
QUERIES
GRAPH THINKING: Network & IT-Operations
GRAPH THINKING: Identity And Access Management
NEO4J USE CASES
Real Time Recommendations
Master Data Management
Fraud Detection
Identity & Access Management
Graph Based Search
Network & IT-Operations
TRUSTS
TRUSTS
ID
ID
AUTHENTICATES
AUTH
ENTI
CATE
S
OWNS
OWNSC
AN
_REA
D
Why?+
The Traditional Approach Towards Data
SYSTEMS OF RECORD
RDBMS
SYSTEMS OF RECORD
HR-tools Supply Payments Logistics CRM Support
TRADITIONAL DATA STRUCTURE
RDBMS RDBMS RDBMSRDBMS RDBMS RDBMS
SHIFT TOWARDS SYSTEMS OF ENGAGEMENT
Users Engaging With DevicesUsers Engaging With Users Devices Engaging With Devices
SYSTEMS OF ENGAGEMENT
SHIFT TOWARDS SYSTEMS OF ENGAGEMENT
You are here!
Data volume
SYSTEMS OF RECORDRelational Database Model
StructuredPre-computed
Based on rigid rules
SYSTEMS OF ENGAGEMENTNoSQL Database Model
Highly FlexibleReal-Time QueriesHighly Contextual
Polyglot Persistence
Polyglot Persistence
• Different types of data in different ways• Take advantage of strengths of different databases
http://martinfowler.com/bliki/PolyglotPersistence.html
Polyglot Persistence
Functionality Database type
Shopping Cart Rapid session reads / writes
Key-value store
Orders / Product Catalog
Frequent reads Document
Customer social graph
Recommendation Graph
http://www.jamesserra.com/archive/2015/07/what-is-polyglot-persistence/
DataStorageandBusinessRulesExecu5on
DataMiningandAggrega5on
Applica'on
GraphDatabaseCluster
Neo4j Neo4j Neo4j
AdHocAnalysis
BulkAnaly'cInfrastructureHadoop,EDW…
DataScien'st
EndUser
DatabasesRela5onalNoSQLHadoop
Added Complexity
Challenge #1: Data Sync
MIGRATEALLDATA
MIGRATESUBSET
DUPLICATESUBSET
Non-GraphQueries GraphQueries
GraphQueriesNon-GraphQueries
AllQueries
Rela3onalDatabase
GraphDatabase
Application
Application
Application
NonGraphData
AllData
?
Course CatalogBrowse & Search
• Good use-case for a document database
Course CatalogBrowse & Search
• Good use-case for a document database
Graph Based Recommendations
Graph Based Recommendations
Graph Based Recommendations
?
Option 1: APOC Procedures
https://github.com/neo4j-contrib/neo4j-apoc-procedures
Java Stored Procedures
User-defined procedures are written in Java, deployed into the database, and called from Cypher.
http://neo4j.com/docs/developer-manual/current/#procedures
Built-in Procedures
Built-in Procedures
User-defined Procedures
https://github.com/neo4j-examples/neo4j-procedure-template
Apoc Procedures
https://github.com/neo4j-contrib/neo4j-apoc-procedures
THERE’S A PROCEDURE FOR THAT
https://github.com/neo4j-contrib/neo4j-apoc-procedures
https://github.com/neo4j-contrib/neo4j-apoc-procedures/blob/master/readme.adoc
https://neo4j-contrib.github.io/neo4j-apoc-procedures/#_loading_data_from_web_apis_json_xml_csv
?
https://docs.mongodb.com/ecosystem/tools/http-interfaces/
Option 2: Neo4j Doc Manager
Neo4j Doc Manager
•Automatically sync documents from MongoDB to Neo4j•Convert documents into graph model
http://neo4j.com/developer/mongodb/
Neo4j Doc ManagerAutomatically sync documents from MongoDB to Neo4j
Document to property graph{ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics": [ "keynote", "spring" ], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}
{ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics": [ "keynote", "spring" ], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}
Document to property graph
https://github.com/neo4j-contrib/neo4j_doc_manager
Demo
Using Data Relationships for Recommendations
Content-based filtering Recommend items based on what users have liked in the past
Collaborative filtering Predict what users like based on the similarity of their behaviors, activities and preferences to others
Movie
Person
Person
RATED
SIMILARITY
rating: 7
value: .92
Challenge #2: Operations
https://hub.docker.com/_/neo4j/
docker pull neo4j
Official Neo4j Docker Image
docker-compose.yml
Resources
http://neo4j.com/blog/couchbase-jdbc-integrations-neo4j-3-0/
Couchbase + Neo4j integration
Neo4j Integrations http://neo4j.com/developer/integration/
Neo4j + MongoDB http://neo4j.com/developer/mongodb/
Neo4j Doc Manager https://github.com/neo4j-contrib/neo4j_doc_manager
