graphconnect europe 2016 - nosql polyglot persistence: tools and integrations - william lyon
TRANSCRIPT
NoSQL Polyglot PersistenceNeo4j Tools and Integrations
William Lyon @lyonwj
Does Neo4j work with ______ ?
Languages, frameworks, databases…
Does Neo4j work with ______ ?
Languages
Does Neo4j work with ______ ?
Frameworks
Does Neo4j work with ______ ?
Databases
Does Neo4j work with ______ ?
Deployment
Does Neo4j work with ______ ?
Frameworks
Does Neo4j work with ______ ?
Languages, frameworks, databases…
Polyglot Persistence
DataStorageandBusinessRulesExecu5on
DataMiningandAggrega5on
Applica'on
GraphDatabaseCluster
Neo4j Neo4j Neo4j
AdHocAnalysis
BulkAnaly'cInfrastructureHadoop,EDW…
DataScien'st
EndUser
DatabasesRela5onalNoSQLHadoop
Polyglot Persistence
• Different types of data in different ways• Take advantage of strengths of different databases
http://martinfowler.com/bliki/PolyglotPersistence.html
Polyglot Persistence
Functionality Database type
Shopping Cart Rapid session reads / writes
Key-value store
Orders / Product Catalog
Frequent reads Document
Customer social graph
Recommendation Graph
http://www.jamesserra.com/archive/2015/07/what-is-polyglot-persistence/
Added Complexity
Challenge #1: Data Sync
MIGRATEALLDATA
MIGRATESUBSET
DUPLICATESUBSET
Non-GraphQueries GraphQueries
GraphQueriesNon-GraphQueries
AllQueries
Rela3onalDatabase
GraphDatabase
Application
Application
Application
NonGraphData
AllData
?
Course CatalogBrowse & Search
• Good use-case for a document database
?
Tool #1: Neo4j Doc Manager
• Automatically sync documents from MongoDB to Neo4j• Convert BSON documents into graph model• Initial version available• Feedback welcome!
http://neo4j.com/developer/mongodb/
Neo4j Doc ManagerAutomatically sync documents from MongoDB to Neo4j
Document to property graph{ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics": [ "keynote", "spring" ], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}
{ "session": { "title": "12 Years of Spring: An Open Source Journey", "abstract": "Spring emerged as a core open source project in early 2003 and evolved to a broad portfolio of open source projects up until 2015." }, "topics": [ "keynote", "spring" ], "room": "Auditorium", "timeslot": "Wed 29th, 09:30-10:30", "speaker": { "name": "Juergen Hoeller", "bio": "Juergen Hoeller is co-founder of the Spring Framework open source project.", "twitter": "https://twitter.com/springjuergen", "picture": "http://www.springio.net/wp-content/uploads/2014/11/juergen_hoeller-220x220.jpeg" }}
Document to property graph
https://github.com/neo4j-contrib/neo4j_doc_manager
Neo4j + MongoDB Webinar
http://neo4j.com/webinars/
Building a polyglot application with Neo4j and MongoDB
http://neo4j.com/blog/couchbase-jdbc-integrations-neo4j-3-0/
Couchbase + Neo4j integration
Challenge #2: Operations
https://hub.docker.com/_/neo4j/
docker pull neo4j
Tool #2: Official Neo4j Docker Image
docker-compose.yml
Challenge #3: Data Models
Tool #3: Neo4j Cassandra Data Import Tool
http://neo4j.com/developer/cassandra/
• Python command line app• Uses Python Neo4j and Cassandra drivers
Neo4j Cassandra Data Import ToolCassandra schema metadata
python connector.py parse -k playlist
Tables become nodes
Artist
Track
Column family to property graph
What’s that?
Neo4j Cassandra Data Import ToolUser defined data model mappings
Neo4j Cassandra Data Import Tool
{p}, node property{r}, relationship{u}, unique constraint field{i}, create index
User defined data model mappings
Neo4j Cassandra Data Import ToolUser defined data model mappings
Neo4j Cassandra Data Import ToolGenerates LOAD CSV Cypher
python connector.py export -k playlist -t track_by_id,artists_by_first_letter
Neo4j 3.0
Neo4j 3.0
Bolt Stored ProceduresDrivers
Neo4j 3.0 Stored procedures
User-defined procedures are written in Java, deployed into the database, and called from Cypher.
http://neo4j.com/docs/java-reference/current/#procedures
https://github.com/neo4j-contrib/neo4j-apoc-procedures
https://github.com/neo4j-contrib/neo4j-apoc-procedures/blob/master/readme.adoc
Northwind
Northwind
Bolt and Neo4j 3.0 Drivers
Binary protocol for Neo4j
http://neo4j.com/docs/developer-manual/current/#driver
https://github.com/jexp/neo4j-spark-connector
Neo4j Spark ConnectorUsing Neo4j 3.0 Java Driver with Bolt
https://github.com/johnymontana/neovis.js
neovis.jsEmbeddable graph visualization powered by Neo4j 3.0 Bolt Javascript driver
