transforming your data: a worked example
TRANSCRIPT
09:00-09:30 09:30-10:15
10:15-11:00 11:00-11:30 11:30-12:30
12:30-13:30 13:30-17:00
Breakfast and RegistrationThe Connected Data Imperative: Why Graphs
Transform Your Data: A Worked ExampleBreakEnterprise Ready: A Look at Neo4j in Production LunchHands-On Training Session
Agenda
APRIL 26, 2017SANTA CLARA
Transform Your Data
Neo4j Graph Day - Santa Clara 2017William Lyon@lyonwj
A Worked Example
Agenda
• Graph database - developer perspective• (relational)-[:TO]->(graph)• Use cases• Fraud Detection• Recommendations
• Developer tooling
Neo4j
Graph Database
• Property graph data model• (open)Cypher query language• Native graph processing• Language drivers• Open source
neo4j.com
Graph Data Model
Labeled Property Graph Model
Labeled Property Graph Model
The Graph
https://github.com/johnymontana/neo4j-datasets/tree/master/yelp
The Graph
https://github.com/johnymontana/mattermark-graphql-neo4j
The Graph
https://github.com/neo4j-meetups/modeling-worked-example
Cypher
Use Case: Fraud Detection
Organized in groups Synthetic Identities Stolen Identities Hijacked Devices
Who Are Today’s Fraudsters?
Fraud Rings and Synthetic Identities
ACCOUNT HOLDER 2
Modeling a fraud ring as a graph
ACCOUNT HOLDER 1
ACCOUNT HOLDER 3
ACCOUNT HOLDER 2
ACCOUNT HOLDER 1
ACCOUNT HOLDER 3
CREDIT CARD
BANKACCOUNT
BANKACCOUNT
BANKACCOUNT
PHONE NUMBER
UNSECURED LOAN
SSN 2
UNSECURED LOAN
Modeling a fraud ring as a graph
ACCOUNT HOLDER 2
ACCOUNT HOLDER 1
ACCOUNT HOLDER 3
CREDIT CARD
BANKACCOUNT
BANKACCOUNT
BANKACCOUNT
ADDRESS
PHONE NUMBER
PHONE NUMBER
SSN 2
UNSECURED LOAN
SSN 2
UNSECURED LOAN
Modeling a fraud ring as a graph
https://neo4j.com/graphgist/9d627127-003b-411a-b3ce-f8d3970c2afa?ref=solutions
https://neo4j.com/graphgists/
Credit Card Fraud
(relational)-[:TO]->(graph)
FROM RDBMS TO GRAPHS
Northwind
Northwind - the canonical RDBMS Example
( )-[:TO]->(Graph)
( )-[:IS_BETTER_AS]->(Graph)
Starting with the ER Diagram
Locate the Foreign Keys
Drop the Foreign Keys
Find the JOIN Tables
(Simple) JOIN Tables Become Relationships
Attributed JOIN Tables -> Relationships with Properties
As a Graph
https://github.com/neo4j-contrib/neo4j-apoc-procedures/blob/master/readme.adoc
Northwind
Northwind
Neo4j Language Drivers
Neo4j Drivers
https://neo4j.com/developer/language-guides/
POWERING AN APP
Simple App
Simple App
Simple Python Code
Simple Python Code
Simple Python Code
Simple Python Code
Use Case: Personalized Recommendations
Recommendations drive user engagement
“35 percent of what consumers purchase on Amazon and 75 percent of what they watch on Netflix come from product recommendations”
http://www.mckinsey.com/industries/retail/our-insights/how-retailers-can-keep-up-with-consumers
Product Recommendations
Effective product recommendation algorithms has become the new standard in online retail — directly affecting revenue streams and the shopping experience.
Logistics/DeliveryRouting recommendations allows companies to save money on routing and delivery, and provide better and faster service.
Promotion recommendations
Building powerful personalized promotion engines is another area within retail that requires input from multiple data sources, and real-time, session based queries, which is an ideal task to solve with Neo4j.
Today Recommendation Engines are At the Core of Digitization in Retail
Powerful recommendation engines rely on the connections between multiple
sources of data
Powerful recommendation engines rely on the connections between multiple
sources of data
Dreamhouse Series 15% off
Dreamhouse Series 15% off
The Store
Search
Hi, loginMy Account
People who bought Side Table also bought:
Coffee Table
$235Low Book Shelf
$150Bed Side Table
$90
Mobile Brick & Mortar
Multi-Channel
Web
The Store
People who bought Side Table also bought: Similar product in from Home Office Series:
Hi, loginMy AccountSearch
Dreamhouse Series 15% off Dreamhouse Series 15% off
All departments Living room | Kitchen | Hallway | Lightning | Bedroom | Garden | Home Office Space
Tr a c k O r d e r s | G i f t C a r d s | S t o r e fin d e r | C r e d i t C a r d | G r o c e r y P i c k u p | H e l p
Wood Side Table
$110Green Side Table
$135Walnut Side Table
$120Coffee Table
$235Low Book Shelf
$150Bed Side Table
$90
Product Recommendations
Dreamhouse Series 15% off
Dreamhouse Series 15% off
The Store
Search
Hi, loginMy Account
People who bought Side Table also bought:
Coffee Table
$235Low Book Shelf
$150Bed Side Table
$90
Mobile Brick & MortarWeb
The Store
People who bought Side Table also bought: Similar product in from Home Office Series:
Hi, loginMy AccountSearch
Dreamhouse Series 15% off Dreamhouse Series 15% off
All departments Living room | Kitchen | Hallway | Lightning | Bedroom | Garden | Home Office Space
Tr a c k O r d e r s | G i f t C a r d s | S t o r e fin d e r | C r e d i t C a r d | G r o c e r y P i c k u p | H e l p
Wood Side Table
$110Green Side Table
$135Walnut Side Table
$120Coffee Table
$235Low Book Shelf
$150Bed Side Table
$90
The Store
People who bought Side Table also bought: Similar product in from Home Office Series:
Hi, loginMy AccountSearch
Dreamhouse Series 15% off Dreamhouse Series 15% off
All departments Living room | Kitchen | Hallway | Lightning | Bedroom | Garden | Home Office Space
Tr a c k O r d e r s | G i f t C a r d s | S t o r e fin d e r | C r e d i t C a r d | G r o c e r y P i c k u p | H e l p
Wood Side Table
$110Green Side Table
$135Walnut Side Table
$120Coffee Table
$235Low Book Shelf
$150Bed Side Table
$90
Personalized Promotions Personalized Real-Time Recommendations
Personalized Real-Time Recommendations
The Store
People who bought Side Table also bought: Similar product in from Home Office Series:
Hi, loginMy AccountSearch
Dreamhouse Series 15% off Dreamhouse Series 15% off
All departments Living room | Kitchen | Hallway | Lightning | Bedroom | Garden | Home Office Space
Tr a c k O r d e r s | G i f t C a r d s | S t o r e fin d e r | C r e d i t C a r d | G r o c e r y P i c k u p | H e l p
Wood Side Table
$110Green Side Table
$135Walnut Side Table
$120Coffee Table
$235Low Book Shelf
$150Bed Side Table
$90
Data-Model (Expressed as
a graph)
Category
Category
Product
Product
Product
Collaborative FilteringAn algorithm that considers users interactions with products, with the
assumption that other users will behave in similar ways.
Algorithm Types
Content BasedAn algorithm that considers
similarities between products and categories of products.
Customer
Customer
Product
Product
Product
Category Price ConfigurationsLocation
Silos & Polyglot Persistence
Purchase ViewReviewReturn In-store PurchasesInventory
Products Customers / Users
Location
Purchases
RELATIONAL DB WIDE COLUMN STORE
Views
DOCUMENT STORE
User Review
RELATIONAL DB
In-Store Purchase
Shopping Cart
KEY VALUE STORE
Product Catalogue
DOCUMENT STORE
Purchases
RELATIONAL DB WIDE COLUMN STORE
Views
DOCUMENT STORE
User Review
RELATIONAL DB
In-Store Purchase
Shopping Cart
KEY VALUE STORE
Product Catalogue
DOCUMENT STORE
Silos & Polyglot Persistence
Category Price ConfigurationsLocation Purchase ViewReviewReturn In-store PurchasesInventory
Products Customers / Users
Location
Purchases
RELATIONAL DB WIDE COLUMN STORE
Views
DOCUMENT STORE
User Review
RELATIONAL DB
In-Store Purchase
Shopping Cart
KEY VALUE STORE
Product Catalogue
DOCUMENT STORE
Category Price ConfigurationsLocation
Polyglot Persistence
Purchase ViewReviewReturn In-store PurchasesInventory LocationCategory Price ConfigurationsLocation Purchase ViewReviewReturn In-store PurchasesInventory
Products Customers / Users
Location
Data Lake
Purchases
RELATIONAL DB
Product Catalogue
DOCUMENT STORE WIDE COLUMN STORE
Views
DOCUMENT STORE
User Review
RELATIONAL DB
In-Store Purchase
Shopping Cart
KEY VALUE STORE
Recommendations require an operational workload — it’s in the moment, real-time!
Good for Analytics, BI, Map ReduceNon-Operational, Slow Queries
Purchases
RELATIONAL DB
Product Catalogue
DOCUMENT STORE WIDE COLUMN STORE
Views
DOCUMENT STORE
User Review
RELATIONAL DB
In-Store Purchase
Shopping Cart
KEY VALUE STORE
Connector
Drivers: Java | JavaScript | Python | .Net | PHP | Go | Ruby
Apps and Systems
Real-Time Queries
Using Data Relationships for Recommendations
Content-based filtering Recommend items based on what users have liked in the past
Collaborative filtering Predict what users like based on the similarity of their behaviors, activities and preferences to others
Movie
Person
Person
RATED
SIMILARITY
rating: 7
value: .92
Collaborative Filtering
Collaborative Filtering
Collaborative Filtering
Collaborative Filtering
Link prediction
In Cypher
In Cypher
Basic initial approach. Improvements:- aggregate across all purchases- scoring / normalize- compute similarity metrics
Content Filtering
Content Filtering
Content Filtering w/ Cypher
Content Filtering - Concept Hierarchy
Content Filtering - Concept Hierarchy
Content Filtering - Concept Hierarchy
Content Filtering - Concept Hierarchy w/ Cypher
Content Filtering - Concept Hierarchy w/ Cypher
Basic initial approach. Improvements:- aggregate across all purchases- cold start- variable length concept hierarchy- tag similarity / clusters
(you)-[:HAVE]->(?)(?)<-[:ANSWERS]-(will)
09:00-09:30 09:30-10:15
10:15-11:00 11:00-11:30 11:30-12:30
12:30-13:30 13:30-17:00
Breakfast and RegistrationThe Connected Data Imperative: Why Graphs
Transform Your Data: A Worked ExampleBreakEnterprise Ready: A Look at Neo4j in Production LunchHands-On Training Session
Agenda
APRIL 26, 2017SANTA CLARA