graphconnect europe 2016 - tuning your cypher - petra selmer, mark needham
TRANSCRIPT
Tuning CypherMark Needham @markhneedham
Petra Selmer@Aethelraed
Why do we need to tune?
‣ No query planner is ever perfect‣ You know your domain better than the
database
The Cost planner
‣ Introduced in 2.2.0‣ It uses the statistics service in Neo4j to
assign costs to various query execution plans, picking the cheapest one
‣ All queries use this by default
Cypher query execution
‣ http://neo4j.com/docs/snapshot/execution-plans.html‣ http://neo4j.com/blog/introducing-new-cypher-query-optimizer
How do I view a query plan?
‣ EXPLAIN• shows the execution plan without actually
executing it or returning any results.
‣ PROFILE• executes the statement and returns the results
along with profiling information.
Neo4j’s longest plan (so far…)
Neo4j’s longest plan (so far…)
Neo4j’s longest plan (so far…)
What is our goal?
At a high level, the goal is simple: get the number of db hits down.
an abstract unit of storage engine work.
What is a database hit?
“”
‣ Operators to look out for• All nodes scan expensive
• Label scan cheaper
• Node index seek cheapest
• Node index scan used for range queries
‣ http://neo4j.com/docs/3.0.0-RC1/execution-plans.html
Execution plan operators
Our data set
Finding The Matrix
MATCH (movie {title: "The Matrix"})
RETURN movie
Finding The Matrix
MATCH (movie
{title: "The Matrix"})
RETURN movie
Tip: Use labels
MATCH (movie:Movie
{title: "The Matrix"})
RETURN movie
Tip: Use labels
MATCH (movie:Movie
{title: "The Matrix"})
RETURN movie
Finding The Matrix MATCH (movie
{title: "The Matrix"})
RETURN movie
MATCH (movie:Movie
{title: "The Matrix"})
RETURN movie
Tip: Use indexes and constraints
‣ Indexes for non unique values‣ Constraints for unique values
CREATE INDEX ON :Movie(title)
CREATE INDEX ON :Person(name)
CREATE CONSTRAINT ON (g:Genre)
ASSERT g.name IS UNIQUE
How does Neo4j use indexes?
‣ Indexes are only used to find the starting point for queries.
Use index scans to look up rows in tables and join them with rows from other tables
Use indexes to find the starting points for a query.
Relational
Graph
Tip: Use indexes and constraints
MATCH (movie:Movie
{title: "The Matrix"})
RETURN movie
Finding The Matrix (no index)MATCH (movie:Movie
{title: "The Matrix"})
RETURN movie
(index)MATCH (movie:Movie
{title: "The Matrix"})
RETURN movie
Actors who appeared together
MATCH (a:Person {name:"Tom Hanks"})
-[:ACTS_IN]->()<-[:ACTS_IN]-
(b:Person {name:"Meg Ryan"})
RETURN COUNT(*)
Actors who appeared together
MATCH (a:Person {name:"Tom Hanks"})
-[:ACTS_IN]->()<-[:ACTS_IN]-
(b:Person {name:"Meg Ryan"})
RETURN COUNT(*)
Tip: Enforce index usage
MATCH (a:Person {name:"Tom Hanks"})
-[:ACTS_IN]->()<-[:ACTS_IN]-
(b:Person {name:"Meg Ryan"})
USING INDEX a:Person(name)
USING INDEX b:Person(name)
RETURN COUNT(*)
Tip: Enforce index usage
MATCH (a:Person {name:"Tom Hanks"})
-[:ACTS_IN]->()<-[:ACTS_IN]-
(b:Person {name:"Meg Ryan"})
USING INDEX a:Person(name)
USING INDEX b:Person(name)
RETURN COUNT(*)
Actors who appeared togetherMATCH (a:Person {name:"Tom Hanks"})
-[:ACTS_IN]->()<-[:ACTS_IN]-
(b:Person {name:"Meg Ryan"})
RETURN COUNT(*)
MATCH (a:Person {name:"Tom Hanks"})
-[:ACTS_IN]->()<-[:ACTS_IN]-
(b:Person {name:"Meg Ryan"})
USING INDEX a:Person(name)
USING INDEX b:Person(name)
RETURN COUNT(*)
Tom Hanks’ colleagues’ movies
MATCH (p:Person {name:"Tom Hanks"})
-[:ACTS_IN]->(m1)<-[:ACTS_IN]-
(coActor)-[:ACTS_IN]->(m2)
RETURN distinct m2.title
Tom Hanks’ colleagues’ movies
MATCH (p:Person {name:"Tom Hanks"})
-[:ACTS_IN]->(m1)<-[:ACTS_IN]-
(coActor)-[:ACTS_IN]->(m2)
RETURN distinct m2.title
Tip: Reduce cardinality of WIP
MATCH (p:Person {name:"Tom Hanks"})
-[:ACTS_IN]->(m1)<-[:ACTS_IN]-
(coActor)
WITH DISTINCT coActor
MATCH (coActor)-[:ACTS_IN]->(m2)
RETURN distinct m2.title
Tip: Reduce cardinality of WIP
MATCH (p:Person {name:"Tom Hanks"})
-[:ACTS_IN]->(m1)<-[:ACTS_IN]-
(coActor)
WITH DISTINCT coActor
MATCH (coActor)-[:ACTS_IN]->(m2)
RETURN distinct m2.title
MATCH (p:Person {name:"Tom Hanks"})
-[:ACTS_IN]->(m1)<-[:ACTS_IN]-(coActor)
WITH DISTINCT coActor
MATCH (coActor)-[:ACTS_IN]->(m2)
RETURN distinct m2.title
Tom Hanks’ colleagues’ moviesMATCH (p:Person {name:"Tom Hanks"})
-[:ACTS_IN]->(m1)<-[:ACTS_IN]-
(coActor)-[:ACTS_IN]->(m2)
RETURN distinct m2.title;
Hints
USING INDEX Force the use of a specific index
MATCH (a:Person {name:"TomHanks"})-[:ACTS_IN]->()
USING INDEX a:Person(name)
RETURN count(*)
Hints
USING SCAN Forces a label scan on lower cardinality labels
MATCH (a:Actor)-->(m:Movie:Comedy)
USING SCAN m:Comedy
RETURN count(distinct a)
Even more tips...
Use parameters
MATCH (p:Person {name: {name}})
-[:ACTS_IN]->(m)
RETURN m.title
MATCH (p:Person {name:"Tom Hanks"})
-[:ACTS_IN]->(m)
RETURN m.title
Avoid Cartesian products
‣ Easy to do this inadvertently:
MATCH (a:Actor), (m:Movie)
RETURN count(a), count(m)
‣ This is correct, and performs betterMATCH (a:Actor)
WITH count(a) as a_count
MATCH (m:Movie)
RETURN a_count, count(m)
Watch out for those warnings!
Cardinalities
Watch those rows!
Only RETURN what you need
‣ This is not recommended:MATCH (a:Actor)
RETURN a
‣ Use this instead:MATCH (a:Actor)
RETURN a.name, a.birthdate, a.height
tl;dr
‣ View query plans with EXPLAIN and PROFILE‣ Use labels‣ Index your starting points‣ Reduce work in progress‣ Remember the hints
Thanks for coming
‣ And don’t forget, if the tips aren’t working ask us for help on Stack Overflow!
Mark Needham @markhneedham Petra Selmer @Aethelraed