how nosql paid off for telenor
DESCRIPTION
This presentation describes how NOSQL solutions such as the Neo4j graph database and Lucene/Solr index was used in a classic middleware stack in Telenor to solve perfomance and scalability issues.TRANSCRIPT
How NoSQL Paid Off for Telenor
JavaZone
13 September 2012 - Oslo
Sebastian Verheughe
Architect and developer
Telenor - mobile middleware services (COS)
Katrina Sponheim
Architect and developer
Telenor – business self service solutions
Telenor NoSQL Experience
o The problem
o The business case
o The solution
o The challenges
o The results
o My 5 cents
The Problem
Min Bedrift
Self service portal where Telenor's corporate customers can manage their entire portfolio of
products.
From small businesses to large corporations
Telenor's Corporate Customer Structure
Customer Acme Corporation
Customer Acme Consulting
Customer Acme Development
Account Construction
Account Demolition
Subscription Huey
Subscription Dewey
Subscription Louie
The Challenge With Large Corporate Customers
Customers with large portfolios presented a couple of challenges for the self service solution Min Bedrift:
1. Middleware Services - Not Designed for Search
The middleware services were not designed for managing large data volumes, resulting in a lot of processing in the client, and the need for extensive caching there.
2. Resource Authorization – Long Calculation Time
User access to resources required the middleware to calculate and cache all accesses at logon, something that could take up to many minutes.
The Nightly Logon & Pre-fetch Solution
In order to achieve acceptable response times in MinBedrift, administrators were logged on and customer data was pre-fetched and put in a cache each night.
However, as the usage of the solution grew, it became obvious that the time window available for pre-fetching each night was closing fast.
0
6
3 9
2012
0
6
3 9
2013
0
6
3 9
2014
The Future - Unhandled
Telenor calculated that the pre-fetch time window would soon be filled, and a increasing percentage of the customers would experience logon response times above the acceptable x sec.
Login
time
Customer Portfolio Size
Portfolio Size
Pre-fetched
x
Not pre-fetched
The Business Impact
In the end, Telenor would risk losing corporate customers due to deteriorated customer experience
Other Caching Drawbacks
o Stale data up to 24 hours old
o Refresh/login for new users still takes a lot of time
o Memory challenges in Min Bedrift
o Unwanted network/middleware/database load
The Business Case
Business Case
The business case is built on the negative consequence of NOT addressing the problem.
Loss of customers (revenue)
Reduced sales transactions (revenue)
Other
Increased manual support (expenses)
The Solution
Solution Requirements - High Level
The middleware search services should be designed to support large data sets in a better way for the all clients.
Resource authorization must be fast enough to deliver real time calculations on demand.
The Previous Architecture
Master Data
RDBMS
Multiple Sources
Client Client Client Client Client Client
Middleware Services
The New Architecture
One Master (r/w) – Several Replicas (r)
Middleware Services
Multiple Sources
Client Client Client Client Client Client
Solr / Lucene Neo4j
Master Data
Sybase RDBMS Search Res Auth
Domain Event Messaging
Solr / Lucene Neo4j
Master Data
RDBMS
Domain Event
Domain Event
Raw
DB
Event
Apache Camel
Res Auth Search Messaging
Putting it All Together
Min Bedrift MW Search MW Auth
search
getAuthorizedResources
filteredSearch
authorized match
Lucene / Solr Solution
Search Service
Today implemented in Min Bedrift
o Cached nightly
o Simple, and iterates over the nodes when searching
o With memory/GC challenges
New Search Service
Data stored in Solr/Lucene search engine
o New middleware module exposing WS using tomcat
o Everything indexed makes search extremely fast
o De-normalized data does not require joins
o Search by relevance, paging, sorting and much more
Solr Cores
Search
Service
Subscription
Customer
Account Client
Entity Denormalization
Subscription
Customer
Account
also contains account name & id
also contains customer name & id
An entity may include data from several tables
Solr/Lucene - Denormalized List View
Arthur | Jackson Total 2341 555 21 1234
Customer
Subscription
Account
Lisa | Simpson Youth 3435 555 64 3634
John | Brown Pro 5352 555 25 5433
user
has
Searching by Relevance
Search some or all rows, and return hits by relevance (or sorted)
User Name Subscription Phone Number Account Ref. Score Rank
Jane Youth 555 21 3253 3253 15
Paul Premium 555 23 4365 5262
John Standard 555 95 1436 7346
Nina Standard 555 15 3263 3734
Lydia Youth 555 92 3253 7334 5
Tom Standard 555 02 6394 3212
Neil Premium 555 03 2583 3523
Subscription
10 5
5
1
2
Neo4j Solution
Resource Authorization Service
Stored procedure in RDBMS calculating all accesses
o Uses several minutes to calculate for large customers
o Cached for up to 24 hours
o Extremely complex to understand (1500 lines of sql)
o Tightly coupled with other services querying the database
New Resource Authorization Service
Customer structure stored in Neo4j graph database
o New middleware module exposing WS using tomcat
o Designed to focus on the relationships between objects
o Very fast – independent of total amount of objects stored
Nodes and Relationships
C
C
A
S
A A
S S
S S S
S
U
o Relationships with type and direction
o Nodes (with type as property)
User Customer Account Subscription
USER_ACCESS (with prop inherit: true/false)
A
S S
C
S
PART_OF
CONTROLLED_BY
SUBSCRIBED_BY
Traversal (query)
C
C
A
S
A A
S S
S S S
S
U
All traversals start from a single node
User Customer Account Subscription
A
S S
C
S
A
S S
The start node is often the user node in our case
Following the Relationships
One custom PathExpander class
o Only follow valid relationships and direction
o Only follow necessary relationships
o Check inheritance rules for current path
Just override the expand method
Iterable<Relationship> expand(Path, BranchState)
Picking the Nodes
Custom Evaluator
o Decide to include or exclude
o Delegate to filter that fits your search
o Filter may further evaluate neighbor nodes
Just override the evaluate method
Evaluation evaluate(Path path){
if (resourceFilter.filter(path)
return Evaluation.INCLUDE_AND_CONTINUE
return Evaluation.EXCLUDE_AND_CONTINUE
}
Example Access Authorization
C
C
A
S
A A
S S
S
S S
S
U
Retrieve all subscriptions using a fan out search
User Customer Account Subscription
A
S S
A
S S
C
S
Example Access Authorization
C
C
A
S
A A
S S
S
S S
S
U
Has access to resource using a reverse search to limit number
of nodes to evaluate. Find all paths, and validate one of them.
User Customer Account Subscription
A
S S
A
S S
C
S
The Challenges
Lucene/Solr
o Using a document store in a relational world – updates
o Change mindset to search by relevance, not sorting
o The time is in the small stuff – not difficult but needs learning
o What type of queries to search on this platform, and NOT
o Scaling & Distribution – Actually, not a challenge…
Neo4j
o Competence
o New way of thinking
o Making them really fast (profile & understand graph impl)
o Getting the classic middleware take use of the new service
o What type of queries to search on this platform, and NOT
o Scaling, not easy across servers (not needed for now)
Messaging
o Mapping from a relational model (need cache)
The Results
Project State
o Phase 1 in production (subscription only, nightly populated)
o Phase 2 in system test (the rest + live population)
The following results are from the test environment now
Neo4J Runtime Environment
Initial State: Prewarmed at startup, all data in heap
Population: ~20 M nodes (all indexed)
~20 M node properties (only 1 per node)
~50 M relationships
Batchwise (50 K nodes) in 35 minutes
Base heap usage: 10 GB (of 16 GB)
Load: Minimal (not measured with heavy load)
Neo4J Measured Performance
Customers measured for performance:
X Y Z
Corporation Customer Accounts Subscriptions
X 160 1 300 147 000
Y 32 000 23 000 52 000
Z 7 18 95 000
Neo4J Measured Performance
X: 1 300 Y: 23 000 Z: 18
10 ms 260 ms 2 ms
Find accounts
Neo4J Measured Performance
X: 147 000 Y: 52 000 Z: 95 000
1 700 ms 750 ms 1300 ms
Find subscriptions
Neo4J Measured Performance
X Y Z
2 ms 2 ms 2 ms
Has access to subscription
Solr/Lucene Measured Performance
X Y Z
7 ms 58 ms 4 ms
Find subscriptions
Service Performance from Min Bedrift
120 ms
55 ms
“Google” search for corporation x: 120 ms
Auth Graph
Min Bedrift
Search Solr
searchAllResources
findAuthorizedResources
Old vs. New Resource Authorization Service
Calculate All Resources RDBMS Graph
X 12 min 18 sec < 2 sec
Y 22 min 58 sec < 2 sec
Z 3 min 15 sec < 2 sec
Cold Warm Heap
Min Bedrift
Demo
Summary
Scalable It allows customer growth
Fast logon On demand resource authorization
Fast search Server side search engine much faster
Reusable All clients may use new services
Fresh data Not up to 24 hours old – almost live
Alternatives
In-Memory Database (Sybase)
This option was discussed, but license cost and the uncertainty if it would be enough made us go for the NoSQL option.
Other NoSQL Solutions
We chose to prototype Neo4j and Lucene/Solr because they were popular and seemed to fit us well, and since it worked we stuck with them.
How We Started Using NoSQL Technology
o Downloaded and prototyped technology very early
o Got training on site to accelerate the development startup
o At the end of development, did a review/QA of the solution
For Lucene/Solr, we got training and support from local Solr/Lucene expert consultant Jan Høydahl
For Neo4j, we got training and excellent support from NeoTech directly
My 5 Cents
Think About…
No Language Standard for Graph Databases
How simple (or possible) is it to change the NoSQL provider?
Working With Relationships
Graph databases are intuitive and fast to work with when interested in how objects are related to each other.
Complexity
You introduce complexity, so make sure it is worth it!
New Technology
Do you have enough in-house competence, or can you easily buy the necessary competence? Also when maintaining the code.
Gentle NoSQL Introduction
Easier to start using when supporting a specific and limited services
The End Questions?