Abdelmonaim Remani | Just.me Inc.

The Rise of NoSQL and Polyglot Persistence

About Me

• Software Architect at Just.me Inc.• Interested in technology evangelism and enterprise software

development and architecture• Frequent speaker (JavaOne, JAX, OSCON, ORDEV, etc…)• Open-source advocate• President and founder of a number of user group

– NorCal Java User Group– The Silicon Valley Spring User Group– The Silicon Valley Dart Meetup

• Bio: http://about.me/PolymathicCoder• Twitter: @PolymathicCoder• Email: abdelmonaim.remani@gmail.com

License

• Creative Commons Attribution Non-Commercial 3.0 Unported– http://creativecommons.org/licenses/by-nc/3.0

• Disclaimer: The graphics and the logo in the presentation belong to their rightful owners

The Golden Age of Relational Databases

Relational Data Stores

• Relational Data Stores have been the predominant choice in storing data– The existence mature solutions

• Oracle, MySQL, Ms SQL Server, etc…

– Wide adoption and familiarity• Developers and even advanced business users

– An abundance of tools– Etc…

• It became the De-Facto standard

The Relational Model

• Data– Stored in

• 2 dimensional tables (Relations)• Rows (tuples) and columns (attributes)

• Has well-define enforced schema– Relations themselves– Integrity constrains

• Normalization– Smaller tables with well-defined relationship between them– Why?

• Minimized redundancy• No modification anomalies

– Modification Propagation or cascading

The Relational Model

• Supported by SQL (Structured Query Language)– A somewhat standardized query language– Very flexible– Many Operations

• Across multiple relations such as JOIN• Aggregations such as GROUP BY• Etc…

The Relational Model

•Transactional• ACID

– Atomicity» All or nothing

– Consistency» From one valid state to another

– Isolation» Concurrency result in a valid state

– Durability» Once committed, it’s forever

The Relational Model

• Designed with the assumptions that

– The end-user will directly interact with database

» It makes sense that the RDBMS should manage concurrency and integrity

» Access Patterns are unknown

» A flexible query language that is close to English

» Data structure with no bias towards a particular pattern of querying

– The database runs on a single machine

» The only way to promise true ACID

Road Bumps

• We started building more complex applications on top of relational databases

– Business logic moved out of the RDBMS

» Fewer triggers and stored procedures and replaced by equivalent application layer code

– The applications themselves evolved beyond the procedural paradigm to a more OOP approach

» The Object-Relational impedance mismatch

» ORM framework to the rescue

Scalability

We became data hoarders!

• As our datasets grew out of control• Performance decreases exponentially

– We buy a beefier machines• Larry Ellison’s most expensive RAC and make

him even richer

• This put off the problem for a little while

Optimization

• We hire a guy– Indexes half of the databases

• Made those queries a little faster

– Creates materialized views for complex joins• Nightmare to maintain, get stale, etc…

– He de-normalizes• Any thing but a smooth transition!• Redundancy

– He introduces Caching• Data too stale• More redundancy

Clustering

• We hire another guy– Tells us that we hit the limit of the one machine– You need to scale out (Horizontally)

• Master/Slave– Assuming you read more than you write– Write to the Master and Read from the Slaves– Master needs to replicate data across the slaves

» Risk incorrect reads– How’s that consistent?!!

• Sharding– Improves reads as much as writes– Can’t join across partitions– No referential integrity– Requires modification of client applications– Introduces a single-point of failure– How’s that consistent?!!

What’s the Point?

• We vertically scale our relational database– We’re no longer consistent– No ACIDity?– We loose query flexibility

• Are we doing something wrong?

The CAP Theorem

The CAP Theorem

• Eric Brewer on distributed systems– Pick tow out of

• Consistency• Availability• Partition Tolerance

• There is Fast Cheap Good service– Cheap Good service won’t be Fast– Fast Good service won’t be Cheap– Fast Cheap service won’t be Good

Relational Model & CAP

• Relational Data Stores happen to favor– Consistency and Availability– For historical reasons

• They are key to certain type of applications• The bank example

– I deposit $100 in my friend’s bank account– Blah blah blah…

• According to CAP, Partition Tolerance is impossible meaning that horizontal scaling is impossible

Scheiße!

• We’re in a pickle– Too much data in CA model– Vertical Scaling

• Too expensive• Not sustainable

• Forced to explore other alternatives in light of CAP

What AP Looks Like

• Partition Tolerance– Since we reached the limit of the one machine

we have no choice but to scale horizontally– Which means to be partition tolerant

• Availability– Nobody is willing to give up most of the time– This becomes even better with distribution– In a cluster of servers

• The individual node might be unreliable by itself• But a whole inherently reliable

What AP Looks Like

• According the CAP we simply cannot have C• Consistency

– I make a update and all subsequent read the most updated value

– Unfortunately this is impossible as it takes time for the change to be replicated across each node of the cluster

• What a bummer?!• Let’s look and AP system

– DNS (Domain Naming Service)• Not all the nodes have the most updated records (You

register that domain name and wait for a few days to guarantee that every DNS knows about it)

Eventual Consistency

• This is no so bad– It means that we just settled for a lesser degree Consistency

• So what if– Mohammad in Morocco updated his relationship status to

single on an some edge node– His cousin who lives Spain saw it immediately because they

happen to be on the same edge node– His secret admirer Sara who lives in the United States could

not see it until an hour later– His bother in Japan got the update the next day– They all got it eventually!

• Eventual Consistency as Opposed to Immediate Consistency

The Compromise

• We settle for weaker consistency model– BASE

• Basically Available• Soft state• Eventual Consistency

• ACID on the individual node BASE on the cluster

The Slippery Slope of the Faithless

You might as well Question…

• Schema– Logical

• Well-defined and rigid in relational databases• Why not a flexible one or even no schema

– Physical• B Trees in most relational databases• Why not use some other underlying data

structure

You might as well Question…

• Integrity Constraints– Who cares?

• A Query Language– Anything would do…

• Security– None

• Name it…

NoSQL: Going Rogue…

NoSQL

• A wide range of specialized data stores with the goal of addressing the challenges of the relational model

• Eric Evans – The whole point of seeking alternatives is that you

need to solve a problem that relational databases are a bad fit for

• Let me make it easier– It is does not anti-SQL or anti-Relational– Any data store that is non-relational

• “Not Only SQL” instead of “NO SQL”

SQL vs. NoSQL

A single machine

CA

Scale Vertically

SQL

ACID

Full Indexes

A cluster

AP/CA/CP

Scale Horizontally

Custom APIs

BASE

Mostly on Keys

There are outliers of course

SQL vs. NoSQL

Rigid Schema

Flexible Queries

Schema-less

Pre-defined Queries

There are outliers of course

• SQL (Relational)– Concerned about what the data consists of

• NoSQL (Non-Relational)– Concerned with how the data is queried

The Zoo

Key-Value Data Stores

• Basically a big hash map associative array– Very Simple– Very fast read and write– No secondary indexes

• Use When– Your data is not highly related– All you need is basic CRUD

• Challenges– Complex queries

• Check out the Amazon Dynamo Paper• http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.

pdf

• Featured Projects– DynamoDB http://hbase.apache.org/– Riak http://wiki.basho.com/– Redis http://redis.io/

Columnar Stores• In a table, data of the same column is stored together

– Storage is not wasted on null value as in row-based stores (RDBMS)– Great for sparse tables– Very fast column operation including aggregation

• Use When– Big Data (Excellent leverage of Map Reduce)– Need compression or versioning

• Challenges– You better know your access patterns before hand– Keys design is not trivial

• Check out Google’s BigTable Paper– http://static.googleusercontent.com/external_content/untrusted_dlcp/researc

h.google.com/en/us/archive/bigtable-osdi06.pdf

• Featured Projects– Hbase http://hbase.apache.org/– Cassanda http://cassandra.apache.org/

Document Data Stores• Nested structures of hashes and their values

– A document can be• Simply a hash and its value• Hash and another document as its value• No limit in depth

– Very Flexible schema– Well-Indexed data– Works well with OOP (No impedance mismatch)– De-normalize as a best practice

• Use when– You don’t know much about the schema– The schema very likely to change

• Challenges– Complex Join-like queries– Self-referencing documents and circular dependencies

• Projects– MongoDB http://www.mongodb.org/– CouchDB http://couchdb.apache.org/

Graph Data Stores

• A graph– Perfect for highly interconnected data– Allows for explicit relationships– Fined graph grained-traversal– Very Flexible– Works well with OOP (No impedance mismatch)

• Use when– Your data looks like a graph and requires graph question– You are smart enough not to try this on another data store

• Challenges– Doesn’t scale-well horizontally

• Featured Projects– Neo4j http://neo4j.org/

• Use when– Your data Highly relational– There is a need to break data into small pieces and assemble it in

different ways– When consistence is king– Access patterns are unknown– Reporting

• Challenges– Doesn’t scale-well horizontally

• Featured Projects– Oracle http://www.oracle.com/index.html– Postgres http://www.postgresql.org/– Ms SQL Server http://dev.mysql.com/– MySQL http://www.mysql.com/

Relational Data Stores

How do you choose?

If It Doesn’t Fit, You Must Acquit!

• Data– Does it have a natural structure?– How it is connected to each other?– How is it distributed?– How much?

• Access Patterns– Reads/Writes ratio?– Uniform or random?

• CAP

Other Considerations

• Maturity• Stability• Maintainability• Durability• Cost• Tools• Familiarity

For Fairness’ Sake!

For Fairness’ Sake!

• Relational data stores did not fail us– They actually perform very well

• We failed ourselves– By using them as solutions for problems

they weren’t designed to solve to begin with

• Take any data store and you’ll get as much trouble

For Fairness’ Sake!

• You can’t expect– A flathead screwdriver to work on a Philips

as well as one with the matching Philips blade

– A crosshead screwdriver to work on flathead screw

Polyglot Persistence

Polyglot Persistence

• Enterprise application are complex and combine complex problems– Assumption that we should use one data store is

absurd– You can’t try to fit all in one model and expect no

problem• Polyglot Persistence

– To leverage multiple data storages, based on the way data is used by the application• Associated with a learning curve• Long term investment (More productive in the long-run)

– Leverage the strength of multiple data stores

Polyglot Persistence

• Example– MongoDB for the product catalog– Redis for shopping cart– DynamoDB for social profile info– Neo4j for the social graph– HBase for inbox and public feed messages– MySQL for payment and account info– Cassandra for audit and activity log

• Disclaimer: I’m not making any recommendation here.

NoSQL in the Cloud

NoSQL in the Cloud

• NoSQL as a commodity– Fully managed data stores (No

maintenance)– Elastic scaling– Cheap storage

• Featured:– Amazon AWS– Heroku Add-ons– CloudFoundry

As Promised!

The A’s the Q’s in the Abstract

• What does the rise of all these NoSQL mean to my enterprise?– I’m guessing a lot

• What is NoSQL to begin with?– Any non-relational data store

• Does it mean “NO SQL”?– No

• Could this be just another fad?– I don’t think so

The A’s the Q’s in the Abstract

• Is a good idea to be the future of my enterprise on these new exotic technologies and simply abandon proven mature RDBMS?– It’s up to you. I will say “No guts, no glory!”

• How scalable is scalable?– However much you need it to be

The A’s the Q’s in the Abstract

• Assuming that I am sold, how do I choose the one that fits my needs the best?– I’ll tell you if you hire me

• Is there a middle ground somewhere?– Polyglot Persistence

• What is this Polyglot Persistence I hear about?– It’s the middle ground

Any Other Questions?

Thank You All!

@PolymathicCoder