the where and when of nosql platforms
DESCRIPTION
Logicworks presents Just Say No to SQL as part of the Digital Media Series, moderated by Mashable's own Christina Warren.TRANSCRIPT
![Page 1: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/1.jpg)
PRESENTS
![Page 2: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/2.jpg)
Moderator
Christina Warren,
Panelists
Harry Heymann, Head of Server Engineering at
Michael Bryzek, CTO & Founder at
Frank Weigel, Director of Products at
Matt Pfeil, VP of Customer Solutions and Co-Founder at
Dave Connors, VP of Operations at
Dwight Merriman, CEO at 10Gen at
![Page 3: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/3.jpg)
The Old World
Pros• Transactional integrity• Sequential nature
Cons• Expensive• Scale up typically not out
![Page 4: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/4.jpg)
The New World
![Page 5: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/5.jpg)
Why ??
• Huge volume of content• Distributed Infrastructure• Relaxed and agile• Throw the RD rules out the window• Speed of development• No DBA needed• Elastic scaling (out not up)• Major cost savings
![Page 6: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/6.jpg)
Platforms
•
•
•
![Page 7: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/7.jpg)
Companies
![Page 8: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/8.jpg)
Michael Bryzek CTO & Founder
![Page 9: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/9.jpg)
GILT GROUPE is an innovative e-commerce company offering highly coveted products and experiences at insider prices.
![Page 10: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/10.jpg)
One of the Most Innovating Companies“Gilt Groupe, the online shopping phenomenon”--Alexandra Kotur, Vogue
“The t-shirt arrived the very next day in a clean box with nicely folded black tissue paper and a friendly note from the founders thanking me for my purchase. They're thanking me? No, Gilt Groupe, thank you.”--Damien Nunes, GQ (on his first purchase)
“The French invented the online sample sale for designer merchandise, spawning numerous American competitors, but none boasts the pedigree of Gilt Groupe.”--Fortune, (from their story “Dotcom Divas”)
“People with a taste for high-end fashion items have turned Gilt into a guiltless addiction.”--Roy Furchgott, The New York Times
![Page 11: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/11.jpg)
10,000 foot view
![Page 12: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/12.jpg)
The Shopping Cart
![Page 13: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/13.jpg)
Harry HeymannHead of Server Engineering
![Page 14: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/14.jpg)
![Page 15: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/15.jpg)
mongodb: some numbers
• 8 clusters
o some sharded, some not
o some master/slave, some replica sets
• ~40 machines (68GB, m2.4xl on EC2)
• 2.3 billion records
• ~15k QPS
![Page 16: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/16.jpg)
Why MongoDB?
Biggest reason (by far): auto sharding:
• Started on a single SQL database. Eventually split to two nodes: 1 for check-ins (our biggest dataset), 1 for everything else.
• It was clear that check-ins would grow beyond what a single machine could handle.
• Major efficiency gains by outsourcing the development of the sharding layer to consultants at 10gen.
![Page 17: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/17.jpg)
Dave Connors, VP of Operations
![Page 18: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/18.jpg)
• Customers are Small Businesses• Email, Event, Survey & now Social Media• Over 450k paying customers
• Business model• Many customers pay as little as $15 a month• ~2 million database transactions per minute
• Business problem• Social Media up to 100 times more data• Challenge with our business model
Constant Contact
![Page 19: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/19.jpg)
Implementation
• Monitoring• Dev took lead with Munin
• Logging• Ops took lead with Scribed
• Roles & Responsibilities• DBA?• Still in progress…
Cultural Challenges: NoSQL
Traditional Roles
![Page 20: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/20.jpg)
• Switchable modes• Mirroring• Dial-able traffic
Managing Risk: Phased Rollout
![Page 21: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/21.jpg)
ConclusionBusiness Value Comparison
Traditional RDBMS NoSQL
Agility
System Standup 4 weeks 4 hours
Add Capacity 1 week 1 hour
Upgrade DB 1 month 3 days
Time to Market for App 9 months 3 months
Estimated 1st Year Cost $2.5M $250k
![Page 22: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/22.jpg)
Technologists
![Page 23: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/23.jpg)
Dwight Merriman CEO
![Page 24: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/24.jpg)
Past:RDBMS
one size fits all
![Page 25: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/25.jpg)
BI / Data warehousing
RDBMS
other segmentse.g. search
![Page 26: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/26.jpg)
RDBMS
other segmentse.g. search
BI / Data warehousingaster, greenplum, neteeza, teradata, vertica, hadoop
RDBMS
NoSQL
![Page 27: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/27.jpg)
RDBMS
other segmentse.g. search
BI / Data warehousingaster, greenplum, neteeza, teradata, vertica, hadoop
RDBMS
NoSQL
• a few gigantic queries• visual client tools important -> a
need for SQL• scales horizontally (better ones)
• very complex transactional semantics• legacy projects• projects requiring SQL
• agile• programmer friendly data modle• horizontall scalable• works for operational data (lots of small
reads and writes)
![Page 28: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/28.jpg)
Frank Weigel, Director of Products
![Page 29: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/29.jpg)
moxi
11211 11210
Memcached Managed Cacheprotocol listener/sender
engine interface
memcapable 1.0 memcapable 2.0
21100 – 2119943698091
httpRE
ST m
anag
emen
t API
/Web
UI
Hea
rtbe
at
Proc
ess
mon
itor
Glo
bal s
ingl
eton
sup
ervi
sor
Confi
gura
tion
man
ager
on each node
Erlang/OTP
Reba
lanc
e or
ches
trat
or
Nod
e he
alth
mon
itor
one per cluster
vBuc
ket s
tate
and
repl
icati
on m
anag
er
HTTP distributed erlangerlang port mapper
Couchbase Server Architecture
Couchbase Storage EngineDat
a M
anag
er
Clus
ter M
anag
er
![Page 30: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/30.jpg)
• Example of online migration of existing application.
• First migrated large or slow performing tables and frequently updated fields from MySQL to Couchbase
: Migrating to Couchbase
memcachedprotocol listener/sender
engine interface
Couchbase Storage Engine
TAP
Web Server
Apache + PHP
Client-side Moxi
MySQL
memcachedprotocol listener/sender
engine interface
Couchbase Storage Engine
TAP
memcachedprotocol listener/sender
engine interface
Couchbase Storage Engine
TAP
Web Server
Apache + PHP
Client-side Moxi
![Page 31: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/31.jpg)
Matt Pfeil, VP of Customer Solutions
![Page 32: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/32.jpg)
Architecture
![Page 33: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/33.jpg)
Better technology
• Multi-master, multi-DC• Linearly scalable• Larger-than-memory datasets• Best-in-class performance (not just writes!)• Fully durable• Integrated caching• Tuneable consistency
![Page 34: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/34.jpg)
A performance retrospective
![Page 35: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/35.jpg)
Credit: Nathan Hurst, 2010
![Page 36: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/36.jpg)
QUESTIONS FROM THE AUDIENCE
![Page 37: The Where and When of NoSQL Platforms](https://reader036.vdocuments.net/reader036/viewer/2022062616/54922d99ac795963288b46bb/html5/thumbnails/37.jpg)
THANK YOU FOR JOINING US!We look forward to seeing you at our upcoming events