alternative database technology in the cloud
DESCRIPTION
This presentation covers how open source technologies are being used to meet the specific needs of large scale problems on the Internet. No one solution meets all needs but open source provides a variety of solutions for different use cases.TRANSCRIPT
Community StackerBret Piatt
LinuxCon 2010Alternative Database Technologies for the Cloud
&
OPEN SOURCE ISDRIVING THE CLOUD
LINUX
XEN
KVM
APACHE
MEMCACHED
VARNISH
WORDPRESS
DRUPAL
MAGENTO
THE HARDEST PARTIS STILL THE DATABASE
TIMES HAVE CHANGED
OTHER THINGSHAVE ALSO CHANGED
THE WEB
WEB APPS ARE BUILT DIFFERENTLY
SERVER-SIDE PROCESSING
BURSTY LOADS, UNPREDICTABILITY
DISTRIBUTED USERS
SIMPLIFIED PROBLEM SET
SIMPLIFIED OPERATING ENVIRONMENT
WEB SCALE REQUIREMENTS
TERABYTES OF DATA, INCLUDING STRUCTURED DATA
MILLIONS TO BILLIONS OF WRITES EACH DAY
GLOBAL REPLICATION AND REDUNDANCY–EVENTUAL CONSISTANCY
COMMODITY HARDWARE–SCALE OUT NOT UP
CACHE EVERYTHING YOU READ
MORE CORES, MORE MEMORY, MORE MACHINES
RIAK
JSON AND REST–WEB GENERATION INTERFACE
MAP/REDUCE TERM MATCHING QUERIES
SCALE HORIZONTALLY WITH MULTIPLE MASTERS
EVENTUAL CONSISTENCY WITH VECTOR CLOCKS
WRITTEN IN ERLANG–CONCURRENT FUNCTIONAL PROGRAMMING
MORE CORES, MORE MEMORY, MORE MACHINES
HBASE
REST-FUL WEB SERVICE GATEWAY
DISTRIBUTED COLUMN STORE MODELED AFTER BIGTABLE
SCALE HORIZONTALLY WITH MULTIPLE MASTERS
ROLLING RESTART FOR CONFIG CHANGES AND MINOR UPGRADES
HIGH PERFORMANCE THRIFT GATEWAY
CASCADING, HIVE, AND PIG SOURCE AND SINK MODULES
MORE CORES, MORE MEMORY, MORE MACHINES
REDIS
API LIBRARIES AVAILABLE IN MANY LANGUAGES
MORE THAN JUST A PLAIN KEY-VALUE STORE
THE DATABASE IS IN MEMORY–RAM IS CHEAP
ASYNCHRONOUS FLUSH TO DISK
REPLICATION IS MASTER->SLAVE
MORE CORES, MORE MEMORY, MORE MACHINES
MONGODB
BSON INTERFACE DOCUMENT DATABASE
FULL INDEX SUPPORT
REPLICATION & HIGH AVAILABLITY ACROSS WANS
AUTO-SHARDING FOR HORIZONTAL SCALE
GRIDFS TO STORE LARGE FILES
MAP/REDUCE BATCH MANIPULATION
MORE CORES, MORE MEMORY, MORE MACHINES
Sources: http://www.businessinsider.com/how-we-use-mongodb-2009-11 -- andhttp://blog.boxedice.com/2010/02/28/notes-from-a-production-mongodb-deployment/
CASSANDRA
FULLY DISTRIBUTED
TRUE HORIZONTAL SCALEABILITY
REPLICATION–INCLUDING WIDE-AREA REPLICATION
PREDICTABLE, BLAZING WRITE PERFORMANCE
CONTROLLABLE READ/WRITE CONSISTENCY
MORE CORES, MORE MEMORY, MORE MACHINES
DRIZZLE
RELEASE EARLY, RELEASE OFTEN–NO 2-YEAR CYCLES
UTF8-THE WEB IS “WORLDWIDE” AFTER ALL
STRONG SQL INTERFACES MEAN STABILITY & INNOVATION
MODULAR ARCHITECTURE ALLOWS EXTENSION…
…OR PERFORMANCE BY REMOVING WHAT YOU DON’T NEED
MORE CORES, MORE MEMORY, MORE MACHINES
WHY OPEN MATTERS
LOCK-IN IS A REAL CONCERN IN THE CLOUD
OPEN SOURCE MAKES IT ACCESSIBLE TO YOU
TECHNOLOGY IS NOT A SUSTAINABLE, LONG-TERM ADVANTAGE
GAIN THE KNOWLEDGE OF A COMMUNITY
SHARE YOUR KNOWLEDGE WITH EVERYONE
TURNS COMPETITION INTO COOPERATION
THERE IS NOSILVER BULLET
MASSIVE SCALESPECIFIC PROBLEMS
REQUIRE OPTIMIZEDSPECIFIC SOLUTIONS
HOW SHOULDI CHOOSE?
Source: http://blog.mozilla.com/data/2010/05/18/riak-and-cassandra-and-hbase-oh-my/
SCALABILITY
ELASTICITY
RELIABILITY
STORAGE
DATA ANALYSIS
COST
SECURITY
EXTENSIBILITY
DR / MIGRATION
ANALYZE THEN DECIDE
