polyglot persistence in today’s data world · polyglot persistence in today’s data world...
TRANSCRIPT
Polyglot Persistence in Today’s Data World
Kimberly Wilkins Principal Engineer –Databases ObjectRocket by Rackspace www.linkedin.com/in/wilkinskimberly, [email protected], @dba_denizen
1
Background • 18+ years working on various database platforms • Mainly Oracle (databases, RAC, Enterprise Manager,
GoldenGate Replication, DataGuard, Database Vault, Exadata)
• MongoDB NoSQL and Big Data techs at OR • Industries – early online auctions, gaming, social
media • Specialties – re-architect enterprise database
environments, infrastructure implementations, RAC, replication, system kernels, database storage
• Re-engineered the database infrastructure for SWTOR – Star Wars The Old Republic MMO game
• Supporting customers large and small and presenting for OR/Rackspace
www.objectrocket.com 2
Overview
• What is Polyglot Persistence • Why Do We Need It • Data Technologies Available • IoT Impacts – Data Store Techs • 3 V’s and CAP • Techs Described • Questions To Ask When Designing • Do’s, Don’t’s, More • Connectors • Polyglot Morphing
www.objectrocket.com
3
What is Polyglot Persistence
www.objectrocket.com
• BasedoffofpremiseofPolyglotProgramming• WriteApplica:onsinamixoflanguages• Takeadvantageofindividualstrengths• Avoidtheirweaknesses
4
TheLanguagesofthe4thMillennium
Python 26%JavaScript 23%C# 20%Java15%C++ 13%
What is Polyglot Persistence…
www.objectrocket.com
• DifferentDatabasetechsfordifferentusecasesandapplica:ons
• RightDBtechnologyforthetask–usetherightDBMStoolforthejob
• NoSQLMantra…• Benefitsmustoutweighcosts
5
Questions To Ask
www.objectrocket.com
• Doesitworkforspecificapplica:onneeds?• Doesitworkforindividualneedsofyouroverallbusiness?
• Howwellforallofyourapplica:ons?• Howmuchcomplexitycanyousupport?• Whatwilladdedcostbe–inhardware/soZware/personnel
6
Why Do We Need It Today
www.objectrocket.com 7
ExplosiveGrowthofBigData
Gartner-Globalinforma:onvolumesrisingover59%eachyearIDC-BigDatamarket"expectedtogrowfrom$3.2billionin2010to$16.9Bin2015.”ForYOU–careerop:ons,careerchangesFacebook500TBdailyInternet1ExabytecreatedperdayBrontobytes,Zetabytes-IoT
Internet of Things/Everything
www.objectrocket.com
• Last2years-moredatathanallofhumanexistenceintheyearsprior
• Mul:ply^^thanwecanimagine• IoTgrowthisdrivingadop:on• Upto20billiondevices–connect,log,automate• CellPhones,SmartPhones,MobilePhones• +>5billionpeoplecalling,tex:ng,twee:ng,andbrowsingworldwide
9
Using Multiple Data Store Technologies
www.objectrocket.com
IndividualRDBMSandNoSQLDatabaseTechnologiesbythemselvescannothandletoday’sdiverseapplica:onanddataneeds
11
Possible Right Tools – Data Structure
www.objectrocket.com
• Rela:onalDatabases–Oracle,MySQL,Postgres
• KeyValueStores–Redis,Riak• ColumnStores–Cassandra,Hbase• DocumentStores–MongoDB,CouchDB• Graphdatabases–Neo4J• SearchEngines–Elas:c,Solr
15
Relational Databases
www.objectrocket.com 16
• Usewhen:- Yourdatasetisrela:onal- Strongconsistencyneeded-ACID- Accessparernsareunknown
• But…Doesn’tscalewellhorizontally
• UseCases:- Duetoearlyadop:onandheavylegacyusage-everywhere- Paymentsystems
Key/Valuestores
www.objectrocket.com 17
• Usewhen:• -Opera:onsarebasedonthekey• -Dataisnothighlyrelated• -BasicCRUDneeded
• But…Complexqueriesarepainful
• Usecases:- SessionData- UserProfile/Preferences- Shoppingcarts
Key/Valuestores–cont’d
www.objectrocket.com 18
• Abighashmapassocia:vearray- Verysimple,Onekey<->Onevalue- Veryfastread/write- Nosecondaryindexes
• {“Key”:(VRN)}=>{value(carfacts)}
• [make#Ford• {YYY0000}=>model#Fiesta• year#2010]
DocumentStoreDatabases
www.objectrocket.com 19
• Nestedstructuresofkeysandtheirvalues- Veryflexibleschema(JSON,XML)- OnekeyOnevaluebutvalueisvisibletoqueries- Supportshierarchicaldata- Supportssecondaryindexes
• {“id”(VRN)}=>{“document”(carfacts)}
• {“make”:Ford”,• {YYY0000}=>“model”:”Fiesta”,• “year”:2010}
DocumentStoreDatabases
www.objectrocket.com 20
• Usewhen:- Youdon’tknowmuchfortheschema–Unstructured,Semi-structured- UnstructuredandHeterogeneousdata
• But…• Joinsandreferencesaretricky• De-normaliza:onrequiresmorespace
• Usecases:• - ProductCatalog- CMS-Adobe,etc.- Eventloggingfromdifferentsources
ColumnarStoreDatabases
www.objectrocket.com 21
• Usewhen:• -BigData(Hugewritevolumes)• -Versioning(Time-seriesdata)• • But…• Knowyourstatementsinadvance• Schemadesignisnottrivial
• Usecases:- Timeseriesdata- Biddingpla�orms- Playlists
DocumentvsColumnStoreDatabases
www.objectrocket.com 22
MongoDB(documentstore)bererwith:– When‘schema’willbefluid,changing– Withheavyreadloadsormixedloads
Cassandra(columnstore)bererwith:– When‘schema’willbefixed,NOTchanging,set– Withheavywriteloads–faster,appendonly
GraphDatabases–6DegreesofSepara:on..
www.objectrocket.com 23
• Usewhen:- Highlyinterconnecteddata- Defineexplicitrela:onshipsandneedtraversalqueries• But…Doesn’tscalewellhorizontally
• UseCases:- Where3rddegree(orhigher)rela:onshipneeded- SocialMedia-Querieslikefriendofafriend,likesanddislikes
Questions to Ask when Picking a Datastore Technology
• Is Data Structured or Unstructured • How is the Data Connected - likes • Size of the data – now? Later? • Read/Write Ratio – heavy read/write/mixed? • Is Access Pattern Consistent All of the time –Monthly reporting, heavy queries in burst, insert batches
www.objectrocket.com 24
Questions to Ask when Picking a Datastore Technology
• Do I Need full DR? • Security/Encryption • Backups – immediate, delayed • Immediately available or delayed • Language Support • Driver Availability • Community Support • Connector Availability • Monitoring – plug-ins • Driver Availability
www.objectrocket.com 25
Challenges : Beware Polyglot Persistence • Not All Roses • Added Complexity • More systems to manage • More expertise required • Interactions • Network Connectivity • Timing Issues • Where to store which data for which apps/data stores • Additional Development Efforts, Upgrades, Testing
www.objectrocket.com 26
Challenges : Plan – Architect Ahead • Decide which to use for what • Possible Migration Headaches if you chose wrong on any level 1st • More systems to deploy • More hardware to provision • More configs to standardize • More Connections to manage • More troubleshooting – different techs • More personnel to support techs • More training for support
www.objectrocket.com 27
Do’s : Plan Ahead • KISS • Keep system streamlined • Keep system as lean as possible • Test and benchmark as much as possible • Stick to more proven use cases • Push boundaries or your customers will do so for you… ;-)
www.objectrocket.com 28
Connectors…
www.objectrocket.com
ObjectRocketcurrentlyhastheElas:ctoMongoDBConnectoravailable:
29
hrp://objectrocket.com/data-connectors/
OtherCommunityconnectorsarealsoavailable.