polyglot persistence in today’s data world · polyglot persistence in today’s data world...

33
Polyglot Persistence in Today’s Data World Kimberly Wilkins Principal Engineer –Databases ObjectRocket by Rackspace www.linkedin.com/in/wilkinskimberly, [email protected], @dba_denizen 1

Upload: others

Post on 27-Apr-2020

10 views

Category:

Documents


0 download

TRANSCRIPT

Polyglot Persistence in Today’s Data World

Kimberly Wilkins Principal Engineer –Databases ObjectRocket by Rackspace www.linkedin.com/in/wilkinskimberly, [email protected], @dba_denizen

1

Background •  18+ years working on various database platforms •  Mainly Oracle (databases, RAC, Enterprise Manager,

GoldenGate Replication, DataGuard, Database Vault, Exadata)

•  MongoDB NoSQL and Big Data techs at OR •  Industries – early online auctions, gaming, social

media •  Specialties – re-architect enterprise database

environments, infrastructure implementations, RAC, replication, system kernels, database storage

•  Re-engineered the database infrastructure for SWTOR – Star Wars The Old Republic MMO game

•  Supporting customers large and small and presenting for OR/Rackspace

www.objectrocket.com 2

Overview

•  What is Polyglot Persistence •  Why Do We Need It •  Data Technologies Available •  IoT Impacts – Data Store Techs •  3 V’s and CAP •  Techs Described •  Questions To Ask When Designing •  Do’s, Don’t’s, More •  Connectors •  Polyglot Morphing

www.objectrocket.com

3

What is Polyglot Persistence

www.objectrocket.com

•  BasedoffofpremiseofPolyglotProgramming•  WriteApplica:onsinamixoflanguages•  Takeadvantageofindividualstrengths•  Avoidtheirweaknesses

4

TheLanguagesofthe4thMillennium

Python 26%JavaScript 23%C# 20%Java15%C++ 13%

What is Polyglot Persistence…

www.objectrocket.com

•  DifferentDatabasetechsfordifferentusecasesandapplica:ons

•  RightDBtechnologyforthetask–usetherightDBMStoolforthejob

•  NoSQLMantra…•  Benefitsmustoutweighcosts

5

Questions To Ask

www.objectrocket.com

•  Doesitworkforspecificapplica:onneeds?•  Doesitworkforindividualneedsofyouroverallbusiness?

•  Howwellforallofyourapplica:ons?•  Howmuchcomplexitycanyousupport?•  Whatwilladdedcostbe–inhardware/soZware/personnel

6

Why Do We Need It Today

www.objectrocket.com 7

ExplosiveGrowthofBigData

Gartner-Globalinforma:onvolumesrisingover59%eachyearIDC-BigDatamarket"expectedtogrowfrom$3.2billionin2010to$16.9Bin2015.”ForYOU–careerop:ons,careerchangesFacebook500TBdailyInternet1ExabytecreatedperdayBrontobytes,Zetabytes-IoT

New Industries, New Sources, Much More Data

www.objectrocket.com 8

Internet of Things/Everything

www.objectrocket.com

•  Last2years-moredatathanallofhumanexistenceintheyearsprior

•  Mul:ply^^thanwecanimagine•  IoTgrowthisdrivingadop:on•  Upto20billiondevices–connect,log,automate•  CellPhones,SmartPhones,MobilePhones•  +>5billionpeoplecalling,tex:ng,twee:ng,andbrowsingworldwide

9

www.objectrocket.com 10

Using Multiple Data Store Technologies

www.objectrocket.com

IndividualRDBMSandNoSQLDatabaseTechnologiesbythemselvescannothandletoday’sdiverseapplica:onanddataneeds

11

Mixed Use and Source Example

www.objectrocket.com 12

The 3 V’s: Velocity Variety Volume

www.objectrocket.com 13

Cap Theorem

www.objectrocket.com 14

Possible Right Tools – Data Structure

www.objectrocket.com

•  Rela:onalDatabases–Oracle,MySQL,Postgres

•  KeyValueStores–Redis,Riak•  ColumnStores–Cassandra,Hbase•  DocumentStores–MongoDB,CouchDB•  Graphdatabases–Neo4J•  SearchEngines–Elas:c,Solr

15

Relational Databases

www.objectrocket.com 16

•  Usewhen:-  Yourdatasetisrela:onal-  Strongconsistencyneeded-ACID-  Accessparernsareunknown

•  But…Doesn’tscalewellhorizontally

•  UseCases:-  Duetoearlyadop:onandheavylegacyusage-everywhere-  Paymentsystems

Key/Valuestores

www.objectrocket.com 17

•  Usewhen:•  -Opera:onsarebasedonthekey•  -Dataisnothighlyrelated•  -BasicCRUDneeded

•  But…Complexqueriesarepainful

•  Usecases:-  SessionData-  UserProfile/Preferences-  Shoppingcarts

Key/Valuestores–cont’d

www.objectrocket.com 18

•  Abighashmapassocia:vearray-  Verysimple,Onekey<->Onevalue-  Veryfastread/write-  Nosecondaryindexes

•  {“Key”:(VRN)}=>{value(carfacts)}

•  [make#Ford•  {YYY0000}=>model#Fiesta•  year#2010]

DocumentStoreDatabases

www.objectrocket.com 19

•  Nestedstructuresofkeysandtheirvalues-  Veryflexibleschema(JSON,XML)-  OnekeyOnevaluebutvalueisvisibletoqueries-  Supportshierarchicaldata-  Supportssecondaryindexes

•  {“id”(VRN)}=>{“document”(carfacts)}

•  {“make”:Ford”,•  {YYY0000}=>“model”:”Fiesta”,•  “year”:2010}

DocumentStoreDatabases

www.objectrocket.com 20

•  Usewhen:-  Youdon’tknowmuchfortheschema–Unstructured,Semi-structured-  UnstructuredandHeterogeneousdata

•  But…•  Joinsandreferencesaretricky•  De-normaliza:onrequiresmorespace

•  Usecases:•  - ProductCatalog-  CMS-Adobe,etc.-  Eventloggingfromdifferentsources

ColumnarStoreDatabases

www.objectrocket.com 21

•  Usewhen:•  -BigData(Hugewritevolumes)•  -Versioning(Time-seriesdata)•  •  But…•  Knowyourstatementsinadvance•  Schemadesignisnottrivial

•  Usecases:-  Timeseriesdata-  Biddingpla�orms-  Playlists

DocumentvsColumnStoreDatabases

www.objectrocket.com 22

MongoDB(documentstore)bererwith:– When‘schema’willbefluid,changing– Withheavyreadloadsormixedloads

Cassandra(columnstore)bererwith:– When‘schema’willbefixed,NOTchanging,set– Withheavywriteloads–faster,appendonly

GraphDatabases–6DegreesofSepara:on..

www.objectrocket.com 23

•  Usewhen:-  Highlyinterconnecteddata-  Defineexplicitrela:onshipsandneedtraversalqueries•  But…Doesn’tscalewellhorizontally

•  UseCases:-  Where3rddegree(orhigher)rela:onshipneeded-  SocialMedia-Querieslikefriendofafriend,likesanddislikes

Questions to Ask when Picking a Datastore Technology

• Is Data Structured or Unstructured • How is the Data Connected - likes • Size of the data – now? Later? • Read/Write Ratio – heavy read/write/mixed? • Is Access Pattern Consistent All of the time –Monthly reporting, heavy queries in burst, insert batches

www.objectrocket.com 24

Questions to Ask when Picking a Datastore Technology

• Do I Need full DR? • Security/Encryption • Backups – immediate, delayed • Immediately available or delayed • Language Support • Driver Availability • Community Support • Connector Availability • Monitoring – plug-ins • Driver Availability

www.objectrocket.com 25

Challenges : Beware Polyglot Persistence • Not All Roses • Added Complexity • More systems to manage • More expertise required • Interactions • Network Connectivity • Timing Issues • Where to store which data for which apps/data stores • Additional Development Efforts, Upgrades, Testing

www.objectrocket.com 26

Challenges : Plan – Architect Ahead • Decide which to use for what • Possible Migration Headaches if you chose wrong on any level 1st • More systems to deploy • More hardware to provision • More configs to standardize • More Connections to manage • More troubleshooting – different techs • More personnel to support techs • More training for support

www.objectrocket.com 27

Do’s : Plan Ahead • KISS • Keep system streamlined • Keep system as lean as possible • Test and benchmark as much as possible • Stick to more proven use cases • Push boundaries or your customers will do so for you… ;-)

www.objectrocket.com 28

Connectors…

www.objectrocket.com

ObjectRocketcurrentlyhastheElas:ctoMongoDBConnectoravailable:

29

hrp://objectrocket.com/data-connectors/

OtherCommunityconnectorsarealsoavailable.

Polyglot Morphing…

www.objectrocket.com

WhatwehaveToday,willalmostsurelychangetomorrow…..

30

Current Pluggable Engines

www.objectrocket.com 31

Questions?

www.objectrocket.com 32

Thank you! Address: 401 Congress Ave Suite 1950 Austin, TX 78701 Support: 1-800-961-4454 Sales: 1-888-440-3242

www.objectrocket.com

33