version 15svoboda/courses/192... · key-valuestores datamodel • themostsimplenosqldatabasetype...
TRANSCRIPT
![Page 1: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/1.jpg)
MDK:Modern Database Conceptsh p://www.ksi.mff.cuni.cz/~svoboda/courses/192-MDK/
Lecture 8
Key-Value Stores: RiakKVMar n [email protected]
25. 4. 2020
Charles University, Faculty of Mathema cs and PhysicsOTH Regensburg, Faculty of Computer Science and Mathema cs
![Page 2: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/2.jpg)
Lecture OutlineKey-value stores• Introduc on
RiakKV• Data model• HTTP interface• CRUD opera ons• Links and Link walking• Data types• Search 2.0• Internal details
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 2
![Page 3: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/3.jpg)
Key-Value StoresData model• The most simple NoSQL database type
Works as a simple hash table (mapping)• Key-value pairs
Key (id, iden fier, primary key)Value: binary object, black box for the database system
Query pa erns• Create, update or remove value for a given key• Get value for a given key
Characteris cs• Simple model⇒ great performance, easily scaled, …• Simple model⇒ not for complex queries nor complex data
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 3
![Page 4: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/4.jpg)
Key ManagementHow the keys should actually be designed?• Real-world iden fiers
E.g. e-mail addresses, login names, …• Automa cally generated values
Auto-increment integers– Not suitable in peer-to-peer architectures!
Complex keys– Mul ple components / combina ons of
me stamps, cluster node iden fiers, …– Used in prac ce instead
Prefixes describing en ty types are o en used as well• E.g. movie_medvidek, movie_223123, …
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 4
![Page 5: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/5.jpg)
Query Pa ernsBasic CRUD opera ons• Only when a key is provided• ⇒ knowledge of the keys is essen al
It might even be difficult for a par cular database systemto provide a list of all the available keys!
Accessing the contents of the value part is not possible in general• But we could instruct the database how to parse the values• … so that we can index them based on certain search criteria
Batch / sequen al processing• MapReduce
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 5
![Page 6: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/6.jpg)
Other Func onalityExpira on of key-value pairs• Objects are automa cally removed from the databasea er a certain interval of me
• Useful for user sessions, shopping carts etc.Links between key-value pairs• Values can be mutually interconnected via links• These links can be traversed when querying
Collec ons of values• Not only ordinary values can be stored, but also theircollec ons (e.g. ordered lists, unordered sets, …)
Par cular func onality always depends on the store we use!
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 6
![Page 7: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/7.jpg)
Riak Key-Value Store
![Page 8: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/8.jpg)
RiakKVKey-value store• h p://basho.com/products/riak-kv/• Features
Open source, incremental scalability, high availability,opera onal simplicity, decentralized design, automa c datadistribu on, advanced replica on, fault tolerance, …
• Developed by Basho Technologies• Implemented in Erlang
General-purpose, concurrent, garbage-collected programminglanguage and run me system
• Opera ng system: Linux, Mac OS X, … (not Windows)• Ini al release in 2009
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 8
![Page 9: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/9.jpg)
Data ModelRiak database system structure
Instance (→ bucket types)→ buckets→ objects
• Bucket = collec on of objects (logical, not physical collec on)Various proper es are set at the level of buckets
– E.g. default replica on factor, read / write quora, …• Object = key-value pair
Key is a Unicode string– Unique within a bucket
Value can be anything (text, binary object, image, …)Each object is also associated with metadata
– E.g. its content type (text/plain, image/jpeg, …),– and other internal metadata as well
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 9
![Page 10: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/10.jpg)
Data ModelDesign Ques ons
How buckets and objects should be modeled?• Buckets with objects of a single en ty type
E.g. one bucket for actors, one for movies,each actor and movie has its own object
• Buckets with objects of various en ty typesE.g. one bucket for both actors and movies,each actor and movie has its own object once againStructured keys might then help
– E.g. actor_trojan, movie_medvidek• Buckets with complex objects containing various data
E.g. one object for all the actors, one for all the movies
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 10
![Page 11: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/11.jpg)
Riak Usage: QueryingBasic CRUD opera ons• Create, Read, Update, and Delete• Based on a key look-up
Extended func onality• Links – rela onships between objects and their traversal• Search 2.0 – full-text queries accessing values of objects• MapReduce• …
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 11
![Page 12: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/12.jpg)
Riak Usage: APIApplica on interfaces• HTTP API
All the user requests are submi ed as HTTP requests withappropriately selected / constructedmethods, URLs, headers,and data
• Protocol Buffers API• Erlang API
Client libraries for a variety of programming languages• Official: Java, Ruby, Python, C#, PHP, …• Community: C, C++, Haskell, Perl, Python, Scala, …
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 12
![Page 13: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/13.jpg)
Riak Usage: HTTP APIcURL tool• Allows to transfer data from / to a server using HTTP
(or other supported protocols)Op ons• -X command, --request command
HTTP request method to be used (GET, …)• -d data, --data data
Data to be sent to the server (implies the POST method)• -H header, --header header
Extra headers to be included when sending the request• -i, --include
Prints both headers and (not just) body of a response
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 13
![Page 14: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/14.jpg)
Basic Opera ons
![Page 15: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/15.jpg)
CRUD Opera onsBasic opera ons on objects• Create: POST or PUT methods
Inserts a key-value pair into a given bucketKey is specified manually, or will be generated automa cally
• Read: GET methodRetrieves a key-value pair from a given bucket
• Update: PUT methodUpdates a key-value pair in a given bucket
• Delete: DELETE methodRemoves a key-value pair from a given bucket
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 15
![Page 16: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/16.jpg)
CRUD Opera onsURL pa ern of HTTP requests for all the CRUD opera ons
// bucketsbuckets // bucketbucket // keyskeys // keykey
?? parameterparameter == valuevalue
&&
Op onal parameters (depending on the opera on)• r, w: read / write quorum to be a ained• …
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 16
![Page 17: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/17.jpg)
CRUD Opera onsCreate and Update
Inserts / updates a key-value pair in a given bucket• PUTmethod
Should be used when a key is specified explicitlyTransparently inserts / updates (replaces) a given object
• POSTmethodWhen a key is to be generated automa callyAlways inserts a new object
• Buckets are created transparently whenever neededExample
curl -i -X PUT-H 'Content-Type: text/plain'-d 'Ivan Trojan, 1964'http://localhost:8098/buckets/actors/keys/trojan
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 17
![Page 18: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/18.jpg)
CRUD Opera onsRead
Retrieves a key-value pair from a given bucket• Method: GET
ExampleRequest
curl -i -X GEThttp://localhost:8098/buckets/actors/keys/trojan
Response...Content-Type: text/plain...
Ivan Trojan, 1964
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 18
![Page 19: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/19.jpg)
CRUD Opera onsDelete
Removes a key-value pair from a given bucket• Method: DELETE• If a given object does not exist, it does not ma er
Examplecurl -i -X DELETE
http://localhost:8098/buckets/actors/keys/trojan
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 19
![Page 20: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/20.jpg)
Bucket Opera onsLists all the buckets (buckets with at least one object)
// bucketsbuckets ?? bucketsbuckets == truetrue
curl -i -X GET http://localhost:8098/buckets?buckets=true
Content-Type: application/json
{"buckets" : [ "actors", "movies" ]
}
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 20
![Page 21: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/21.jpg)
Bucket Opera onsLists all the keys within a given bucket• Not recommended to be used in produc on environmentssince it is a very expensive opera on
// bucketsbuckets // bucketbucket // keyskeys ?? keyskeys == truetrue
curl -i -X GET http://localhost:8098/buckets/actors/keys?keys=true
Content-Type: application/json
{"keys" : [ "trojan", "machacek", "schneiderova", "sverak" ]
}
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 21
![Page 22: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/22.jpg)
Bucket Opera onsSe ng and retrieval of bucket proper es• Proper es
n_val: replica on factorr, w, …: read / write quora and their alterna ves…
• RequestsGET / PUT method: retrieve / set bucket proper es
// bucketsbuckets // bucketbucket // propsprops
Example{
"props" : { "n_val" : 3, "w" : "all", "r" : 1 }}
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 22
![Page 23: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/23.jpg)
Links and Link Walking
![Page 24: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/24.jpg)
Links and Link WalkingLinks• Links are metadata that establish one-way rela onshipsbetween pairs of objects
Act as lightweight pointers between individual key-value pairsI.e. represent and extension to the pure key-value data model
• Each link…is defined within the source objectis associated with a tag (sort of link type)can be traversed in a given direc on onlymay connect objects even from different buckets
• Mul ple links can lead from / to a given objectLink walking• New way of querying – naviga on between objects using links
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 24
![Page 25: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/25.jpg)
LinksHow are links defined?• Special Link header is used for this purpose• Mul ple link headers can be provided,or equivalently mul ple links within one header
LinkLink :: << targettarget >> ;; riaktagriaktag == "" tagtag ""
,,
Examplecurl -i -X PUT
-H 'Content-Type: text/plain'-H 'Link: </buckets/actors/keys/trojan>; riaktag="tactor"'-H 'Link: </buckets/actors/keys/machacek>; riaktag="tactor"'-d 'Medvídek, 2007'http://localhost:8098/buckets/movies/keys/medvidek
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 25
![Page 26: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/26.jpg)
Link WalkingHow can links be traversed?• Standard GET requests with link traversal descrip on
Exactly one object where the traversal is ini ated– Accessed in a standard way
Single or mul ple naviga onal steps then follow
// bucketsbuckets // bucketbucket // keyskeys // keykey
// bucketbucket
__
,, tagtag
__
,, 11
00
__
//
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 26
![Page 27: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/27.jpg)
Link WalkingParameters of naviga on steps• Bucket
Only objects from a certain target bucket are selected_ when not limited to any par cular bucket
• TagOnly links of a given tag are considered_ when not limited to any par cular tag
• Keep1 when the discovered objects should be included in the result0 otherwise_means 1 for the very last step, 0 for all the other preceding
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 27
![Page 28: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/28.jpg)
Link WalkingExamples
Actors who played inMedvídekmovie
curl -i -X GEThttp://localhost:8098/buckets/movies/keys/medvidek
/actors,tactor,1
Content-Type: multipart/mixed; boundary=...
Movies in which appeared actors fromMedvídekmovie(assuming that the corresponding actor→movie links also exist)
curl -i -X GEThttp://localhost:8098/buckets/movies/keys/medvidek
/actors,tactor,0/movies,tmovie,1
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 28
![Page 29: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/29.jpg)
Data Types
![Page 30: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/30.jpg)
Data TypesMo va on• Riak began as a pure key-value store
I.e. was completely agnos c toward the contents of values• However, if availability is preferred to consistency,mutually conflic ng replicas might exist
Such conflicts can be resolved at the applica on level,but this is o en (only too) difficult for the developers
• And so the concept of Riak Data Types was introducedWhen used (it is not compulsory),Riak is able to resolve conflicts automa cally
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 30
![Page 31: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/31.jpg)
Data TypesConvergent Replicated Data Types (CRDT)• Generic concept• Various types for several common scenarios• Specific conflict resolu on rules (convergence rules)
Available data types• Register, flag
Can only be used embedded in maps• Counter, set, and map
Can be used embedded in mapsas well as directly at the bucket level
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 31
![Page 32: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/32.jpg)
Data TypesRegister• Allows to store any binary value (e.g. string, …)• Convergence rule: the most chronologically recent value wins
Flag• Boolean values: enable (true), and disable (false)• Convergence rule: enable wins over disable
Counter• Opera ons: increment / decrement by a given integer value• Convergence rule: all requested increments and decrementsare eventually applied
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 32
![Page 33: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/33.jpg)
Data TypesSet• Collec on of unique binary values• Opera ons: addi on / removal of one / mul ple elements• Convergence rule: addi on wins over removal of elements
Map• Collec on of fields with embedded elements of any data type(including other nested maps)
• Opera ons: addi on / removal of an element• Convergence rule: addi on / update wins over removal
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 33
![Page 34: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/34.jpg)
Search 2.0
![Page 35: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/35.jpg)
Search 2.0Riak Search 2.0 (Yokozuna)• Full-text search over object values• Uses Apache Solr
Distributed, scalable, failure tolerant, real- me search pla ormHow does it work?• Indexa on
Riak object extractor−−−−−→ Solr document schema−−−−→ Solr index• Querying
Riak search query→ Solr search query→ Solr response:list of bucket-key pairs→ Riak response: list of objects
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 35
![Page 36: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/36.jpg)
Search 2.0: ExtractorsExtractor• Parses the object value and produces fields to be indexed• Chosen automa cally based on a MIME type
Available extractors• Common predefined extractors
Plain text, XML, JSON, noop (unknown content type)• Built-in extractors for Riak Data Types
Counter, map, set• User-defined custom extractors
Implemented in Erlang, registered with Riak
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 36
![Page 37: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/37.jpg)
Search 2.0: ExtractorsPlain text extractor (text/plain)• Single field with the whole content is extracted
ExampleInput Riak object
Ivan Trojan, 1964
Output Solr document[
{ text, <<"Ivan Trojan, 1964">> }]
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 37
![Page 38: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/38.jpg)
Search 2.0: ExtractorsXML extractor (text/xml, application/xml)• One field is created for each element and a ribute
Only fields with type suffixes are consideredE.g. _s for string, _i for integer, _b for boolean, …Dot nota on is used to compose fla en names of nested items
ExampleInput Riak object / Output Solr document
<?xml version="1.0" encoding="UTF-8" ?><actor year_i="1964">
<name_s>Ivan Trojan</name_s><actor>
[{ <<"actor.name_s">>, <<"Ivan Trojan">> },{ <<"actor.@year_i">>, <<"1964">> }
]
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 38
![Page 39: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/39.jpg)
Search 2.0: ExtractorsJSON extractor (application/json)• Similar principles as for XML documents are applied
ExampleInput Riak object
{name_s : "Ivan Trojan",year_i : 1964
}
Output Solr document[
{ <<"name_s">>, <<"Ivan Trojan">> },{ <<"year_i">>, <<"1964">> }
]
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 39
![Page 40: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/40.jpg)
Search 2.0: Indexa onSolr document• Automa cally extracted fields + a few auxiliary fields such as:
_yz_rb (bucket name), _yz_rk (key), …Solr schema• Describes how fields are indexed within Solr
Values of fields are analyzed and split into termsTerms are normalized, stop words removed…Triples (token, field, document) are produced and indexed
• Default schema available (_yz_default)Suitable for debugging,but custom schemas should be used in produc on
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 40
![Page 41: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/41.jpg)
Search 2.0: Index Crea onHow is index created?• Index must be created first,then associated with a single bucket
Examplecurl -i -X PUT
-H 'Content-Type: application/json'-d '{ "schema" : "_yz_default" }'http://localhost:8098/search/index/iactors
curl -i -X PUThttp://localhost:8098/search/index/iactors
curl -i -X PUT-H 'Content-Type: application/json'-d '{ "props" : { "search_index" : "iactors" } }'http://localhost:8098/buckets/actors/props
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 41
![Page 42: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/42.jpg)
Search 2.0: Index UsageSearch queries• Parameters
q – search query (correctly encoded)– Individual search criteria
wt – response write– Query result format
start / rows – pagina on of matching objects…
// searchsearch // queryquery // indexindex ?? parameterparameter == valuevalue
&&
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 42
![Page 43: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/43.jpg)
Search 2.0: Index UsageAvailable search func onality• Wildcards
E.g. name:Iva*, name:Iva?• Range queries
E.g. year:[2010 TO *]• Logical connec ves and parentheses
AND, OR, NOT• Proximity searches• …
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 43
![Page 44: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/44.jpg)
Internal Details
![Page 45: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/45.jpg)
ArchitectureSharding + peer-to-peer replica on architecture• Any node can serve any read or write user request• Physical nodes run (several) virtual nodes (vnodes)
Nodes can be added and removed from the cluster dynamically• Gossip protocol
Each node periodically sends its current view of the cluster,its state and changes, bucket proper es, …
CAP proper es• AP system: availability + par on tolerance
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 45
![Page 46: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/46.jpg)
ConsistencyBASE principles• Availability is preferred to consistency• Default proper es of buckets
n_val: replica on factorr: read quorumw: write quorum (node par cipa on is sufficient)dw: write quorum (write to durable storage is required)
• Specific op ons of requests override the bucket proper esStrong consistency can be achieved• When quora set carefully, i.e.:
w > n_val/2 for write quorumr > n_val− w for read quorum
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 46
![Page 47: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/47.jpg)
Causal ContextConflic ng replicas are unavoidable (with eventual consistency)⇒ how are they resolved?• Causal context = auxiliary data and mechanisms that arenecessary in order to resolve the conflicts
• Low-level techniquesTimestamps, vectors clocks, do ed version vectorsThey can be used to resolve conflicts automa cally
– Might fail, then we must make the choice by ourselvesOr we can resolve the conflictsmanually
– Siblings then need to be enabled (allow_mult)= mul ple versions of object values
• User-friendly CRDT data types with built in resolu onRegister, flag, counter, set, map
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 47
![Page 48: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/48.jpg)
Causal ContextVector clocks• Mechanism for tracking object update causalityin terms of logical me (not chronological me)
• Each node has its own logical clock (integer counter)Ini ally equal to 0Incremented by 1 whenever any event takes place
• Vector clock = vector of logical clocks of all the nodesEach node maintains its local copy of this vectorWhenever a message is sent, the local vector is sent as wellWhenever a message is received, the local vector is updated
– Maximal value for each individual node clock is taken
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 48
![Page 49: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/49.jpg)
Vector Clocks
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 49
![Page 50: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/50.jpg)
Vector Clocks
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 50
![Page 51: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/51.jpg)
Vector Clocks
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 51
![Page 52: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/52.jpg)
Vector Clocks
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 52
![Page 53: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/53.jpg)
Vector Clocks
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 53
![Page 54: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/54.jpg)
Vector Clocks
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 54
![Page 55: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/55.jpg)
Riak RingReplica placement strategy• Consistent hashing func on
Consistent = does not change when cluster changesDomain: pairs of a bucket name and object keyRange: 160-bit integer space = Riak Ring
Riak Ring• The whole ring is split into equally-sized disjoint par ons
Physical nodes are mutually interleaved⇒ reshuffling when cluster changes is less demanding
• Each virtual node is responsible for exactly one par onExample• Cluster with 4 physical nodes, each running 8 virtual nodes• I.e. 32 par ons altogether
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 55
![Page 56: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/56.jpg)
Riak Ring
Source: h p://docs.basho.com/
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 56
![Page 57: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/57.jpg)
Riak RingReplica placement strategy• The first replica…
Its loca on is directly determined by the hash func on• All the remaining replicas…
Placed to the consecu ve par ons in a clockwise direc onWhat if a virtual node is failing?• Hinted handoff
Failing nodes are simply skipped,neighboring nodes temporarily take responsibilityWhen resolved, replicas are handed off to the proper loca ons
• Mo va on: high availability
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 57
![Page 58: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/58.jpg)
Request HandlingRead and write requests can be submi ed to any node• This nodes is called a coordina ng node• Hash func on is calculated, i.e. replica loca ons determined• Internal requests are sent to all the corresponding nodes• Then the coordina ng node waitsun l sufficient number of responses is received
• Result / failure is returned to the user
But what if the cluster changes?• The value of the hash func on does not change,
only the par ons and their mapping to virtual nodes change• However, the Ring knowledge a given node has might be obsolete!
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 58
![Page 59: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/59.jpg)
![Page 60: Version 15svoboda/courses/192... · Key-ValueStores Datamodel • ThemostsimpleNoSQLdatabasetype Worksasasimplehashtable(mapping) • Key-valuepairs Key(id,idenfier,primarykey) Value:binaryobject,blackboxforthedatabasesystem](https://reader036.vdocuments.net/reader036/viewer/2022063018/5fda8dacd5018c2e6b106df3/html5/thumbnails/60.jpg)
Lecture ConclusionRiakKV• Highly available distributed key-value store• Sharding with peer-to-peer replica on architecture• Riak Ring with consistent hashing for replica placement
Query func onality• Basic CRUD opera ons• Link walking• Search 2.0 full-text based on Apache Solr
MDK: Modern Database Concepts | Lecture 8: Key-Value Stores: RiakKV | 25. 4. 2020 60