![Page 1: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/1.jpg)
Bigtable15799 - Advanced Topics in DB
October 2, 2013
stolen slides from Jeff Dean (a lot) and Edward Yoon (one)
![Page 2: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/2.jpg)
![Page 3: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/3.jpg)
![Page 4: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/4.jpg)
Comparing with Dynamo
I Data Model: Table vs. Key-ValueI Consistency: Atomic row mutation vs. record-at-a-time and
eventual consistencyI How to run: Centralized management vs. each application run
its own instanceI Access control vs. NoI Focus: Easy to use vs. availability
![Page 5: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/5.jpg)
Outline
Topics will be covered today:
I Data Model and APII System OverviewsI Implementation of Tablet ServersI Current State of Bigtable
Things are ignored
I Refinement to improved the performanceI Experiments
![Page 6: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/6.jpg)
![Page 7: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/7.jpg)
![Page 8: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/8.jpg)
![Page 9: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/9.jpg)
![Page 10: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/10.jpg)
![Page 11: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/11.jpg)
![Page 12: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/12.jpg)
![Page 13: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/13.jpg)
API
I Atomic single row mutation
RowMutation r1(T, "com.cnn.www");
r1.Set("anchor:www.c-span.org", "CNN");
r1.Delete("anchor:www.abc.com");
Operation op;
Apply (&op, &r1);
I scanning cells by rows, column, and timestamp
Scanner scanner(T);
ScanStream *stream;
stream = scanner.FetchColumnFamily("anchor");
stream ->SetReturnAllVersions ();
scanner.Lookup("com.cnn.www");
for (; !stream ->Done (); stream ->Next ()) {}
I server-side scripts: Sawzall
I integration with other products: Mapreduce, Pregel,Parameter Server, ....
![Page 14: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/14.jpg)
![Page 15: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/15.jpg)
![Page 16: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/16.jpg)
![Page 17: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/17.jpg)
![Page 18: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/18.jpg)
![Page 19: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/19.jpg)
![Page 20: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/20.jpg)
![Page 21: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/21.jpg)
![Page 22: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/22.jpg)
![Page 23: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/23.jpg)
![Page 24: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/24.jpg)
![Page 25: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/25.jpg)
![Page 26: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/26.jpg)
![Page 27: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/27.jpg)
![Page 28: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/28.jpg)
![Page 29: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/29.jpg)
![Page 30: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/30.jpg)
![Page 31: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/31.jpg)
Find Tablet Location
I Three-level hierarchy analogous to a B+ treeI 1st level: bootstrapped from chubbyI 2nd level: use META0 tablet to find the owner of appropriate
META 1 tabletsI 3rd level: META1 table holds locations of all other tablets
![Page 32: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/32.jpg)
Tablet Assignment
I Master assigns tablets to tablet servers
![Page 33: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/33.jpg)
Store a tablet
I A tablet is maintained by one tablet server.
I A tablet consists of serveral SSTable blocks with an index tostore the first and last key of the block, and stored in GFS
index
SSTable64Kb
SSTable64Kb
SSTable64Kb
![Page 34: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/34.jpg)
Tablet Serving
I Updates are stored in tablet log andI new ones in memtable (in memory)I old ones in SSTables
I For read, first check memtable, then SSTables
![Page 35: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/35.jpg)
Compaction
minor compaction
I convert memtable into an new SSTable and write into diskI save memory
merging(major) compaction
I read several SSTable and memtable and merging into a few(exact one) SSTable
I save disk due to high compression rate, remove deleted entries
![Page 36: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/36.jpg)
![Page 37: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/37.jpg)
![Page 38: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/38.jpg)
![Page 39: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/39.jpg)
![Page 40: Mu Li muli@cs.cmu.edu October 2, 2013pavlo/courses/fall2013/static/slides/bigtable.pdfBigtable 15799 - Advanced Topics in DB Mu Li muli@cs.cmu.edu October 2, 2013 stolen slides from](https://reader034.vdocuments.net/reader034/viewer/2022042115/5e92fafee2cd850ed17443e9/html5/thumbnails/40.jpg)