sharding for the masses
DESCRIPTION
An introduction to Spider, a MySQL storage engine for transparent sharding, and the vertical partitioning engineTRANSCRIPT
![Page 1: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/1.jpg)
Sharding for the massesIntroducing the Spider
storage engine, and more
Giuseppe MaxiaKentoku Shiba
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Tuesday, 13 April 2010
![Page 2: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/2.jpg)
What is sharding?
"shard" is a piece of broken ceramic or glass
"Sharding" means breaking a database to pieces
Tuesday, 13 April 2010
![Page 3: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/3.jpg)
WHY SHARDING?
Tuesday, 13 April 2010
![Page 4: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/4.jpg)
WHY SHARDING?SCALING
SCALING
SCALING
SCALING
SCALINGTuesday, 13 April 2010
![Page 5: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/5.jpg)
Scaling: the problem
You start with one server
Too much data
Too much traffic
Now what?
Tuesday, 13 April 2010
![Page 6: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/6.jpg)
Scaling: the solution
The MySQL way
Also known as the Yahoo and Google way
Replication
Tuesday, 13 April 2010
![Page 7: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/7.jpg)
Replication: how it worksMaster
client slaveslave
slave
WRITE
READload
balancer
Tuesday, 13 April 2010
![Page 8: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/8.jpg)
Replication: how it scalesMaster
client slave
WRITE
READ load balancer
clientclientclientclientclient
slaveslaveslaveslaveslaveslaveslave
slaveslaveslaveslave
Tuesday, 13 April 2010
![Page 9: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/9.jpg)
Replication: how it chokesMaster
client slave
WRITE
READ load balancer
clientclientclientclientclient
slaveslaveslaveslaveslaveslaveslave
slaveslaveslaveslave
> data
Tuesday, 13 April 2010
![Page 10: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/10.jpg)
MasterNruleN
MasterNruleN
MasterNruleN
Homemade sharding
Master1client
WRITEREAD
application logic
rule1
Master2rule2
MasterNruleN
Tuesday, 13 April 2010
![Page 11: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/11.jpg)
MasterNruleN
MasterNruleN
MasterNruleN
How homemade sharding breaks
Master1client
WRITEREAD
application logic
rule1
Master2rule2
MasterNruleN
Tuesday, 13 April 2010
![Page 12: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/12.jpg)
MasterNruleN
MasterNruleN
MasterNruleN
How homemade sharding breaks
Master1client
WRITEREAD
application logic
rule1
Master2rule2
MasterNruleN
Tuesday, 13 April 2010
![Page 13: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/13.jpg)
MasterNruleN
MasterNruleN
MasterNruleN
How homemade sharding breaks
Master1
client
WRITEREAD
application logic
rule1Master2
rule2MasterN
ruleN
Tuesday, 13 April 2010
![Page 14: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/14.jpg)
MasterNruleN
MasterNruleN
MasterNruleN
How homemade sharding breaks (2)
Master1client
WRITEREAD
application logic
rule1
Master2rule2
MasterNruleN
rule1rule1
rule2
ruleNruleNruleNruleN
rule2rule2rule2
rule1rule1
Tuesday, 13 April 2010
![Page 15: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/15.jpg)
MasterNruleN
MasterNruleN
MasterNruleN
How homemade sharding breaks (2)
Master1client
WRITEREAD
application logic
rule1
Master2rule2
MasterNruleN
rule1rule1
rule2
ruleNruleNruleNruleN
rule2rule2rule2
rule1rule1
Tuesday, 13 April 2010
![Page 16: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/16.jpg)
The quest for magic sharding
Tuesday, 13 April 2010
![Page 17: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/17.jpg)
The quest for magic sharding
MySQL Proxy
HSCALE
SpockProxy
Tuesday, 13 April 2010
![Page 18: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/18.jpg)
The quest for magic sharding
MySQL Proxy
HSCALE
SpockProxy
DON'T SCALE (SPoF)
Tuesday, 13 April 2010
![Page 19: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/19.jpg)
Horizontal partitioning
Tuesday, 13 April 2010
![Page 20: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/20.jpg)
Introducing Spider
A MySQL storage engine
Developed by Kentoku Shiba
Built on top of the partitions engine
Associates a partition with a remote server
Transparent to user
Easy to expand
Independent from application
Tuesday, 13 April 2010
![Page 21: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/21.jpg)
Note about partitions
A feature introduced in MySQL 5.1
Horizontal partitioning
Transparent to users
Increases insertion and selection performance
presentations
http://tinyurl.com/mysql-partition-tut
http://tinyurl.com/mysql-partition-perf
Tuesday, 13 April 2010
![Page 22: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/22.jpg)
host1MySQL server
with SPIDER
table employeestable employeestable employeespartition 1 year 1997 host2partition 2 year 1998 host3partition 3 year 1999 host4partition 4 year 2000 host5partition 5 year 2001 host6partition 6 year 2002 host7
host2 MySQL server without SPIDER
host3 MySQL server without SPIDER
host4 MySQL server without SPIDER
host5 MySQL server without SPIDER
host6 MySQL server without SPIDER
host7 MySQL server without SPIDER
Spider conceptual model
Tuesday, 13 April 2010
![Page 23: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/23.jpg)
host1MySQL server
with SPIDER
table employeestable employeestable employeespartition 1 year 1997 host2partition 2 year 1998 host3partition 3 year 1999 host4partition 4 year 2000 host5partition 5 year 2001 host6partition 6 year 2002 host7
host2 MySQL server without SPIDER
host3 MySQL server without SPIDER
host4 MySQL server without SPIDER
host5 MySQL server without SPIDER
host6 MySQL server without SPIDER
host7 MySQL server without SPIDER
Spider conceptual model
no data here
Tuesday, 13 April 2010
![Page 24: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/24.jpg)
host1MySQL server
with SPIDER
table employeestable employeestable employeespartition 1 year 1997 host2partition 2 year 1998 host3partition 3 year 1999 host4partition 4 year 2000 host5partition 5 year 2001 host6partition 6 year 2002 host7
host3 MySQL server without SPIDER
host4 MySQL server without SPIDER
host5 MySQL server without SPIDER
host6 MySQL server without SPIDER
host7 MySQL server without SPIDER
select * from employees where
date = '1998-01-01'
Tuesday, 13 April 2010
![Page 25: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/25.jpg)
host1MySQL server
with SPIDER
table employeestable employeestable employeespartition 1 year 1997 host2partition 2 year 1998 host3partition 3 year 1999 host4partition 4 year 2000 host5partition 5 year 2001 host6partition 6 year 2002 host7
host2 MySQL server without SPIDER
host3 MySQL server without SPIDER
host4 MySQL server without SPIDER
host5 MySQL server without SPIDER
host6 MySQL server without SPIDER
host7 MySQL server without SPIDER
select * from employees where
date = '1998-01-01'
Tuesday, 13 April 2010
![Page 26: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/26.jpg)
host1MySQL server
with SPIDER
table employeestable employeestable employeespartition 1 year 1997 host2partition 2 year 1998 host3partition 3 year 1999 host4partition 4 year 2000 host5partition 5 year 2001 host6partition 6 year 2002 host7
host2 MySQL server without SPIDER
host3 MySQL server without SPIDER
host4 MySQL server without SPIDER
host5 MySQL server without SPIDER
host6 MySQL server without SPIDER
host7 MySQL server without SPIDER
Tuesday, 13 April 2010
![Page 27: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/27.jpg)
host1MySQL server
with SPIDER
table employeestable employeestable employeespartition 1 year 1997 host2partition 2 year 1998 host3partition 3 year 1999 host4partition 4 year 2000 host5partition 5 year 2001 host6partition 6 year 2002 host7
host2 MySQL server without SPIDER
host3 MySQL server without SPIDER
host4 MySQL server without SPIDER
host5 MySQL server without SPIDER
host6 MySQL server without SPIDER
host7 MySQL server without SPIDER
select * from employees where date = '1998-01-01' limit 0,
9223372036854775807
Tuesday, 13 April 2010
![Page 28: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/28.jpg)
host1MySQL server
with SPIDER
table employeestable employeestable employeespartition 1 year 1997 host2partition 2 year 1998 host3partition 3 year 1999 host4partition 4 year 2000 host5partition 5 year 2001 host6partition 6 year 2002 host7
host2 MySQL server without SPIDER
host3 MySQL server without SPIDER
host4 MySQL server without SPIDER
host5 MySQL server without SPIDER
host6 MySQL server without SPIDER
host7 MySQL server without SPIDER
Tuesday, 13 April 2010
![Page 29: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/29.jpg)
host1
table employeestable employeestable employeespartition 1 year 1997 host2partition 2 year 1998 host3partition 3 year 1999 host4partition 4 year 2000 host5partition 5 year 2001 host6partition 6 year 2002 host7
host2 MySQL server without SPIDER
host3 MySQL server without SPIDER
host4 MySQL server without SPIDER
host5 MySQL server without SPIDER
host6 MySQL server without SPIDER
host7 MySQL server without SPIDER
host8
table employeestable employeestable employeespartition 1 year 1997 host2partition 2 year 1998 host3partition 3 year 1999 host4partition 4 year 2000 host5partition 5 year 2001 host6partition 6 year 2002 host7
MySQL server with SPIDER
MySQL server with SPIDER
Tuesday, 13 April 2010
![Page 30: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/30.jpg)
data data datadata data
Tuesday, 13 April 2010
![Page 31: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/31.jpg)
data data datadata data
Tuesday, 13 April 2010
![Page 32: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/32.jpg)
data data datadata data
Tuesday, 13 April 2010
![Page 33: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/33.jpg)
data data datadata data
Tuesday, 13 April 2010
![Page 34: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/34.jpg)
data data datadata data
Tuesday, 13 April 2010
![Page 35: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/35.jpg)
data data datadata data
client
Tuesday, 13 April 2010
![Page 36: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/36.jpg)
clientclientclient
data data datadata data
client
Tuesday, 13 April 2010
![Page 37: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/37.jpg)
data
data
HAdata
data
HAdata
data
HAdata
data
HAdata
data
HA
Tuesday, 13 April 2010
![Page 38: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/38.jpg)
INSTALLATION
Tuesday, 13 April 2010
![Page 39: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/39.jpg)
INSTALLATION (1)
Get the source code for MySQL 5.1.39
http://dev.mysql.com/downloads
Get the source code for Spider 2.5
http://launchpad.net/spiderformysql
Get the patch for condition pushdown
https://launchpad.net/partitionconditionpushdownformysql
Tuesday, 13 April 2010
![Page 40: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/40.jpg)
INSTALLATION (2)
Unpack MySQL source code
Unpack Spider source code and docs
Unpack the condition pushdown patch
mkdir spidercd spidertar -xzf mysql-5.1.39.tar.gztar -xzf spider-src-1.0-for-5.1.39.tgztar -xzf spider-doc-1.0-for-5.1.39.tgztar -xzf partition_cond_push-0.1-for-5.1.36.tgz
Tuesday, 13 April 2010
![Page 41: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/41.jpg)
/ l .' : __.....__..._ ____ / / \ _.-" "-. "" "-. (`-: .---: .--.' _....J. "-. """y \,.' \ __..--"" `+""--. `. : .'/ .-"""-. _. `. "-. `._.._ ; _.'.' .-j `. \ "-. "-._`. : / .-" : \ `-. `- "-. \ ; /.' ; :; ." \ `, :_:/ ::\ ;: ( / .-" .') ; ;-" ; "-. / ; .^. .' .' / .-" / .- : `. '. : .- / __.-j.'.' .-" /.---' / / `,\. .' "":' /-" .' \__.' : : ,\"" ; .' .' .-"" _J ; ; `. /.' _/ \.-" / "-: /"--.b-..-' .' ; / / ""-..' .--'.-'/ , : :`. : / : `-i" ,',_: _ \ : \ '._ :__; .'.-"; ; ; j `.l \ \ "-._ `" :_/ :_/ `.;\ "-._ :_"-._ "-. `. l "-. ) `. ""^--""^-. : \ "; \ : `._ ; / \ `._ ""---. / / _ `.--.__.' : : / ; :". \ ; ; : : ; `. `. / ; : ; : `. `. / /: ; : ; "-' :_.' ; ; ; : / / :_l `-'
STOP!What the hell is a
"condition pushdown"?
Tuesday, 13 April 2010
![Page 42: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/42.jpg)
MASTER
REMOTE SERVER
SELECT * FROM sometable WHERE col1 = 2
remote server
management
SELECT col1,col2,col3 FROM sometable LIMIT
0,9999999Without
condition pushdown
Tuesday, 13 April 2010
![Page 43: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/43.jpg)
MASTER
REMOTE SERVER
SELECT * FROM sometable WHERE col1 = 2
remote server
management
SELECT col1,col2,col3 FROM sometable WHERE col1 = 2 LIMIT 0,9999999With
condition pushdown
Tuesday, 13 April 2010
![Page 44: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/44.jpg)
INSTALLATION (3)
Move the spider directory into MySQL source code
mv spider mysql-5.1.39/storage
Tuesday, 13 April 2010
![Page 45: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/45.jpg)
INSTALLATION (4)
Apply the Spider patches to MySQL code
cd mysql-5.1.39patch -p2 < ../mysql-5.1.39.spider.diffpatch -p2 < \ ../mysql-5.1.36.partition_cond_push.diff
Tuesday, 13 April 2010
![Page 46: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/46.jpg)
INSTALLATION (5)
Compile MySQL code (see the docs for details)autoconfautomake./configure --enable-thread-safe-client \ --enable-local-infile \ --with-pic --with-fast-mutexes \ --with-client-ldflags=-static \ --with-mysqld-ldflags=-static --with-zlib-dir=bundled \ --with-big-tables --with-ssl --with-readline \ --with-embedded-server --with-partition \ --with-innodb --without-ndbcluster \ --without-archive-storage-engine \ --without-blackhole-storage-engine \ --with-csv-storage-engine \ --without-example-storage-engine \ --without-federated-storage-engine \ --with-extra-charsets=complex && make
Tuesday, 13 April 2010
![Page 47: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/47.jpg)
INSTALLATION (6)
create a binary tarball
./scripts/make_binary_distribution
Tuesday, 13 April 2010
![Page 48: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/48.jpg)
INSTALLATION (7)
Install manually in your main server
OR
use MySQL Sandbox
make_sandbox \ $PWD/mysql-5.1.39-osx10.5-i386.tar.gz \ --sandbox_directory=spider_main
Tuesday, 13 April 2010
![Page 49: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/49.jpg)
SETUP
Tuesday, 13 April 2010
![Page 50: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/50.jpg)
SETUP (1)
Get the SQL from the docs
or get it from my site
http://datacharmer.org/downloads/spider_setup.sql
Run it
cd $HOME/sandboxes/spider_mainwget http://datacharmer.org/downloads/spider_setup.sql./use < spider_setup.sql
Tuesday, 13 April 2010
![Page 51: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/51.jpg)
SETUP (2)
Check the engines
./useselect engine,support,transactions,xa -> from information_schema.engines;+------------+---------+--------------+------+| engine | support | transactions | xa |+------------+---------+--------------+------+| SPIDER | YES | YES | YES | | MRG_MYISAM | YES | NO | NO | | CSV | YES | NO | NO | | MyISAM | DEFAULT | NO | NO | | InnoDB | YES | YES | YES | | MEMORY | YES | NO | NO | +------------+---------+--------------+------+
Tuesday, 13 April 2010
![Page 52: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/52.jpg)
USING SPIDER(Simple case)
Tuesday, 13 April 2010
![Page 53: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/53.jpg)
Preparing remote servers (1)
Create three servers using MySQL Sandbox
make_multiple_sandbox \ --group_directory=spider_dir \ --sandbox_base_port=6000 \ --check_base_port 5.1.39
Tuesday, 13 April 2010
![Page 54: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/54.jpg)
Preparing remote servers (2)
Check the port numbers
~/sandboxes/spider_dir/use_all \ "show variables like 'port'" # server: 1: Variable_name Valueport 6001# server: 2: Variable_name Valueport 6002# server: 3: Variable_name Valueport 6003
Tuesday, 13 April 2010
![Page 55: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/55.jpg)
Preparing remote servers (3)
create table definition (tablea.sql)
drop schema if exists myspider;create schema myspider;use myspider;
Create table tbl_a( col_a int, col_b int, primary key(col_a) );
Tuesday, 13 April 2010
![Page 56: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/56.jpg)
Preparing remote servers (4)
create table in remote servers
cd $HOME/sandboxes/spider_dir./use_all "source tablea.sql"
Tuesday, 13 April 2010
![Page 57: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/57.jpg)
setting the main server (1)
create table definition (tablea_main.sql)
drop schema if exists myspider;create schema myspider;use myspider;
Create table tbl_b( col_a int, col_b int, primary key(col_a) ) engine = Spider -- continues ...
Tuesday, 13 April 2010
![Page 58: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/58.jpg)
setting the main server (2)
create table definition (tablea_main.sql) (continues)
Connection ' table "tbl_a", user "msandbox", password "msandbox" 'partition by range( col_a ) ( partition pt1 values less than (1000) comment 'host "127.0.0.1", port "6001"', partition pt2 values less than (2000) comment 'host "127.0.0.1", port "6002"', partition pt3 values less than (MAXVALUE) comment 'host "127.0.0.1", port "6003"' );
Tuesday, 13 April 2010
![Page 59: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/59.jpg)
setting the main server (3)
create table
./use < tablea_main.sql
Tuesday, 13 April 2010
![Page 60: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/60.jpg)
(Finally) using it (1)
in the main server
./use myspiderinsert into tbl_b values (500,1), \(1500,2), (5000,3);Query OK, 3 rows affected (0.01 sec)Records: 3 Duplicates: 0 Warnings: 0
Tuesday, 13 April 2010
![Page 61: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/61.jpg)
(Finally) using it (2)
in the main server
select * from tbl_b;+-------+-------+| col_a | col_b |+-------+-------+| 500 | 1 | | 1500 | 2 | | 5000 | 3 | +-------+-------+3 rows in set (0.01 sec)
Tuesday, 13 April 2010
![Page 62: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/62.jpg)
WHERE IS THE DATA?
500 < 1000 = host 1
1500 < 2000 = host2
5000 < MAXVALUE = host3
Tuesday, 13 April 2010
![Page 63: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/63.jpg)
Looking for the data
in the "remote" servers
$HOME/sandboxes/spider_dir/use_all \ "select * from myspider.tbl_a"# server: 1: col_a col_b500 1# server: 2: col_a col_b1500 2# server: 3: col_a col_b5000 3
Tuesday, 13 April 2010
![Page 64: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/64.jpg)
Using Spider(more complex case)
Tuesday, 13 April 2010
![Page 65: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/65.jpg)
Setting more remote servers (1)
in the main server
./use myspiderdrop table tbl_b;
Tuesday, 13 April 2010
![Page 66: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/66.jpg)
Setting more remote servers (2)
create 20 remote servers
make_multiple_sandbox \ --how_many_nodes=20 \ --group_directory=spider_dir \ --sandbox_base_port=6000 \ 5.1.39
Tuesday, 13 April 2010
![Page 67: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/67.jpg)
Setting more remote servers (3)
create tables for the employees database
cd $HOME/sandboxes/spider_dirwget http://datacharmer.org/downloads/spider_remote_employees.sql./use_all "source spider_remote_employees.sql"
# see also http://launchpad.net/test-db
Tuesday, 13 April 2010
![Page 68: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/68.jpg)
the test employees databasehttp://launchpad.net/test-db
Tuesday, 13 April 2010
![Page 69: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/69.jpg)
Setting the main server (1)
create tables for the employees database
cd $HOME/sandboxes/spider_mainwget http://datacharmer.org/downloads/spider_main_employees.sql./use < spider_main_employees.sql
# see also http://launchpad.net/test-db
Tuesday, 13 April 2010
![Page 70: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/70.jpg)
checking the remote servers
see how many rows have you got after loading
cd $HOME/sandboxes/spider_dir./use_all "select count(*) from employees.salaries"# server: 1: count(*)0# server: 2: count(*)18293# server: 3: count(*)37957# server: 4: count(*)57440...
Tuesday, 13 April 2010
![Page 71: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/71.jpg)
Performance
Tuesday, 13 April 2010
![Page 72: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/72.jpg)
Spider engine performance
Comparable to the gains offered by partitioning
(from 30 to 1000% depending on query type)
Load easily split across masters
Tuesday, 13 April 2010
![Page 73: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/73.jpg)
Running remote commands with Spider
Tuesday, 13 April 2010
![Page 74: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/74.jpg)
data data datadata data
UDFCREATE TABLE db.t1(i int)
"" host "127.0.0.1",port "6001",
user "msandbox"
1 = success0 = failure
Tuesday, 13 April 2010
![Page 75: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/75.jpg)
1 = success0 = failure
data data datadata data
UDFINSERT INTO db.t1VALUES (1000), (2000), (3000)
"" host "127.0.0.1",port "6001",
user "msandbox"
Tuesday, 13 April 2010
![Page 76: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/76.jpg)
data data datadata data
my_temp_table
CREATE TEMPORARY TABLE my_temp_table (i int)
Tuesday, 13 April 2010
![Page 77: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/77.jpg)
data data datadata data
UDFSELECT * FROM db.t1
"my_temp_table"
host "127.0.0.1",port "6001",
user "msandbox"
my_temp_table
100020003000
Tuesday, 13 April 2010
![Page 78: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/78.jpg)
creating a remote table
select spider_direct_sql("create table myspider.t1 (id int)", "","port '6001', host '127.0.0.1', user 'msandbox', password 'msandbox'");
# result: 1
Tuesday, 13 April 2010
![Page 79: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/79.jpg)
inserting records into remote table
select spider_direct_sql("insert into myspider.t1 values (1000),(2000),(3000)", "","port '6001', host '127.0.0.1', user 'msandbox', password 'msandbox'");
# result 1
Tuesday, 13 April 2010
![Page 80: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/80.jpg)
getting remote records (1)
create temporary table remote6001 (i int);
select spider_direct_sql("select * from myspider.t1", "remote6001","port '6001', host '127.0.0.1', user 'msandbox', password 'msandbox'");
# result: 1
Tuesday, 13 April 2010
![Page 81: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/81.jpg)
getting remote records (2)
select * from remote6001;+------+| i |+------+| 1000 || 2000 || 3000 |+------+
Tuesday, 13 April 2010
![Page 82: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/82.jpg)
Verticalpartitioning
Tuesday, 13 April 2010
![Page 83: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/83.jpg)
The vertical partition engine
Same author of the Spider engine
Open source https://launchpad.net/vpformysql
Simple concept
Tuesday, 13 April 2010
![Page 84: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/84.jpg)
employeesemployeesemployeesemployeesemployees
id name salary dept email
1 Joe 1300 1 [email protected]
2 Rick 1250 4 [email protected]
3 Fred 1600 11 [email protected]
Original table
Tuesday, 13 April 2010
![Page 85: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/85.jpg)
empl2empl2empl2id name salary1 Joe 1300
2 Rick 1250
3 Fred 1600
Split tablesempl1empl1empl1
id dept email1 1 [email protected]
3 11 [email protected]
employeesemployeesemployeesemployeesemployeesid name salary dept email1 Joe 1300 1 [email protected]
2 Rick 1250 4 [email protected]
3 Fred 1600 11 [email protected], 13 April 2010
![Page 86: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/86.jpg)
empl2empl2empl2id name salary1 Joe 1300
2 Rick 1250
3 Fred 1600
Split tablesempl1empl1empl1
id dept email1 1 [email protected]
3 11 [email protected]
employeesemployeesemployeesemployeesemployeesid name salary dept email1 Joe 1300 1 [email protected]
2 Rick 1250 4 [email protected]
3 Fred 1600 11 [email protected]
no data here
Tuesday, 13 April 2010
![Page 87: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/87.jpg)
Vertical partitioning installation
Similar to Spider:
download the MySQL source code
download the engine source from https://launchpad.net/vpformysql
copy source under $basedir/storage
apply patch
compile
load engineTuesday, 13 April 2010
![Page 88: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/88.jpg)
vertical partitioning syntaxCREATE TABLE empl1 ( id int not null, name varchar(50), salary decimal(10,3), primary key (id));
CREATE TABLE empl2 ( id int not null, dept int, email varchar(100), primary key (id));
Tuesday, 13 April 2010
![Page 89: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/89.jpg)
vertical partitioning syntaxCREATE TABLE employees ( id int not null, name varchar(50), salary decimal(10,3), dept int, email varchar(100), primary key (id)) engine=VP COMMENT='table_name_list "empl1 empl2"';
Tuesday, 13 April 2010
![Page 90: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/90.jpg)
inserting into the vertical partitioning engine
insert into employees values (1, 'Joe', 1300,1,'[email protected]');Query OK, 1 row affected (0.00 sec)
insert into employees values (2, 'Rick', 1250,4,'[email protected]');Query OK, 1 row affected (0.00 sec)
Tuesday, 13 April 2010
![Page 91: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/91.jpg)
retrieving data from the vertical partitioning engine
select * from employees;+----+------+----------+------+-----------------+| id | name | salary | dept | email |+----+------+----------+------+-----------------+| 1 | Joe | 1300.000 | 1 | [email protected] || 2 | Rick | 1250.000 | 4 | [email protected] |+----+------+----------+------+-----------------+
The data is actually in empl1 and empl2 (check the data directory file sizes to make sure)
Tuesday, 13 April 2010
![Page 92: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/92.jpg)
READING MOREmy blog (search for Spider)
http://datacharmer.blogspot.com
home of the engines
http://launchpad.net/spiderformysql
http://launchpad.net/vpformysql
Look for these slides:
http://slideshare.net/datacharmer
Tuesday, 13 April 2010
![Page 93: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/93.jpg)
MORE SPIDER
ADVANCED SPIDER TECHNIQUES
tomorrow, same time
Tuesday, 13 April 2010
![Page 94: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/94.jpg)
THANKS
Q&AThis work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Tuesday, 13 April 2010
![Page 95: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/95.jpg)
Tuesday, 13 April 2010
![Page 96: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/96.jpg)
Tuesday, 13 April 2010
![Page 97: Sharding for the masses](https://reader031.vdocuments.net/reader031/viewer/2022020116/555a6a0ad8b42a972b8b4aae/html5/thumbnails/97.jpg)
Tuesday, 13 April 2010