apache phoenix: transforming hbase into a sql...
TRANSCRIPT
![Page 1: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/1.jpg)
Apache Phoenix: Transforming HBase into a SQL database James Taylor @JamesPlusPlus http://phoenix.apache.org
![Page 2: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/2.jpg)
About me
Completed
o Architect at Salesforce.com in Big Data group o Started Phoenix as internal project ~3 years ago o Open-source on Github ~1.5 years ago o Apache incubator for ~5 months o Graduated as Top Level Project in May 2014
o Engineer at BEA Systems o XQuery-based federated query engine o SQL-based complex event processing engine
![Page 3: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/3.jpg)
Agenda
Completed
o What is Apache Phoenix? o Why is it so fast? o How does it help HBase scale? o Roadmap o Q&A
![Page 4: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/4.jpg)
What is Apache Phoenix?
Completed
![Page 5: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/5.jpg)
What is Apache Phoenix?
Completed
1. Turns HBase into a SQL database o Query Engine o MetaData Repository o Embedded JDBC driver o Only for HBase data
![Page 6: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/6.jpg)
What is Apache Phoenix?
Completed
2. Fastest way to access HBase data o HBase-specific push down o Compiles queries into native
HBase calls (no map-reduce) o Executes scans in parallel
![Page 7: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/7.jpg)
Completed
SELECT * FROM t WHERE k IN (?,?,?)
Phoenix Stinger (Hive 0.11)
0.04 sec 280 sec
* 110M row table
7,000x faster
![Page 8: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/8.jpg)
What is Apache Phoenix?
Completed
3. Lightweight o No additional servers required o 100% Java o Included in HDP 2.1 distribution o Available in Amazon EMR o Otherwise copy Phoenix jar into
HBase lib directory on each RS
![Page 9: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/9.jpg)
HBase Cluster Architecture
![Page 10: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/10.jpg)
HBase Cluster Architecture
Phoenix
![Page 11: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/11.jpg)
HBase Cluster Architecture
Phoenix
Phoenix
![Page 12: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/12.jpg)
What is Apache Phoenix?
Completed
4. Integration-friendly o Map to existing HBase table o Integrate with Apache Pig o Integrate with Apache Flume o Integrate with Apache Sqoop (wip)
![Page 13: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/13.jpg)
What is Apache Phoenix?
Completed
1. Turns HBase into a SQL database 2. Fastest way to access HBase data 3. Lightweight 4. Integration-friendly
![Page 14: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/14.jpg)
Why is Phoenix so fast?
Completed
![Page 15: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/15.jpg)
Why is Phoenix so fast?
Completed
1. HBase o Fast, but “dumb” (on purpose)
2. Data model o Support for composite primary key o Binary data sorts naturally
3. Client-side parallelization 4. Push down
o Custom filters and coprocessors
![Page 16: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/16.jpg)
Phoenix Data Model
HBase Table
Phoenix maps HBase data model to the relational world
![Page 17: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/17.jpg)
Phoenix Data Model
HBase Table Column Family A Column Family B
Phoenix maps HBase data model to the relational world
![Page 18: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/18.jpg)
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3
Phoenix maps HBase data model to the relational world
![Page 19: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/19.jpg)
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 KeyValue
Phoenix maps HBase data model to the relational world
![Page 20: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/20.jpg)
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 KeyValue
Row Key 2 KeyValue KeyValue
Phoenix maps HBase data model to the relational world
![Page 21: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/21.jpg)
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 KeyValue
Row Key 2 KeyValue KeyValue
Row Key 3 KeyValue
Phoenix maps HBase data model to the relational world
![Page 22: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/22.jpg)
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 Value
Row Key 2 Value Value
Row Key 3 Value
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 KeyValue
Row Key 2 KeyValue KeyValue
Row Key 3 KeyValue
Phoenix maps HBase data model to the relational world
![Page 23: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/23.jpg)
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 Value
Row Key 2 Value Value
Row Key 3 Value
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 Value
Row Key 2 Value Value
Row Key 3 Value
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 KeyValue
Row Key 2 KeyValue KeyValue
Row Key 3 KeyValue
Phoenix maps HBase data model to the relational world
![Page 24: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/24.jpg)
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 Value
Row Key 2 Value Value
Row Key 3 Value
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 Value
Row Key 2 Value Value
Row Key 3 Value
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 KeyValue
Row Key 2 KeyValue KeyValue
Row Key 3 KeyValue
Phoenix maps HBase data model to the relational world
Multiple Versions
![Page 25: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/25.jpg)
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 KeyValue
Row Key 2 KeyValue KeyValue
Row Key 3 KeyValue
Phoenix maps HBase data model to the relational world Phoenix Table
![Page 26: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/26.jpg)
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 KeyValue
Row Key 2 KeyValue KeyValue
Row Key 3 KeyValue
Phoenix maps HBase data model to the relational world Phoenix Table
Key Value Columns
![Page 27: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/27.jpg)
Phoenix Data Model
HBase Table Column Family A Column Family B
Qualifier 1 Qualifier 2 Qualifier 3 Row Key 1 KeyValue
Row Key 2 KeyValue KeyValue
Row Key 3 KeyValue
Phoenix maps HBase data model to the relational world Phoenix Table
Key Value Columns Primary Key Constraint
![Page 28: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/28.jpg)
Example
Row Key
SERVER METRICS
HOST VARCHAR DATE DATE RESPONSE_TIME INTEGER GC_TIME INTEGER CPU_TIME INTEGER IO_TIME INTEGER
Over metrics data for servers with a schema like this:
![Page 29: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/29.jpg)
Example Over metrics data for servers with a schema like this:
Key Values
SERVER METRICS
HOST VARCHAR DATE DATE RESPONSE_TIME INTEGER GC_TIME INTEGER CPU_TIME INTEGER IO_TIME INTEGER
![Page 30: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/30.jpg)
Example
CREATE TABLE SERVER_METRICS (
HOST VARCHAR, DATE DATE, RESPONSE_TIME INTEGER, GC_TIME INTEGER, CPU_TIME INTEGER, IO_TIME INTEGER, CONSTRAINT pk PRIMARY KEY (HOST, DATE))
DDL command looks like this:
![Page 31: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/31.jpg)
With data that looks like this:
SERVER METRICS HOST + DATE RESPONSE_TIME GC_TIME SF1 1396743589 1234 SF1 1396743589 8012 … SF3 1396002345 2345 SF3 1396002345 2340 SF7 1396552341 5002 1234 …
Example
Row Key
![Page 32: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/32.jpg)
With data that looks like this:
SERVER METRICS HOST + DATE RESPONSE_TIME GC_TIME SF1 1396743589 1234 SF1 1396743589 8012 … SF3 1396002345 2345 SF3 1396002345 2340 SF7 1396552341 5002 1234 …
Example
Key Values
![Page 33: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/33.jpg)
Phoenix Push Down: Example
Completed
SELECT host, avg(response_time) FROM server_metrics WHERE date > CURRENT_DATE() – 7 AND host LIKE ‘SF%’ GROUP BY host
![Page 34: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/34.jpg)
Phoenix Push Down: Example
Completed
SELECT host, avg(response_time) FROM server_metrics WHERE date > CURRENT_DATE() – 7 AND host LIKE ‘SF%’ GROUP BY host
![Page 35: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/35.jpg)
Phoenix Push Down: Example
Completed
SELECT host, avg(response_time) FROM server_metrics WHERE date > CURRENT_DATE() – 7 AND host LIKE ‘SF%’ GROUP BY host
![Page 36: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/36.jpg)
Phoenix Push Down: Example
Completed
SELECT host, avg(response_time) FROM server_metrics WHERE date > CURRENT_DATE() – 7 AND host LIKE ‘SF%’ GROUP BY host
![Page 37: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/37.jpg)
Phoenix Push Down: Example
Completed
SELECT host, avg(response_time) FROM server_metrics WHERE date > CURRENT_DATE() – 7 AND host LIKE ‘SF%’ GROUP BY host
![Page 38: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/38.jpg)
Phoenix Push Down
1. Skip scan filter 2. Aggregation 3. TopN 4. Hash Join
![Page 39: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/39.jpg)
Phoenix Push Down: Skip scan SELECT host, avg(response_time)
FROM server_metrics WHERE date > CURRENT_DATE() – 7 AND host LIKE ‘SF%’ GROUP BY host
![Page 40: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/40.jpg)
Phoenix Push Down: Skip scan
Completed
R1
R2
R3
R4
![Page 41: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/41.jpg)
Phoenix Push Down: Skip scan Client-side parallel scans
Completed
R1
R2
R3
R4
scan1
scan3
scan2
![Page 42: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/42.jpg)
Phoenix Push Down: Skip scan Server-side filter
Completed
SKIP
![Page 43: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/43.jpg)
Phoenix Push Down: Skip scan Server-side filter
Completed
INCLUDE
![Page 44: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/44.jpg)
Phoenix Push Down: Skip scan Server-side filter
Completed
SKIP
![Page 45: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/45.jpg)
Phoenix Push Down: Skip scan Server-side filter
Completed INCLUDE
![Page 46: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/46.jpg)
Phoenix Push Down: Skip scan Server-side filter
SKIP
![Page 47: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/47.jpg)
Phoenix Push Down: Skip scan Server-side filter
INCLUDE
![Page 48: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/48.jpg)
Phoenix Push Down: Skip scan Server-side filter
INCLUDE
INCLUDE
INCLUDE
![Page 49: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/49.jpg)
Phoenix Push Down: Aggregation SELECT host, avg(response_time)
FROM server_metrics WHERE date > CURRENT_DATE() – 7 AND host LIKE ‘SF%’ GROUP BY host
![Page 50: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/50.jpg)
SERVER METRICS HOST DATE KV1 KV2 KV3 SF1 Jun 2 10:10:10.234 239 234 674 SF1 Jun 3 23:05:44.975 23 234 SF1 Jun 9 08:10:32.147 256 314 341 SF1 Jun 9 08:10:32.147 235 256 SF1 Jun 1 11:18:28.456 235 23 SF1 Jun 3 22:03:22.142 234 314 SF1 Jun 3 22:03:22.142 432 234 256 SF2 Jun 1 10:29:58.950 23 432 SF2 Jun 2 14:55:34.104 314 876 23 SF2 Jun 3 12:46:19.123 256 234 314 SF2 Jun 3 12:46:19.123 432 SF2 Jun 8 08:23:23.456 876 876 235 SF2 Jun 1 10:31:10.234 234 234 876 SF3 Jun 1 10:31:10.234 432 432 234 SF3 Jun 3 10:31:10.234 890 SF3 Jun 8 10:31:10.234 314 314 235 SF3 Jun 1 10:31:10.234 256 256 876 SF3 Jun 1 10:31:10.234 235 234 SF3 Jun 8 10:31:10.234 876 876 432 SF3 Jun 9 10:31:10.234 234 234 SF3 Jun 3 10:31:10.234 432 276 … … … … …
Phoenix Push Down: Aggregation Aggregate on server-side
SERVER METRICS HOST AGGREGATE VALUES SF1 3421 SF2 2145 SF3 9823
![Page 51: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/51.jpg)
Phoenix Push Down: TopN
Completed
SELECT host, date, gc_time FROM server_metrics WHERE date > CURRENT_DATE() – 7 AND host LIKE ‘SF%’ ORDER BY gc_time DESC LIMIT 5
![Page 52: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/52.jpg)
Phoenix Push Down: TopN Client-side parallel scans
Completed
R1
R2
R3
R4
scan1
scan3
scan2
![Page 53: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/53.jpg)
Phoenix Push Down: TopN Each region holds N rows
Completed
R1
R2
R3
R4
scan1
![Page 54: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/54.jpg)
Phoenix Push Down: TopN Each region holds N rows
Completed
R1
R2
R3
R4
scan2
![Page 55: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/55.jpg)
Phoenix Push Down: TopN Each region holds N rows
Completed
R1
R2
R3
R4
scan3
![Page 56: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/56.jpg)
SERVER METRICS HOST DATE GC_TIME SF3 Jun 2 10:10:10.234 22123
SF5 Jun 3 23:05:44.975 19876
SF2 Jun 9 08:10:32.147 11345
SF2 Jun 1 11:18:28.456 10234
SF1 Jun 3 22:03:22.142 10111
Phoenix Push Down: TopN Client-side final merge sort
Scan1
Scan2
Scan3
![Page 57: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/57.jpg)
Phoenix Push Down: TopN Secondary Index
Completed
CREATE INDEX gc_time_index ON server_metrics (gc_time DESC, date DESC) INCLUDE (response_time)
Row Key
GC_TIME_INDEX GC_TIME INTEGER DATE DATE HOST VARCHAR RESPONSE_TIME INTEGER
![Page 58: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/58.jpg)
Phoenix Push Down: TopN Secondary Index
Completed
CREATE INDEX gc_time_index ON server_metrics (gc_time DESC, date DESC) INCLUDE (response_time)
Key Value
GC_TIME_INDEX GC_TIME INTEGER DATE DATE HOST VARCHAR RESPONSE_TIME INTEGER
![Page 59: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/59.jpg)
Phoenix Push Down: TopN Secondary Index
Completed
o Original query doesn’t change o Phoenix rewrites query to use index table o All referenced columns must exist in index table
for it to be considered o Stats coming soon!
![Page 60: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/60.jpg)
Phoenix Push Down: Hash Join
Completed
SELECT m.*, i.location FROM server_metrics m JOIN host_info i ON m.host = i.host WHERE m.date > CURRENT_DATE() – 7 AND i.location = ‘SF’ ORDER BY m.gc_time DESC LIMIT 5
![Page 61: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/61.jpg)
Phoenix Push Down: Hash Join Separate LHS and RHS
Completed
SELECT m.*, i.location FROM server_metrics m JOIN host_info i ON m.host = i.host WHERE m.date > CURRENT_DATE() – 7 AND i.location = ‘SF’ ORDER BY m.gc_time DESC LIMIT 5
![Page 62: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/62.jpg)
Phoenix Push Down: Hash Join Separate LHS and RHS
Completed
SELECT m.*, i.location FROM server_metrics m JOIN host_info i ON m.host = i.host WHERE m.date > CURRENT_DATE() – 7 AND i.location = ‘SF’ ORDER BY m.gc_time DESC LIMIT 5
![Page 63: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/63.jpg)
Phoenix Push Down: Hash Join Separate LHS and RHS
Completed
LHS SELECT * FROM server_metrics WHERE date > CURRENT_DATE() – 7 ORDER BY gc_time DESC LIMIT 5
RHS SELECT location FROM host_info WHERE location = ‘SF’
![Page 64: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/64.jpg)
Phoenix Push Down: Hash Join Execute & broadcast RHS to each RS
Completed
RS1
RS2
RHS
![Page 65: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/65.jpg)
Phoenix Push Down: TopN Server-side map lookup during scan
Completed
R1
R2
R3
R4
RHS
RHS
LHS scan1
scan2
scan3
scan4
![Page 66: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/66.jpg)
How does Phoenix help HBase scale?
![Page 67: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/67.jpg)
How does Phoenix help HBase scale?
1. Phoenix allows multiple tables to share same physical HBase table o Updateable VIEW o Multi-tenant TABLE + tenant-specific VIEW o Support for secondary indexes on VIEWs
![Page 68: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/68.jpg)
How does Phoenix help HBase scale?
2. HBase wants small # of big tables instead of large # of small tables o Each region for each column family of each table
consumes resources
![Page 69: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/69.jpg)
Phoenix Shared Tables: VIEW
Completed
CREATE TABLE event ( type CHAR(1), event_id BIGINT, created_date DATE, created_by VARCHAR, CONSTRAINT pk PRIMARY KEY (type, event_id));
CREATE VIEW web_event ( referrer VARCHAR) AS
SELECT * FROM event WHERE type=‘w’;
• Includes columns from TABLE • Cannot define PK • Updateable if only equality
expressions separated by AND
![Page 70: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/70.jpg)
Phoenix Shared Tables: VIEW Same physical HBase table
Completed
type = ‘c’
type = ‘m’
type = ‘p’
type = ‘w’
EVENT
CHAT_EVENT
MOBILE_EVENT
PHONE_EVENT
WEB_EVENT
![Page 71: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/71.jpg)
Phoenix Shared Table: MULTI_TENANT
Completed
CREATE TABLE event ( tenant_id VARCHAR, type CHAR(1), event_id BIGINT, created_date DATE, created_by VARCHAR, CONSTRAINT pk PRIMARY KEY (tenant_id, type, event_id))
MULTI_TENANT=true;
First PK column identifies tenant ID
![Page 72: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/72.jpg)
Phoenix Shared Table: MULTI_TENANT
Completed
CREATE VIEW web_event ( referrer VARCHAR) AS SELECT * FROM event WHERE type=‘w’; DriverManager.connect(“jdbc:phoenix:localhost;tenantId=me”);
CREATE VIEW my_web_event AS SELECT * FROM web_event;
Tenant-specific view
Tenant-specific connection
![Page 73: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/73.jpg)
Phoenix Shared Tables: MULTI_TENANT Same physical HBase table
tenant_id = ‘aaa’
…
EVENT
tenant_id = ‘aab’
tenant_id = ‘zzz’
![Page 74: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/74.jpg)
Phoenix Shared Tables: MULTI_TENANT Same physical HBase table
type = ‘c’
type = ‘m’
type = ‘p’
type = ‘w’
EVENT
CHAT_EVENT
MOBILE_EVENT
PHONE_EVENT
WEB_EVENT
PER tenant_id
![Page 75: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/75.jpg)
Phoenix Shared Tables: MULTI_TENANT o Tenant-specific connection may only see and
operate on their data o MetaData APIs honor this o Phoenix automatically manages scan ranges
o Primary key constraint of base table may not be changed
o Indexes in separate shared table may be added to a VIEW o DDL operations restricted
o No ALTER of base table o No DROP of columns referenced in WHERE clause
![Page 76: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/76.jpg)
Phoenix Roadmap
Completed
o Derived/nested tables (in 3.1/4.1) o Local Indexes (in 4.1) o Transactions o More Join strategies o Correlated sub-queries o Cost-based query optimizer o OLAP extensions
![Page 77: Apache Phoenix: Transforming HBase into a SQL databasephoenix.apache.org/presentations/HadoopSummit2014-16x9.pdf · HBase Table Column Family A Column Family B Qualifier 1 Qualifier](https://reader031.vdocuments.net/reader031/viewer/2022022808/5e0dba3b7694c04d1b2aaa57/html5/thumbnails/77.jpg)
Thank you! Questions/comments?