druid和kylin - ouyangchen.com’Œkylin在美团点评的选型与实践.pdf · druid和kylin...
TRANSCRIPT
-
DruidKylin
2017-08-05
-
OLAP
Druid
Kylin
-
OLAP
ROLAP (Relational OLAP)
RDBMS/
Presto / Impala / SparkSQL / Drill
MOLAP (Multi-dimensional OLAP)
Kylin / Druid
Search Engines
Elasticsearch / Solr
-
ROLAP (Presto / SparkSQL)
SQL
-
MOLAP (Kylin, Druid)
OLAP
-
Search Engine (ES)
Join
OLAP
-
1.
2. /
3.
-
OLAP
SQL
-
KylinDruid
Presto / Spark Kylin Druid ES
SQL
-
OLAP
Druid
Kylin
-
Druid
OLAP
40100Datasource20 TB
Kafka
150TP99~1
-
Druid/JVM
JVM
Coordinator/Overlord 2 x 8g VMs 6g Heap
Broker 8 x 8core 16g VMs 11g Heap, 3g Non-Heap
Historical 19 x 40core 128g 12disk 12g Heap, 10g Non-Heap
MiddleManager 19 x 40core 128g 8 x 6g Heap Peons
Tranquility Clients 2 x 40core 128g 2~3g per JVM
-
Druid
DataSourceQPSLatencyIngestion Rate
/CPUI/OJVM
DashboardOLAP
Kafka ESEmitter
druid
Spark Streaming
-
DruidDashboard
-
Druid
Filter
Agg
Grouping
-
Tier
historical historical
historical historical
broker broker
Load Balance
_default_tier
historical historical
historical historical
broker broker
Load Balance
workload
-
(1/2)
Historicalsys cpu
NUMAzone reclaim modedirect page scan
echo 0 > /proc/sys/vm/zone_reclaim_mode
-
(2/2)
CentOS 6.5
echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag
-
SQL
-
OLAP
Druid
Kylin
-
Kylin
KylinHadoopOLAPSQL
KylinMOLAPCube
CubeCube Cube
-
Kylin
-
Kylin
OLAP
20+Project350Cube
8000Cube140TB
130TP99~1
-
Kylin
RS RS
job jobHA
A
query queryB
query query
HBase
RS
RS RSRS
Cube
-
1. 2. 3. 4. Union
1. 2. HBaseHA 3. 4.
1. 2. 3. 4. JMX
3Kylin Committer
-
Cube