Download - Myfox on NodeJS
![Page 2: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/2.jpg)
关于NodeJS
![Page 3: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/3.jpg)
服务器端的JavaScript
□ Google V8引擎
• 属性访问优化
• 缓存机器码
• 垃圾回收
□ libev
□ libeio
□ 单线程,高性能
□ 事件机制
□ 非阻塞异步IO
![Page 4: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/4.jpg)
性能:Socket
□ 远程Memcache
□ 长连接
□ 50连接池
![Page 5: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/5.jpg)
性能:HTTP
□ QPS:4392 □ GC更频繁
![Page 6: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/6.jpg)
什么是MyFOX
![Page 7: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/7.jpg)
什么是Myfox
云梯数据魔方
存储集群 MyISAM
数据装载 数据查询
MyFOX
![Page 8: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/8.jpg)
SQL示例:SELECT IF(INSTR(f.keyword,' ') > 0, UPPER(TRIM(f.keyword)),
CONCAT(b.brand_name,' ',UPPER(TRIM(f.keyword)))) AS f0,
SUM(f.search_num) AS f1,
SUM(f.uv) AS f2,
ROUND(SUM(f.search_num) / SUM(f.uv), 2) AS f3,
ROUND(AVG(f.uv),2) AS f4
FROM dm_fact_keyword_brand_d f
INNER JOIN dim_brand b ON f.keyword_brand_id = b.brand_id
WHERE f.keyword_type_id = 1 AND f.keyword != ''
AND keyword_cat_id IN ('50002535')
AND thedate <= '2011-05-10'
AND thedate >= '2011-05-08'
GROUP BY f0
ORDER BY SUM(f.search_num) DESC LIMIT 0, 100
![Page 9: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/9.jpg)
查询过程
结果合并
取分片数据
路由层
查询层
计算层
缓存
缓存
SQL解析
语义理解
查询路由 字段改写
分片SQL 计算规则
APC
![Page 10: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/10.jpg)
语义理解
WHERE thedate <= '2011-05-10'
AND thedate >= '2011-05-08'
AND f.keyword != ''
AND keyword_cat_id IN ('50002535')
dm_fact_keyword_brand_d
2011-05-08 3 # dm_fact_keyword_brand_d_0.t_686_280 ...
2011-05-09 2 # dm_fact_keyword_brand_d_0.t_3b8_300 ...
2011-05-10 8 # dm_fact_keyword_brand_d_0.t_3ac_293 ...
![Page 11: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/11.jpg)
字段改写
SELECT a AS f0,
SUM(f.search_num) AS f1,
SUM(f.uv) AS f2,
ROUND(SUM(f.search_num) / SUM(f.uv), 2) AS f3,
AVG(f.uv) AS f4
□ AVG(f.uv)
□ 1 + SUM(aa)
□ SELECT a FROM … ORDER BY b
□ 重复查询列
![Page 12: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/12.jpg)
□ 困难
• 分片SQL很多
• PHP单线程
• MySQL查询阻塞
□ 异步并发
• nginx + drizzle
• http协议,curl_multi_get
• JSON格式
mysql
nginx
drizzle
查询SQL
取分片数据
![Page 13: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/13.jpg)
□ 局部有序,局部最优
□ 无聚合字段
• 二路归并
• LIMIT运算
□ 有聚合字段
• 聚合规则处理(SUM,MAX,MIN,DISTINCT)
• 表达式求值
• 堆排序
• LIMIT运算
结果合并
![Page 14: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/14.jpg)
小结:什么是Myfox?
□ 中间层
• 被机器访问
• 功能单一
□ 特点:
• CPU密集
• 无文件IO
• 数据只读
![Page 15: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/15.jpg)
为什么要Node化?
![Page 16: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/16.jpg)
Why?
□ 兴趣,推劢社区发展
□ 榨干CPU,提高并发能力
□ 简化部署
□ 使用连接池
□ 请求状态共享
![Page 17: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/17.jpg)
简化部署
[pengchun]$ cd ${nginx} && ./configure \
--add-module=../drizzle-nginx/ \
--add-module=../rds-json-nginx/ \
--add-module=../nginx-form-input/ \
--add-module=../nginx-devel-kit/ \
--add-module=../nginx-set-misc/
[pengchun]$ cat ./nginx.conf
location /mysql_06_02 {
charset utf-8;
set_form_input $sql '__SQL__';
set_unescape_uri $sql $sql;
drizzle_query $sql;
drizzle_pass cluster_node_06_02;
rds_json on;
}
![Page 18: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/18.jpg)
使用连接池
[pengchun]$ netstat –at | wc –l
25694
[pengchun]$ netstat –at | grep –c ‘TIME_WAIT’
13106
[pengchun]$ cat /etc/sysctl.conf
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 30
[pengchun]$ tail ~/logs/mysql.log
2011-04-28 16:41:44 WARNING CONNECT_ERROR -
{"host":“**","port":3306,"user":“**","pass":"3*******2","error":"Can't
connect to MySQL server on '172.23.*.*' (4)“}
![Page 19: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/19.jpg)
请求状态共享
![Page 20: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/20.jpg)
那么,关注什么?
□ 分层设计
□ 连接管理
□ 稳定性
□ 数据缓存
![Page 21: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/21.jpg)
分层设计
![Page 22: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/22.jpg)
为什么要分层?
□ 利用多核
□ 提高并发能力
• “工作量大的地方多派几个人”
![Page 23: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/23.jpg)
分层逻辑
![Page 24: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/24.jpg)
master
□ 启劢worker和router
□ 心跳检测
□ 监听端口对外服务
□ 权限控制
□ 请求转发给worker
![Page 25: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/25.jpg)
master:压测
并发 CPU占用 RT QPS
600 96% 0.05 1116
700 92% 0.08 1225
800 87% 0.18 1227
900 82% 0.28 1203
1000 81% 0.36 1157
□ HTTP接口
□ RH5.4, 6G MEM, 4 * 2.2 GHz CPU
□ node 0.4.0, siege2.70
![Page 26: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/26.jpg)
worker
□ 向router要路由
□ 取分片数据
□ 结果汇总计算
□ 分布式缓存管理
![Page 27: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/27.jpg)
router
□ 加载路由到内存
□ SQL解析
□ 字段改写
□ 查”路由表”
□ SQL重新生成
![Page 28: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/28.jpg)
连接管理
![Page 29: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/29.jpg)
连接池
var mpool = require('../../lib/pool.js').create(2);
mpool.get(function(conn, pos) {// xxx: blabla
mpool.release(pos);});
mpool.close();
□ mysql
□ memcache
![Page 30: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/30.jpg)
连接池get : function(callback){
if (this.stoped) {return;
}this.queue.push(callback);
},
release : function(id) {this.stack.push(id);
},
while (this.queue.length > 0 && this.stack.length > 0) {var id = this.stack.pop();var cb = this.queue.shift();cb(this.conn[id], id);
delete cb, id;}
![Page 31: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/31.jpg)
PIPE
master
workerrouter
□ Unix socketpair
□ Heartbeat
□ 请求转发
![Page 32: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/32.jpg)
稳定性
![Page 33: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/33.jpg)
异常处理
process.on('uncaughtException', function(err) {console.log('Error : ' + err);// 告警
process.exit(1);});
[pengchun@edp2 bin]$ nohup ./supervise ${nodefox-dir} &
http://cr.yp.to/daemontools/supervise.html
![Page 34: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/34.jpg)
捕捉信号
this.dispatcher.sigaction('SIGHUP', function() {// 忽略HUP信号
});
this.dispatcher.sigaction('SIGTERM', function() {_self.block = true;
// 优雅退出
process.exit();}
// …
![Page 35: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/35.jpg)
心跳检测
□ master -> worker
□ master -> router
-> Hello<- 我还有11个没处理
![Page 36: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/36.jpg)
任务分配
□ “闲”者多劳
![Page 37: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/37.jpg)
数据缓存
![Page 38: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/38.jpg)
两种缓存
分布式缓存 本地缓存
存储介质 memcache 本地内存
数据量 大 小
共享范围 多机共享 进程独享
使用者 worker router
存储数据 分片SQL、最终结果 路由表
![Page 39: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/39.jpg)
本地缓存
var Caches = {};
var Lcache = function(key, ttl) {this.prefix = key.toString().trim();this.expire = ttl;
}
Lcache.prototype.set = function(key, val, ttl) {Caches[this.prefix + key] = {
'v': val,'t': time() + (empty(ttl) ? this.expire: ttl),
};}
![Page 40: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/40.jpg)
可能的问题
var http = require('http');var counter = 0;var restime = 0;var cache = require('../../lib/cache/lcache.js').create(
'benchTest', 3600);
http.createServer(function (req, res) {var tm1 = time();cache.set(++counter, {
'i' : counter,'t' : 'blabla....',
});res.writeHead(200, {'Content-Type': 'text/plain'});res.end('!--STATUS OK--\n');var tm2 = time() - tm1;restime = restime < tm2 ? tm2 : restime;
}).listen(8346, "127.0.0.1");
![Page 41: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/41.jpg)
可能的问题(续)
console.log('num\theapTotal\theapUsed\trss\tvsize\trt');
var mem = process.memoryUsage();setInterval(function() {
mem = process.memoryUsage();console.log(counter + '\t' + mem.heapTotal + '\t' +
mem.heapUsed + '\t' + mem.rss + '\t' +mem.vsize + '\t' + restime);
restime = 0;}, 1000);
[pengchun@edp2 bin]$ ./siege -c1000 127.0.0.1:8346
![Page 42: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/42.jpg)
可能的问题(内存)
0
100
200
300
400
500
600
700
800
900
0
20
40
60
80
100
120
140
160
180
200
MB
MB
Memory Used (from 201 ~ 300 s)
heapTotal heapUsed rss vsize
![Page 43: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/43.jpg)
可能的问题(堆内存)
0
100
200
300
400
500
600
700
0
20
40
60
80
100
120
140
160
180
200
千MB
Memory Used (Interval 6 senconds)
Heap Total Item Count
1G
![Page 44: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/44.jpg)
可能的问题(响应时间)
-10
0
10
20
30
40
50
60
70
ms
Response Time (from 201 ~ 300 s)
response time
![Page 45: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/45.jpg)
改进
□ 目标:
• 内存可控
• 规避GC
□ 方案:
• 摒弃本地缓存?
• B+ Tree with Buffer
var buffer = new Buffer(1024 * 1024 * 200);var http = require('http');var counter = 0;
http.createServer(function (req, res) {counter++;res.writeHead(200, {'Content-Type': 'text/plain'});res.end('!--STATUS OK--\n');
}).listen(8346, "127.0.0.1");
![Page 46: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/46.jpg)
With Buffer
0
100
200
300
400
500
600
700
800
900
1000
0
10
20
30
40
50
60
70
80
百万MB
Memory Used (with Buffer)
heapTotal heapUsed rss items count * 500 vsize
![Page 47: Myfox on NodeJS](https://reader031.vdocuments.net/reader031/viewer/2022013105/549a3edbb479593d6a8b4b61/html5/thumbnails/47.jpg)
Thanks