toros: python framework for recommender system

48

Upload: kwangseob-kim

Post on 21-Apr-2017

885 views

Category:

Data & Analytics


2 download

TRANSCRIPT

Page 1: TOROS: Python Framework for Recommender System
Page 2: TOROS: Python Framework for Recommender System
Page 3: TOROS: Python Framework for Recommender System
Page 4: TOROS: Python Framework for Recommender System
Page 5: TOROS: Python Framework for Recommender System
Page 6: TOROS: Python Framework for Recommender System
Page 7: TOROS: Python Framework for Recommender System
Page 8: TOROS: Python Framework for Recommender System
Page 9: TOROS: Python Framework for Recommender System
Page 10: TOROS: Python Framework for Recommender System
Page 11: TOROS: Python Framework for Recommender System

* http://wallpapers.ws/10852-sea-water-stone.html

* https://www.sitepoint.com/full-stack-developer/

Page 12: TOROS: Python Framework for Recommender System
Page 13: TOROS: Python Framework for Recommender System
Page 14: TOROS: Python Framework for Recommender System
Page 15: TOROS: Python Framework for Recommender System

Kafka

……

Page 16: TOROS: Python Framework for Recommender System
Page 17: TOROS: Python Framework for Recommender System
Page 18: TOROS: Python Framework for Recommender System

>>> import multiprocessing

Page 19: TOROS: Python Framework for Recommender System
Page 20: TOROS: Python Framework for Recommender System

>>> import multiprocessing

>>> pool = multiprocessing.Pool(8)

>>> pool.map(do_some_work, jobs)

>>> Q = kafka.get_consumer(group_id=‘group_a’)

>>> Q.get_message()

Page 21: TOROS: Python Framework for Recommender System

load balancing(distributed!)fault tolerance

scalabilitysimpler/better code/flow

Page 22: TOROS: Python Framework for Recommender System
Page 23: TOROS: Python Framework for Recommender System
Page 24: TOROS: Python Framework for Recommender System
Page 25: TOROS: Python Framework for Recommender System
Page 26: TOROS: Python Framework for Recommender System

10~40x

http://blog.parsely.com/post/3886/pykafka-now/

Page 27: TOROS: Python Framework for Recommender System
Page 28: TOROS: Python Framework for Recommender System
Page 29: TOROS: Python Framework for Recommender System
Page 30: TOROS: Python Framework for Recommender System
Page 31: TOROS: Python Framework for Recommender System
Page 32: TOROS: Python Framework for Recommender System
Page 33: TOROS: Python Framework for Recommender System

Method Cores Elapsed(secs)

mput 1 93

sst 1 98

sst 4 47

sst 6 27

sst 12 22

Page 34: TOROS: Python Framework for Recommender System
Page 35: TOROS: Python Framework for Recommender System
Page 36: TOROS: Python Framework for Recommender System
Page 37: TOROS: Python Framework for Recommender System
Page 38: TOROS: Python Framework for Recommender System
Page 39: TOROS: Python Framework for Recommender System
Page 40: TOROS: Python Framework for Recommender System
Page 41: TOROS: Python Framework for Recommender System

redirect_stderr = trueprocess_name=%(program_name)snumprocs=1

[program:meta.ab.watcher]command = python MetaWatcher.py ./MetaABdirectory = /daum/kskim/aurochs.app.git/1boonstdout_logfile = ./Log/meta_ab_watcher.logstdout_logfile_maxbytes = 50MBstdout_logfile_backups = 3redirect_stderr = trueprocess_name=%(program_name)snumprocs=1

[program:meta.ab.picker]command = python MetaPicker.py ./MetaABdirectory = /daum/kskim/aurochs.app.git/1boonstdout_logfile = ./Log/meta_ab_picker.logstdout_logfile_maxbytes = 50MBstdout_logfile_backups = 3redirect_stderr = trueprocess_name=%(program_name)snumprocs=1

[eventlistener:ev]command = python ev.pydirectory = /daum/kskim/aurochs.app.git/events=PROCESS_STATE,TICK_60

[supervisord]

[supervisorctl]

[inet_http_server]port = *:23231username = xxxpassword = xxx

[rpcinterface:supervisor]supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

Page 42: TOROS: Python Framework for Recommender System
Page 43: TOROS: Python Framework for Recommender System
Page 44: TOROS: Python Framework for Recommender System
Page 45: TOROS: Python Framework for Recommender System
Page 46: TOROS: Python Framework for Recommender System
Page 47: TOROS: Python Framework for Recommender System