event driven with libuv and zeromq
TRANSCRIPT
![Page 2: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/2.jpg)
Outline
● Algo Trading Business Flow Review● Order Module Design: Current Flaw● Distributed System: Theory Review● Event Driven System: Review● Event Driven Design In Algo Trading● Project Plan
![Page 3: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/3.jpg)
Algo Trading Flow
I1 I2 I3 …... In
任务(下达 /取消 /停止 )
Instrument (Buy/Sell)
下单
撤单
下单
撤单
步长
步长
.
.
.
...
...
….........
...
...
….........
...
...
….........
End
…............
![Page 4: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/4.jpg)
Current Design Flaw
● Bad Assumptions– Serial Execution Among Instructments
– Batch Job ( 1 Sec Interval)
– I/O Cost/ Latency: Neglectable
– Computation Cost/Latency: Neglectable
– No Uncertainty
![Page 5: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/5.jpg)
Reality
● Parallel execution of Instruments● Market change in Ms, 1 Sec (or More) fixed reponse interval
is 1000X● I/O is SLOW comaring to CPU speed (10^9 order)● Communication Channel will Break● Only God have ALL TIMELY information for decision making.
We people live in UNCERTAINTY● Tradeoff: Best Effort Decision versus Perfect Decision
![Page 6: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/6.jpg)
Distributed System: Our Reality
● Parallel: Everything is Moving● Error Prone (Fault): No 99.9999999....%
Gurantee● Uncertainty: Who/What/Where/When/Why● Concensus/Consistency with HIGH COST● Speed limit: Light Travels at (300,000km/Sec)
![Page 7: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/7.jpg)
CAP: No Perfect World
![Page 8: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/8.jpg)
Case:Two Generals' Problem
![Page 9: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/9.jpg)
Case:Two Generals' Problem
● 1975 . E. A. Akkoyunlu, K. Ekanadham, and R. V. Huber "Some Constraints and Trade-offs in the Design of Network Communications"
● “A pragmatic approach to dealing with the Two Generals' Problem is to use schemes that accept the uncertainty of the communications channel and not attempt to eliminate it, but rather mitigate it to an acceptable degree.”
![Page 10: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/10.jpg)
TCP: Connection Establishment
![Page 11: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/11.jpg)
TCP: Sliding Window
![Page 12: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/12.jpg)
Event Driven: History
● 1960s: Mainframe. Batch Job Oriented● 1970s: Unix. Interactive,Timesharing Terminal● GUI programming:
– Xerox : mouse
– Apple: Apple II
– Microsoft: Windows
– MIT: X windows
– …...
![Page 13: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/13.jpg)
Event Driven: Now
● Libuv: Chrome browser, node.js...● Libev/libevent● Boost ASIO● ACE/ICE● Qt event loop● Glib/Gevent
![Page 14: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/14.jpg)
Event Driven: Now
● Nginx,apache,Twisted,Comet● Node.js● Python Asyncio● Go channel● Scala Actor● C++ 11 Promise/Future,Coroutine,Closure● …........
![Page 15: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/15.jpg)
I/O Modelling
● blocking I/O● nonblocking I/O● I/O multiplexing (select and poll)● signal driven I/O (SIGIO)● asynchronous I/O (the POSIX aio_functions)
Quoted from “UNIX Network Programming Volume 1”
![Page 16: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/16.jpg)
![Page 17: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/17.jpg)
Async/Event Driven Design: Reactor
![Page 18: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/18.jpg)
![Page 19: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/19.jpg)
Proactor
![Page 20: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/20.jpg)
![Page 21: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/21.jpg)
So we are event-driven ready. But...
● Components are mostly event driven● Interface Among components are BLOCKING
INTERFACE! (block/wait/timeout....)● One blocking will ruin the whole infrastructure!● “Public Land Tragic”
![Page 22: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/22.jpg)
class StringQue : public MsgQueBase
![Page 23: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/23.jpg)
StringQueue is everywhere
[luke@rmbp service_multichannel]$ find . -name StringQue.h./orderservice/OMSInfterfaceDemo/StringQue.h./optionhedgeservice/optionhedgeservice/StringQue.h./queryservice/queryservice/StringQue.h./queryservice/queryservice_auto_qt_profitchart/StringQue.h./queryservice/queryservice_auto_qt/StringQue.h./strategyservice_new/hedgefutureservice/StringQue.h./strategyservice_new/algoorderservice/StringQue.h./strategyservice_new/algoorderserviceFuture/StringQue.h./strategyservice/hedgefutureservice/StringQue.h./strategyservice/optionhedgeservice/optionhedgeservice/StringQue.h./strategyservice/algoorderservice/StringQue.h./strategyservice/algoorderserviceFuture/StringQue.h./quoteservice/quoteservicelevel2/StringQue.h./quoteservice/quoteservicelevel2datafeed/StringQue.h./quoteservice/quoteservicelevel2_new/StringQue.h./quoteservice/quoteservicelevel2datafeed_clb/StringQue.h
![Page 24: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/24.jpg)
How we use StringQueue?
![Page 25: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/25.jpg)
Vision
● A fault tolerant, responsive trading platorm– Respond to every marktet tick/heartbeat
● Maintain “fresh” market snapshot● React as every tick comes
– Async, Parallel order fullfillment ● Handle > 1000 orders one server instance without
pressure● Milisec level processing delay per order action
![Page 26: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/26.jpg)
Vision: Unified Event Loop
Libuvevent Loop
ZeroMqsocket
Normal socket
timer
Other Async events
eventfd
fd
timer_t handle
async_t handle
Parallel Tasks req_t
Thread PoolAsync Tasks
Order Fullfill
Market Snapshotupdate
Other Event Loop
Other businessmodules
![Page 27: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/27.jpg)
Async Programming
● Callback● Task(threading)● Future/Promise● Async I/O● Coroutine●
![Page 28: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/28.jpg)
Trading Version 2
I1 I2 I3 …... In
下单
撤单
下单
撤单
timeout
timeout.
.
.
...
...
...
...
….........
...
...
….........
End
…............
tick
….........
EndEndEnd
tick
tick
tick
![Page 29: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/29.jpg)
Project Plan
● Goal: A testable new framwork – Algotrading as typical business test case
– High throughput, low latency
– Fault Tolerant with sensible tradeoff strategy
Side by Side comparison with current order system
![Page 30: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/30.jpg)
Task:
● Project Managment (Wei Song)● Event Loop, ZeroMQ (Tie Gang/Luke)● Market snapshot update (Wei Song/Ze Yu)● Order business module rewrite(Ze Yu/Wei Song)● Testing environment (Wei Song/Tie Gang)
– Simulation data/replay
– OMS simulator
● Testing– All
– + Calvin + other business staff
![Page 31: Event Driven with LibUV and ZeroMQ](https://reader031.vdocuments.net/reader031/viewer/2022012311/55a7949a1a28ab751f8b45a9/html5/thumbnails/31.jpg)
Schedule?
● What can we accomplish before Spring Festival?