monitoring electronic trading environments using spark by fergal toomey and pierre lacave
TRANSCRIPT
![Page 1: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/1.jpg)
Monitoring Electronic Trading Environments using Spark
Fergal Toomey and Pierre LacaveCorvil
![Page 2: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/2.jpg)
Monitoring Goals
•Performance and technical risk
•Client experience
•Compliance
![Page 3: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/3.jpg)
Challenges
•Distributed across sites and firms
•Latency sensitive
•Multiple protocols and data formats
EXCHANGES
Matching Engine Trading
Gateways
Market Data
Validation/ Routing
Consolidation/ Distribution
MARKET DATA SERVICES
BROKERAGE SERVICES
TRADERS
•Transformations:•Change of custody•Different symbologies•Parent/child orders•Variable routing
![Page 4: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/4.jpg)
Data Sources
![Page 5: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/5.jpg)
Data Volumes
![Page 6: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/6.jpg)
Platform
Corvil Streaming
DataCorvil
Streaming DataCorvil Streaming
Data
Message BusDistributed Real-Time Processing
StorageData Sources
Exploration and Visualization
![Page 7: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/7.jpg)
Metrics Calculation
Windowing & Correlation
Raw Data•Enriched (correlated data)•Searchable
Custom Metrics•Latencies•Counts•Filterable by key dimensions•Rolled up
Clients MarketsA B C D
![Page 8: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/8.jpg)
ID: 1
CLIENT: C1SYMBOL: ABC
ID: APARENTID: 1
SYMBOL: ABC
ID: ZPARENTID: A
SYMBOL: ABC
ID: 10PARENTID: Z
SYMBOL: ABCMARKET: NYSE
Clients Markets
Events Correlation
Filter by destination market in point A?
A B C D
![Page 9: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/9.jpg)
ID: 1
CLIENT: C1SYMBOL: ABC
ID: APARENTID: 1
SYMBOL: ABC
ID: ZPARENTID: A
SYMBOL: ABC
ID: 10PARENTID: Z
SYMBOL: ABCMARKET: NYSE
Clients Markets
Events CorrelationA B C D
![Page 10: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/10.jpg)
ID: 1
CLIENT: C1SYMBOL: ABCMARKET: NYSE
ID: APARENTID: 1
CLIENT: C1 SYMBOL: ABCMARKET: NYSE
ID: ZPARENTID: A
CLIENT: C1 SYMBOL: ABCMARKET: NYSE
ID: 10PARENTID: Z
CLIENT: C1SYMBOL: ABCMARKET: NYSE
Clients Markets
Events CorrelationA B C D
![Page 11: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/11.jpg)
Events Windowing
Batch #1 Batch #2 Batch #3
window #1
window #2
A B C D
Spark Streaming window
Reconstruct batch based on original time
Spark Streaming micro-batch
![Page 12: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/12.jpg)
Direct Data Access
Access metrics via SQL / JDBC
BI tools, self-developed application, etc
Calculation done in HBASE
![Page 13: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/13.jpg)
![Page 14: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/14.jpg)
Data Extracts
• Filtering pushed down to HBASE• Aggregation done In SPARK (SparkSQL)
CSV extraction to HDFS
![Page 15: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/15.jpg)
Extract Consumption
![Page 16: Monitoring Electronic Trading Environments using Spark by Fergal Toomey and Pierre Lacave](https://reader033.vdocuments.net/reader033/viewer/2022051520/58f9a94e760da3da068b6d52/html5/thumbnails/16.jpg)
THANK YOU.www.corvil.com