grid computing and trade analytics with elastic - jay chin's presentation at elasticontour 2015
TRANSCRIPT
Jay Chin – [email protected]
Principal Consultant, Excelian
3 November 2015
0
Grid Computing and Trade Analytics
with Elastic
Excelian Technical Consulting – Who we are
1
Financial Services specialists
Distributed computing specialists since 2006
Experts in niche and emerging technologies
Financial Services – Insatiable appetite for Compute
• Algorithms (Computers) that actually do the trading - Remember the Flash Crash of May 6 2010? This is a result of HFT stopping and not trading causing the Market to drop 6% in mere minutes.
• Financial modelling - Use complex mathematical models to deal with asset prices, market movement, portfolio returns, etc.
• Huge amounts of data to process - e.g. connecting to one of the exchanges FIX, there will be up to 100,000 messages to process per second.
2
Source: Information Week, Wall Street & Technology Source: The Telegraph
What do compute grids look like ?
3
Typical Numbers For A Standard Grid
- 40k cores/engines
- 30m tasks
- 120 GB of Log metrics
- 60 – 80% Average Utilisation
- Data retention up to 6 Months
https://flic.kr/p/ydnEvw
Grid Maturity in Financial Services
4
HPC Maturity
Benchmark
2014
Tier I = Tier I banks
Tier II = Tier II banks
Point = point solutions used only for a specific use case (e.g. behind a software package, only for one business line…)
Mat
uri
ty L
evel
It is fairly common for bank to have grids. Larger banks tend to have at least 30,000 cores.
Case Study: ELK for Enterprise Grid Reporting Framework
Requirements:
• Enterprise Grid with 40,000 Cores across 4 Data centers in 2 Countries
• Reporting Dashboard for Grid Metrics
• Scalable up to 100,000 cores and 200 million Grid tasks per day
5
Goal: Architect an Enterprise Grid and design a Grid metrics reporting framework
for a top-tier investment bank.
The Case for ELK
6
Features ElasticSearchIntuitive Interface
Ease of Use
Security Integration
Scalability
Support
Pricing
Features
Integration with Grid Middleware
Elasticsearch met all the requirements except for the last one,which required some work on our part.
Initial Architecture: Single cluster across 2 regions
7
curl -XPUT localhost:9200/GridA_metrics/_settings -d '{"index.routing.allocation.include.tag" : “region_A" }'
Architecture After Consultation with Elastic Platinum Support
8
Challenges
9
• Bespoke deployment due to security restrictions in Bank’s Datacentre
• https://github.com/Excelian/ansible_fs_elkstack
• Development of custom ETL to query Grid Metrics database and load them into ElasticSearch
https://flic.kr/p/eqJHbr
More ELK Goodness
• Bank was very impressed with the reporting capabilities
• Support team at Elastic was also superior compared to some of the big vendors we were dealing with
10
AS A RESULT
1. We were tasked to do log centralization using Logstash
2. Explore Watcher for monitoring Grid and applications
Feedback from Investment Bank
• For the first time ever, developers were able to view Grid metrics and correlate them with logging events from a single interface
• Application teams are experimenting with Elastic
• Developers rethinking logging
11
Key Takeaways
• Lots of opportunities and interest in ElasticSearch in Financial services
• Single tool to do log analytics, alerting, events, searching, and metrics
• Elastic ticks all the right boxes for financial services: Security, scalability, support SLAs, etc.
• Elastic Platinum support has been fantastic
• Advanced Use Cases : Fraud Detection, Trade surveillance, Market Sentiment Analysis
12
13
Thank you!
If you have any feedback, please get in touch:
If you would like to join our community of technologists at Excelian please have a look at our careers page for the latest vacancies:
www.excelian.com/careers/
@Excelian
@Excelian
@ExcelianLTD
www.elastic.co
14