"an introduction to kx technology: a big data solution" chris leckey, a data scientist...
TRANSCRIPT
An Introduction to Kx Technology: A Big Data Solution
8th December 2016
Chris Leckey Kx Systems
Est. 1993, California, USA
Founder's Concern: Volume
Widely adopted in financial services over two decades
Global company, Large user community
Other industries: Energy, Pharmaceuticals, Telecommunications, Sport
About Kx Systems
Kx Clients & Industries
About the Technology
• Integrated in-memory, columnar
database & programming system
• Map-Reduce built-in
• Parallelism
• Time as native
• Streaming, real-time and historical
data.
• Lightweight Executable (800kB)
• Low TCO: Low O&M costs Standard OS & Hardware
q at a Glance
• Functional array processing language
• Vector based
• Descended from Ken Iverson's APL
• Right to Left Execution
• Low Latency:
Interpreted, not compiled
Runs inside database
• Terse - Get more from less code
Project Euler Problem 1 in qProject Euler Problem 1 in q
Project Euler Problem 1 in JavaProject Euler Problem 1 in Java
44 Characters44 Characters
233 Characters233 Characters
q – Simple Query Syntax
6
sym price sizeJPM 34.51 105
GE 71.23 97BP 18.69 23
JPM 34.53 16
JPM 34.55 85BP 18.72 81
GE 71.15 48
sym price size
JPM 34.51 105
JPM 34.53 16
JPM 34.55 85
Trade Table Query
Result
Kx Performance Comparison
• Dataset: NYSE TAQ 5000 days. 1.1 Trillion quotes. 65 Billion trades. 100TB raw text.
• Machine: 16 core 256GB. Date partitioned. Symbol indexed.
• All queries in RAM (kdb+ 3.x March 2015)
• Time to query for smallest day trades. 2003.09.10 35M quotes. 5M trades.
Kx Performance
“Kx’s kdb+, running on Dell EMC DSSD D5 and PowerEdge servers, has established performance records while testing on the largest STAC-M3 data scale so far.”
Peter Lankford, STAC Director
• The Independent Electricity System Operator (IESO)• Largest multi-tenant meter data processing service (4.7 million meters)• > 300 billion rows of data, growing by 120 million per day• Selected and applied Kx to support:
Analytics not previously possible On demand access to all of the data by utilities and their customers 24x7x365 availability and near real-time synchronization with transaction
system.
Kx for Utilities Case Study
Service PreviousSolution
KxSolution
Web-services response time 30 sec 30 ms
Number of requests 30/sec 20,000/sec
Ad-hoc queries, aggregations, bulk extracts Hours-Days Second-Minutes
Service Availability 20 hours,6 days/week 24 hours,7 days/week
Kx for Utilities Case Study
• Maintains investment in existing system.
• Rapid implementation of Kx technology.
• Low risk or impact on existing system.
• Functionality and availability improved.
Data Feed
Existing Systemof
Record
Time series & Master
Data in kdb+
Kx Technologies
Ad hocQueries
Analysis
Dashboards
Time series data
Real-time or scheduled replication
Query results
Kx for Utilities Case Study
Dashboards for Kx
• Real-time streaming support.
• Easy to use, no programming
experience required.
• Drag-and-drop HTML 5 interface, with
over
35 ready-made components.
• Compatible with all modern browsers.
Dashboards for Kx
Dashboards for Kx
Dashboards for Kx
Köszönöm!
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.
Email: [email protected]
Free 32-bit Download: https://kx.com/download/