"an introduction to kx technology: a big data solution" chris leckey, a data scientist...

An Introduction to Kx Technology: A Big Data Solution

8th December 2016

Chris Leckey Kx Systems

Est. 1993, California, USA

Founder's Concern: Volume

Widely adopted in financial services over two decades

Global company, Large user community

Other industries: Energy, Pharmaceuticals, Telecommunications, Sport

About Kx Systems

Kx Clients & Industries

About the Technology

• Integrated in-memory, columnar

database & programming system

• Map-Reduce built-in

• Parallelism

• Time as native

• Streaming, real-time and historical

data.

• Lightweight Executable (800kB)

• Low TCO: Low O&M costs Standard OS & Hardware

q at a Glance

• Functional array processing language

• Vector based

• Descended from Ken Iverson's APL

• Right to Left Execution

• Low Latency:

Interpreted, not compiled

Runs inside database

• Terse - Get more from less code

Project Euler Problem 1 in qProject Euler Problem 1 in q

Project Euler Problem 1 in JavaProject Euler Problem 1 in Java

44 Characters44 Characters

233 Characters233 Characters

q – Simple Query Syntax

6

sym price sizeJPM 34.51 105

GE 71.23 97BP 18.69 23

JPM 34.53 16

JPM 34.55 85BP 18.72 81

GE 71.15 48

sym price size

JPM 34.51 105

JPM 34.53 16

JPM 34.55 85

Trade Table Query

Result

Kx Performance Comparison

• Dataset: NYSE TAQ 5000 days. 1.1 Trillion quotes. 65 Billion trades. 100TB raw text.

• Machine: 16 core 256GB. Date partitioned. Symbol indexed.

• All queries in RAM (kdb+ 3.x March 2015)

• Time to query for smallest day trades. 2003.09.10 35M quotes. 5M trades.

Kx Performance

“Kx’s kdb+, running on Dell EMC DSSD D5 and PowerEdge servers, has established performance records while testing on the largest STAC-M3 data scale so far.”

Peter Lankford, STAC Director

• The Independent Electricity System Operator (IESO)• Largest multi-tenant meter data processing service (4.7 million meters)• > 300 billion rows of data, growing by 120 million per day• Selected and applied Kx to support:

Analytics not previously possible On demand access to all of the data by utilities and their customers 24x7x365 availability and near real-time synchronization with transaction

system.

Kx for Utilities Case Study

Service PreviousSolution

KxSolution

Web-services response time 30 sec 30 ms

Number of requests 30/sec 20,000/sec

Ad-hoc queries, aggregations, bulk extracts Hours-Days Second-Minutes

Service Availability 20 hours,6 days/week 24 hours,7 days/week

Kx for Utilities Case Study

• Maintains investment in existing system.

• Rapid implementation of Kx technology.

• Low risk or impact on existing system.

• Functionality and availability improved.

Data Feed

Existing Systemof

Record

Time series & Master

Data in kdb+

Kx Technologies

Ad hocQueries

Analysis

Dashboards

Time series data

Real-time or scheduled replication

Query results

Kx for Utilities Case Study

Dashboards for Kx

• Real-time streaming support.

• Easy to use, no programming

experience required.

• Drag-and-drop HTML 5 interface, with

over

35 ready-made components.

• Compatible with all modern browsers.

Dashboards for Kx

Dashboards for Kx

Dashboards for Kx

Köszönöm!

Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.

Email: cleckey@firstderivatives.com

Free 32-bit Download: https://kx.com/download/