velocimetrics technology whitepaper 072015

Velocimetrics

Technology Whitepaper An overview of the technology that underpins Velocimetrics

UK Office: Juxon House, 2nd Floor, 100 St Paul’s Churchyard, London, EC4M 8BU USA Office: 230 Park Avenue, 10th Floor, New York, NY 10169 London: +44 (0)203 102 7650 / New York: +1 212 551 1720 / Email: [email protected]

Technology Whitepaper Copyright © 2009-2015 Velocimetrics Ltd. All Rights Reserved. 2

Contents

Introduction ............................................................................................................................. 3 Velocimetrics Technical Overview ........................................................................................ 4 Architecture ......................................................................................................................................... 4 Deployment Topology ......................................................................................................................... 6

Single appliance deployment .................................................................................................................................... 6 Local deployment ......................................................................................................................................................... 6 Distributed deployment .............................................................................................................................................. 6

Instrumenting the Application ............................................................................................... 7 Packet Sniffing .................................................................................................................................... 7 Event Interception ............................................................................................................................... 8 In-Process Data Collection ................................................................................................................. 8 File Processing .................................................................................................................................... 8 Logging Probes ................................................................................................................................... 9 Choice of Probe Type ......................................................................................................................... 9 Probe Management ............................................................................................................................. 9 Anatomy of a Monitoring Event ......................................................................................................... 9 Enriching the Monitoring Event Stream .......................................................................................... 10

The Velocimetrics Expression Language ............................................................................................................ 10 Timestamps and Clock Accuracy .................................................................................................... 11 Configurability and Management Over Time .................................................................................. 12 Real-Time Analysis Services ............................................................................................... 13 Correlation ......................................................................................................................................... 14

Item History .................................................................................................................................................................. 15 Interval Events ............................................................................................................................................................ 15 Data Normalisation .................................................................................................................................................... 15

Association and Tick-to-Trade ......................................................................................................... 16 Connectivity Inferencing .................................................................................................................. 17 Aggregation and Real-Time Slice-and-Dice .................................................................................... 18

Business Aggregations ............................................................................................................................................ 18 Aggregation Calculators .......................................................................................................................................... 18

Alerts .................................................................................................................................................. 19 Item Tracing ....................................................................................................................................... 20 Performance Distribution Analysis ................................................................................................. 21 Historical Analysis ............................................................................................................................ 21 Visualisation .......................................................................................................................... 23 Eclipse RCP ....................................................................................................................................... 24 About Velocimetrics ............................................................................................................. 25


Introduction

Traditional performance monitoring approaches use tools to monitor CPU, memory and network usage, but in today's complex distributed applications, interpreting these statistics to identify and remedy application-level performance and reliability issues is, at best, a black art. Velocimetrics takes a different approach by providing an independent, transparent and detailed view of the business impact of all events taking place across the entire environment by monitoring both within applications and on the network, in real-time. Velocimetrics believes that monitoring the flow of business entities and transactions within applications is the key to understanding overall application performance. Velocimetrics uses innovative, event-driven software architecture to monitor and manage high-performance, high-throughput and highly distributed applications, to provide the key technical and business performance indicators requested by each client to meet their unique requirements. This whitepaper provides a technology and architectural overview of the Velocimetrics solution.


Velocimetrics Technical Overview

Architecture The Velocimetrics architecture has three distinct tiers, as shown in illustration 1:

1. Instrumentation probes, which provide raw monitoring events from the application being monitored 2. A suite of real-time analysis services that process the raw monitoring events, integrate with existing

enterprise management systems and capture live events for historical analysis and comparison with live running

3. A visualisation tier that provides a feature rich GUI console that can be easily configured by clients Monitoring events, captured by the probes, are delivered to the Analysis Services tier using a highly optimised protocol that sends small batches of minimal monitoring events. The Velocimetrics analysis server is designed to cope with dropped and out-of-order/late messages, making it highly suitable for WAN-based global distribution. Analysis Services correlate, pair and analyse the monitoring events. Built as a suite of services around a common high-performance in-memory cache, each service has access to raw monitoring events, or higher- level events and data generated by other analysis services. The services themselves are stateless and use the cache for state storage. The Analysis Service architecture is open and extensible, so that client-specific analysis or intelligent management services can be incorporated via Spring as additional services. Some existing services also provide plug-in APIs to allow custom or business-specific code to enrich the monitoring stream and participate in the correlation and analytical processing. Additionally, Analysis Services also provide service APIs to facilitate console interactions. The separation of the capture and analysis tiers facilitates a highly scalable solution that can support businesses as they grow by taking a clustering approach to the capture element in high volume situations.


Illustration 1: Velocimetrics Logical Architecture

Bus

ines

s Tr

ansa

ctio

n C

onsu

mer

s

!Velocimetrics

Console

Support

Velocimetrics Console

Business

Velocimetrics Console

Operations

Customer support Business Dashboard Operations Desk

Security and M

anagement

Reporting Query API Middleware SNMP

Event Dispatch

Analysis and Alerting

Correlation, Association and Aggregation

Event Collection and Enrichment

Core

Schema

User

Schema

Secu

rity

and

Man

agem

ent

Archival

Alert Automation

In-House Monitoring Archival 3rd Party

Monitoring

Velo

cim

etric

s B

usin

ess

Flow

A

naly

sis

!

In-Process Data

Collection

Packet Sniffing

File Processing

Log Scraping

Bus

ines

s Tr

ansa

ctio

n Fl

ows

!

Recs Accounting

Recs Settlement

Middleware

Back Office

Credit Control

Position Management

Risk ETC

Middle Office

Market GWs

FHs Trading

Middleware

Front Office

Event Interception

Middleware

Client GWs


Deployment Topology The separation of instrumentation probes and analysis services mean that there is great flexibility in deployment of the Velocimetrics system, enabling client requirements to be effectively met. Probes can be placed either close by or embedded in the applications they monitor, whilst the server can either be co-hosted with the instrumentation probes, or separated. Common options include:

Single appliance deployment Velocimetrics provide a packaged appliance option which contains all you need to deploy network-based instrumentation and analysis services in a single 2U server together with network capture support and an on-board database.

Local deployment In local deployment, the application being monitored and the Velocimetrics analysis services are deployed in the same geographic region or data centre. If true low-latency analysis feedback to the application is required (for example for active application decisions based on analysis metrics) then the Velocimetrics analysis server should be closely located with the application or the set of network probes that feed it, to ensure minimum latency in the instrumentation-to-analysis path.

Distributed deployment The application (and therefore the monitoring probes) are deployed at multiple geographic locations. The Velocimetrics analysis server is located in one centre and monitoring events are fed to it from each centre, using internet or private WAN. Velocimetrics is designed to be resilient to out-of-order delivery and dropped data and is therefore suitable for WAN separation of the probes and analysis server. Timestamps are taken at the point of instrumentation and therefore monitoring event propagation delays does not impact the accuracy of measurement. The analysis server provides a single view of the application with the ability to drill-down to different geographic centres.


Instrumenting the Application

Like most monitoring systems Velocimetrics needs raw data to drive it. Generating monitoring data is often referred to as "instrumenting" the application.

Illustration 2: Instrumentation Options

Uniquely, Velocimetrics provides a wide range of techniques to capture application business data and offers an open architecture that is open to extensions, providing clients with visibility into areas where they may otherwise be blind. The choice of technique may vary depending on a range of factors such as time-sensitivity within the application to the practicalities of deploying a workable solution within a client’s infrastructure.

Packet Sniffing By monitoring and decoding network-based conversations between application components, Velocimetrics’ network probes can re-hydrate full business-level content and make it available for analysis in a completely non-intrusive manner. When combined with network taps, packet sniffing has zero impact on the application being monitored, providing a truly independent measure. Velocimetrics network probes perform TCP and UDP packet re-assembly and follow the ISO 7-layer model to segregate decode levels. Velocimetrics provides decoders for major middleware providers including 29West, Rendezvous, Reuters RDMS and many market data and trading exchanges. A decoder SDK is available allowing client-specific protocols to be decoded using Velocimetrics’ network probes. Network probes derive their timestamps from the packet, including external timestamping switches and concentrators, allowing them to participate as required in network time synchronisation schemes.

In-Process Data

Collection

Packet Sniffing

File Processing

Log Scraping

Business Transaction Flows

!

Event Interception

Velocimetrics Business Flow Analysis


Event Interception Velocimetrics can participate passively in publish/subscribe and queue-based messaging technologies including JMS, EMS, TIBCO Rendezvous, MQ, 29West LBM and others, to consume business-level messages for analysis.

In-Process Data Collection An embedded probe is literally embedded in the application software and its API is called at an important point in the software's operation to generate a monitoring event. Velocimetrics embedded probes are provided in the Java, C#/.NET and C++ languages. Embedding is either done by explicit coding, or implicitly using a variety of techniques including JEE configuration, AOP and Spring. The probe API is simple to use, having a single "inject" method that the application calls to pass a business object, such as an order or trade, to the probe. The probe timestamps the injection call and arranges for a monitoring event to be marshalled off the application box in the background. Multiple probes can be installed within a single application process enabling fine-grained monitoring of performance within and between sub-process level application components. Using embedded probes, the act of monitoring is minimally invasive and does not delay the main thread of execution other than to use less than 0.05 microseconds to take an accurate timestamp. Nonetheless, the embedded probe does take some CPU and network bandwidth from the application machine on which it is installed. Velocimetrics provide low garbage collection (GC) variants of their Java and C# embedded probes which allow probe actions to participate fully in object pooling and recycling schemes often found on low-GC applications; this participation being at the cost of a more complex programming paradigm than simple one method injection.

File Processing Still often the technology of choice in some business processing flows, file-based processing via FTP can be monitored using Velocimetrics file probes. These probes monitor the file system for file arrivals and then parse the file contents into first class business objects that are injected into Velocimetrics for analysis. An SDK is provided to allow the speedy development of bespoke content parsers. If appropriate, the files themselves can be monitored and related to their contents, for example where files represent the first stage of batch processing. Such relationships are fully visible and traversable within Velocimetrics itself.


Logging Probes Logging probes monitor log output to extract business data and timestamp information. Probes are provided to plug into the application's slf4j, log4j or commons logging chain through simple configuration. Alternatively, and for other types of logging output, a textual parsing probe is provided to interpret the logging output directly from a file or other text-based logging stream. Timestamps can be extracted from the log content, or from the logging stream as appropriate.

Choice of Probe Type When access to the application internals is possible, embedded probes are often the simplest to install and use. External network probes provide a truly non-invasive means of monitoring "black-box" systems. In instrumenting a given distributed application, Velocimetrics facilitates the mix and matching of embedded and external probes as desired, to effectively meet client requirements, enabling clients to instrument at a fine-grained level within in-house developed software, as well as monitor input and output events to closed parts of the application infrastructure.

Probe Management Probes are configured and managed centrally from the Velocimetrics server they feed. Control features include:

• enablement / disablement of individual probes • how probes logically relate to each other to form a logical flow • what data is extracted from the available business content • how data from a given probe is mapped onto a canonical model for the flow

Anatomy of a Monitoring Event When a probe is set up to monitor a given point within an application, it is typically configured with a logical "point" name and the logical name of the component it is monitoring. For example a point name might be "FIX Interface In" and the component be "FIX Gateway". When a monitoring event is generated by a probe it is in response to the injection of a business object to the probe; for example an order or trade. The probe takes a timestamp immediately (of which more later) and then hands off to a background thread pool to generate the monitoring event and marshal the event off to the Velocimetrics analysis system. In generating the monitoring event, the probe applies a transformer to the injected business object. The role of the transformer is to "skin" the business object's relevant attributes to provide a minimal bag of attributes to pass with the event. These attributes can be simply identifiers or aliases such as an order or trade id, but can also include additional business attributes such as trader or client id, price, quantity or any other data that will be used to drive the Velocimetrics analysis stages.


Enriching the Monitoring Event Stream In capturing data from the application there may be situations where it would be beneficial to manipulate the raw data captured from the probes. This may be to add more information than is present in the captured data, manipulate the captured properties to form properly unique identifiers, or to introduce conditional rules for extraction. Velocimetrics provides both server-side programming hooks to facilitate enrichment and a simple-to-use expression language that allows non-programmers to manage and manipulate raw captured content.

The Velocimetrics Expression Language The Velocimetrics Expression Language VEx allows clients to extract attributes from an injected business object or already captured target attributes and manipulate them using simple scripting. Prior to VEx's introduction in Velocimetrics 4.0 you could use the Velocimetrics Agent Wizard to map selected business object attributes to target attributes in a capture model. For example, if there is a business object class ClientOrder with attributes orderID, clientID, symbol, price etc. these could be captured and assigned to target attributes; for example: orderID -> clientOrderId where clientOrderId is an attribute in the capture model for order flow. VEx lets you do much more than simply extract and assign the orderID. You can use VEx expressions to:

• Extract a substring from a business attribute • Concatenate multiple attributes • Add date components to correlation and alternative ids to ensure uniqueness over time • Optimise property extraction for a given class of business data • Populate capture attributes conditionally; for example, only assign the order id if the type of the order

is NEW.

VEx also supports the addition of user functions to the system, enabling programmers to implement commonly used routines that can then be referenced in VEx expressions. This separation of concerns means that a given Velocimetrics monitoring implementation remains transparent and easy to maintain over time. Here are few simple examples: Extracting a property: myProperty myMap.myMapEntry myProperty.myNestedObject.nestedProperty A string literal: 'A literal string' Extracting a substring: myStringProperty.substring(0,5) Making a property unique for a given day: orderID + ':' + #getUTCDateString() Invoking a user function: #reverseString('dlroW olleH') Conditional extraction of a property: myOtherProperty == 10 ? myProperty : null Conditional extraction using regexp: myOtherProperty.matches('[Bb]ar*') ? myProperty : null


Timestamps and Clock Accuracy For any performance monitoring system, accurate time measurement is important. Exact precision requirements vary significantly, for example a low-latency trading application will require timing accuracy to the micro-second, whilst non time-sensitive applications may only require accuracy of a millisecond or worse. Velocimetrics takes timestamps from the best available information. Embedded Java probes use the nanosecond system timer, whilst C#/.NET and C++ use a specialised timer based on low-level CPU clock and frequency OS calls, circumventing precision issues with system clock APIs on the Windows platform. Such CPU-based timers also suffer from drift over time so Velocimetrics ensures that its timers micro-adjust to follow the system clock, which may be externally coordinated. This gives the best blend of maximum accuracy over the short-term without drift over the longer term. External probes use the timestamp available in the TCP network packet header, or from packet augmentation timestamp information from the new generation of timestamping network switches and aggregators. Coordinating accurate clock time across different machines, which are potentially distributed around the globe, presents its own challenges. Velocimetrics recommends the use of PTP to synchronise machine clocks accurately, but has found that good practical results can be achieved in synchronising clocks to around 1 millisecond of accuracy across machines using suitably configured NTP. If high precision timestamps are required within a distributed environment, Velocimetrics can provide a solution that utilises GPS clock synchronisation and can timestamp information to within a small number of microseconds. Timings measured using multiple embedded probes within a single process will use a single timing and clock, and resolve to the maximum resolution of the nanosecond timer. Velocimetrics network capture agents can read timestamps embedded within network packets by network capture cards or by external aggregation switches. This also provides the capability to timestamp to the microsecond level as close to the live network packets as possible.


Configurability and Management Over Time Business-level monitoring tools such as Velocimetrics live, by necessity, in close proximity with the applications they monitor. Historically such tools have been bedevilled by complex and brittle configuration mechanisms that require significant effort to maintain and manage them over time as the application environment they monitor evolves and changes. Velocimetrics tackles this issue head-on and uniquely provides:

• Wizard-based configuration that leads users through the tasks of setting up instrumentation points, data extraction, and defining correlation rules.

• Separation of logical and physical concerns; for example the mapping of network addresses and ports onto logical monitoring points.

• Dynamic tracking of application change. Embedded probes publish hostname and process ids automatically without the need for manual configuration. All application instances multiply or are deployed on different hardware, Velocimetrics tracks all such changes automatically.

• Simple to learn expression language that integrates with the Velocimetrics configuration wizards and makes even complex construction and manipulation of identifiers and other business content transparent to non-programmers.

Illustration 3: Velocimetrics Wizard-Based Configuration


Real-Time Analysis Services

Velocimetrics analysis services are implemented as a service orientated architecture (SOA) around a highly scalable distributed cache. Raw events, and higher-level analysis events pass through the cache. This architecture is extensible so that additional application-specific analysis or functional services can be integrated. The following sections describe the standard Velocimetrics analysis services.

Illustration 4: Analysis Server

Security and M

anagement

Core

Schema

User

Schema

Secu

rity

and

Man

agem

ent

Distribution Analysis

Alerting Trend Analysis Network Inference

Event Enrichment

Business Event Collection

Aggregation

Correlation

Interval Calculation

!

TCP Event Collection

Association

External Event

Collection

LDA

P A

D

JMX


Correlation and Interval Events The first stage in analysing incoming monitoring events is to correlate them; that is to say, identify all monitoring events associated with the same business entity. Once events are correlated, interval pairings and item histories can be generated.

Correlation In principle correlation is easy: each monitoring probe sees the same business object with the same identifier and uses that identifier to match up all the related events. However, in practice, correlation can be difficult to achieve. Real-world systems often comprise several distinct subsystems that use different identifiers for the same business entity. For example an order may be known by a client-allocated identifier initially and is only later allocated an entirely different identifier by an intermediate application, and then a third identifier by an external service. In some extreme cases, there may be parts of the business flow where no identifier is present at all. The Velocimetrics Correlation Service's job is to effectively manage these complexities and it successfully supports multiple alias identities, cross-reference lookups and heuristic matching using business attributes. Following Velocimetrics’ general design philosophy, each correlation strategy for each segment of the application flow is pluggable and in extreme cases custom correlation schemes can be developed and plugged in. Velocimetrics provides a high-performance distributed in-memory cache to facilitate correlation and unlike pure stream-oriented systems, Velocimetrics can happily deal with out-of-order and missing events without destroying the integrity of the analysis stages and can provide detailed analytics for complex environments in real-time.

Illustration 5: Velocimetrics Correlation


Item History As each monitoring event arrives and is correlated, a composite picture of each business object emerges. Velocimetrics’ terms this an "item history". Using Velocimetrics' property-based approach to business modelling, business attributes present in the incoming event can be extracted, processed and added to the correlated item's history. In real-world applications it can be that the Velocimetrics item history is the one place where all relevant identifiers, aliases and other attributes are present and available in one place. Items can be searched for by id or by indexed properties such as client id.

Interval Events In flow-oriented applications, there is usually a logical flow order, for example A->B->C. By pairing correlated monitoring events, the Velocimetrics Correlation Service generates interval events. Each interval represents the journey of a single item between two nominated points in the application. It would, of course, be possible to generate an interval event for every possible combination of two monitoring points within the application, but more typically there is interest in certain pairings only. In the simple example above, you may be interested in timings from A->B and B->C, but not A->C. The Correlation Service can be configured to generate only specified intervals to minimise loading and event traffic when used with applications that have many monitoring points. Systems that operate with bi-directional action and response flows are also catered for. For example, an order management application may insert an order which passes through points Ain->Bin->C, where C is an exchange that responds by sending an acknowledgement back from C->Bout->Aout. In this type of application topology, the interesting intervals are the inbound legs Ain->Bin and Bin->C, and the return acknowledgement legs C->Bout->Aout. But additionally we are probably also interested in full round-trip intervals such as Ain->Aout, which represents the total time from order injection to acknowledgement receipt at the point of injection. As well as configuring the interval events that are of interest, we can configure the business properties that should be extracted for each of the original monitoring events that form the pair. Making business properties available in the interval event enables these events to feed into higher-level analysis services, which are described later in this whitepaper.

Data Normalisation Capturing business attributes into item history and interval events also provides an opportunity to "normalise" the business data. For example, one part of the system may use strings to represent a percentage attribute, whilst another uses a decimal value. Velocimetrics provides extensive easy-to-use hooks that enable incoming monitoring data to be intercepted and transformed, ensuring a consistent representation across all parts of the monitored system. This is just one example of Velocimetrics' ability to deal with heterogeneity that makes it ideal for complex real-world enterprise environments.


Association and Tick-to-Trade Velocimetrics uses the term “correlation” to mean the process of tying up the same business object, possibly in different guises, at various points of its transit through an application. By its nature, correlation implies a one-to-one relationship between the entities seen.

Velocimetrics also models relationships between different entities for example a basket order and its constituents, or a quote and a trade against that quote. Relationships between entities that Velocimetrics can identify include:

• one-to-one • one-to-many • many-to-one • many-to-many

Velocimetrics terms the process of tying up related business objects “association”. Association allows Velocimetrics to tie together cause and effect, such as market data ticks that drive a quote that results in a trade. In addition to being able to navigate such relationships relatively simply within the GUI, the system delivers timing for the entire relationship graph; for example, time between last market data tick ingress to quote publication from the client gateway.

Illustration 6: Velocimetrics Associations


Connectivity Inferencing Often, the most significant cause of production functional performance issues in distributed systems is misconfiguration. There is often a distinction between how the operational environment was designed to function and how it actually does. Misconfiguration can be a problem such as setting up the wrong parameters for a load balancer, to complex connectivity and scaling issues within more dynamic, "elastic" and grid environments. As the name suggests, the Connectivity Inferencing Service, infers how components are connected to each other using the monitoring event data. Connectivity can be inferred between logical components, geographical locations, physical machines, individual processes and even internal software components. The connectivity graph is fully dynamic. If no traffic is detected on a particular connection within a given time period, the connection is pruned from the graph. Overlaid on the connectivity graph, Velocimetrics computes flow rate, latency and other statistical properties which can be viewed using heat maps, size, ghosting and other visual effects to represent relative statistical values as shown in illustration 7.

Illustration 7: Velocimetrics Connectivity Inferencing GUI


Aggregation and Real-Time Slice-and-Dice There are many ways to slice-and-dice monitoring events to provide real-time summary information. Velocimetrics provides sophisticated on-line analytical processing (OLAP) to enable clients to analyse and aggregate information in real-time, and use those aggregated values to drive other parts of the monitoring system such as alerts. Aggregation can be performed using any attributes present in the monitoring events. For example, analysis of an operational environment aggregation may be sub-divided by process, machine, logical subsystem and location. A client-centric aggregation may sub-divide by, say, type of order, client connection id, client id and client sector id.

Business Aggregations Unlike many monitoring systems, Velocimetrics enables aggregation to be performed using any of the business attributes extracted by the probes, or added as enriched values. An example of business attribute aggregation would be to use retail order feed monitoring to provide a real-time roll up of the total value of sales of an item, by type, manufacturer and sales channel. In a financial trading environment, aggregation could be by trader, desk, market sector or instrument, and provide real-time trade analysis such as Profit & Loss.

Aggregation Calculators Calculators plug into the aggregation environment to operate on the incoming events and provide higher-level analysis. Velocimetrics provides a number of standard calculators such as Count, Max, Min, Moving Max, Moving Min, Sum, Average, Average Flow, Average Latency, Moving Average, Percentage, Standard Deviation and Percentile. Analysis values are generated for each level of the aggregation hierarchy. A new aggregation can be quickly defined using a wizard to specify the attribute names to be used for each level of the aggregation, the calculations to be performed and how calculation results should be formatted. The Velocimetrics calculator framework is open and additional Java calculators can be quickly developed to fulfill application-specific analysis needs. The user's own calculators are automatically integrated within the wizard making them available to end-users.


Alerts Velocimetrics’ Alerts Service monitors both raw events and higher-level analysis value events to detect alert conditions that trigger some action, enabling clients to understand a situation is evolving before it becomes critical and take preventative action. For example, an alert can be triggered if an individual item is not traced fully through the system, or if general moving average latency between the start and end points of an item’s journey takes longer than a given threshold. The Alerts Service also supports plug-in alert condition detectors allowing business-specific alert conditions to be integrated within the solution. Values recorded for historical analysis purposes can also be played back in the correct time frame into the Alerts Service to enable condition detection against past performance. For example, an alert can be triggered if latency in a given part of the monitored system comes within, say, 10% of the 30-day moving average value for that time of day. The ability to evaluate across any combination of live and historical event streams enables highly sophisticated trigger conditions to be detected. The Alerts Service provides damping and threshold cross detection logic to ensure that false or repeated Alerts are not triggered. Alerts can be enabled or disabled manually from the Velocimetrics GUI. Generated alert events trigger visual alert lights on the Velocimetrics console, but also integrate with enterprise alert systems via SNMP, TIBCO Hawk and so on. A Java plugin API is also provided to enable clients to write their own alert integration logic.

Illustration 8: Velocimetrics Alerts


Item Tracing Tracing orders and other flow-orientated business entities around globally distributed systems has traditionally been a time-consuming forensic operational task, often undertaken by trawling log files after a problem has been reported. Velocimetrics Item Tracing enables clients to find the route a given business item has taken simply by typing its correlation id, an application specific id, or other captured property such as client id, even if item is still “in flight”. Item tracing reports the full journey of the item, including processes and machines visited. It's also possible to navigate through the paths taken by associated items. Item tracing can also be linked to the Alerts subsystem to trigger alerts when single items take an unacceptable amount of time to travel along some part of the route, or get lost en-route. Ad-hoc queries allow users to select multiple items based on common properties, interval timings, time windows and so on. The GUI displays scatter and distribution analysis graphs of items selected this way.

Illustration 9: Velocimetrics Item Trace Scatter


Performance Distribution Analysis It is common to look at performance statistically by examining the distribution of timings. In trading it is often the issues with items above the 95th or 99th percentile that cause losses. Velocimetrics provides live, as-it-happens distribution analysis. Any timing value can be configured for distribution analysis. Once configured, standard statistical measures such as standard deviation and percentiles are computed incrementally against the real-time data flow. Visualisation via a live distribution view lets you see how this selected measure is performing. Velocimetrics also provides the ability to click through the distribution to find the individual items behind any particular data point on the distribution.

Illustration 10: Velocimetrics Distribution Analysis

Historical Analysis Part of any monitoring system is its ability to monitor and report on trends over time. Velocimetrics provides comprehensive trend analysis allowing any live statistical value to be captured as a time series. Time series can be used for charting and other long-term trend analysis.


In addition to capturing statistical time series, Velocimetrics’ persistence wizard enables clients to set up rules to capture individual business items. Capture options range from full capture of every item, to the capture of items that meet certain conditions such as having latency greater than the 95th percentile. This latter option is useful in monitoring high volume pricing systems where capturing the performance outliers is the primary goal. Persistence filtering is also aware of associations between items. For example, you may have a high-latency persistence rule for prices and a “persist-everything” rule for trades. When a trade occurs the association graph is traversed to also persist the quote against which the trade was done and persist the price ticks that generated the quote. The Velocimetrics console provides in-built charting with zoom and drill-down capabilities. Time series data in the database can also be extracted for use with standard reporting tools. In order to integrate with an enterprise's existing technology selections, Velocimetrics provides both a time series service abstraction and a hibernate-mapped implementation to an RDBMS.

Illustration 11: Velocimetrics Historical Data Analysis


Visualisation

Velocimetrics provides a rich GUI console that provides an operational view on to Velocimetrics' real-time analysis data and a means of configuring and managing Velocimetrics.

Illustration 12: Example of an exchange user’s view


Eclipse RCP Built on the Eclipse Rich Client Platform, the Velocimetrics client has the same plug-in architecture that has helped make Eclipse the independent development environment (IDE) of choice. It is open to client-specific extensions; for example, the addition of a view panel to interact with a custom analysis service. In common with the rest of the Velocimetrics architecture, the technical protocols by which the client and analysis service tiers interact are pluggable to provide maximum deployment flexibility. The product supports a wide range of protocols out of the box.

Illustration 13: Velocimetrics Configuration Wizard Console features include:

• Connectivity inferencing and flow/latency hotspot identification • Logical and physical connection views with filtering and zoom capabilities • Live aggregator view for real-time OLAP slice 'n' dice • Interactive item trace with scatter and distribution • Alert lights and alert history views • Real-time tick line graphing • Speed dials to highlight important metrics • Performance distribution graphs with drill-through to individual items • Analytical charts with zoom-in/out and drill down capabilities • Drag-and-drop alert configuration • Drag-and-drop chart configuration • Individual business item path tracing • Wizard-based system set-up • Role-based security model controlling which view, edit and configuration capabilities are assigned to

which users. Security integrates with Active Directory and LDAP-based enterprise user/group management systems.


About Velocimetrics

Velocimetrics delivers comprehensive, real-time business insight to the global financial community. Designed by industry experts to meet the specific monitoring and performance analysis needs of complex environments, Velocimetrics provides an independent, transparent and detailed view of the business impact of all activities taking place across each and every system and network as they are happening. By rapidly detecting potential risks, identifying inefficiencies and exposure to financial loss, Velocimetrics instantly alerts the business to issues that could adversely impact success as a situation is emerging, before it’s too late. Velocimetrics is headquartered in London. For more information please email [email protected] or visit www.velocimetrics.com

UK Office: Juxon House, 2nd Floor, 100 St Paul’s Churchyard, London, EC4M 8BU USA Office: 230 Park Avenue, 10th Floor, New York, NY 10169 London: +44 (0)203 102 7650 / New York: +1 212 551 1720 / Email: [email protected]

velocimetrics technology whitepaper 072015

Documents