Monitoring Real-Time Trading Networks with the nGenius SolutionDetecting and troubleshooting microbursts and out-of-sequence packets for latency-sensitive market data and trade/exchange applications and protocols.

Introduction

In the highly competitive investment services industry, differentiation can often be measured in milliseconds. The firms that can process and analyze the ever-growing volumes of market and trade data the fastest can execute trades more swiftly and pass this advantage on to their customers. In fact, according to Information Week, “a millisecond advantage in trading applications can be worth $100 million a year to a major brokerage firm.” 1 The ability to identify and monitor microbursts, latency and packet loss for critical market data and trade/exchange applications and the protocols associated with them, such as FIX, Multicast, TIBCO and others, is essential to the day-to-day operation of financial exchanges, trading houses, hedge funds, investment banks and other global financial institutions and market exchanges with trading desks.

Algorithmic trading is driving firms to re-architect their trading systems and networks for low latency and optimal performance. Firms are automating trade processes and adding throughput and processing power to their market data and trading systems. Market data delayed by tens of milliseconds may no longer be usable for trading purposes, so designing the network to minimize latency is also essential.

In addition, these networks also support 24x7 access for customers checking account bal-ances, current stock prices, validating orders that have been executed or for brokers and traders viewing their clients’ investment and trading history or researching investment al-ternatives. This increased dependency on the network has brought many IT departments additional responsibilities and formidable challenges in managing, maintaining, and optimizing the performance of their valuable, global networks.

NetScout applies its high-definition CDM technology to identify and monitor application traffic flows, mapping them to the powerful real-time and historical views and reports available in nGenius® Performance Manager. This approach enables nGenius InfiniStream and nGenius Probes, when strategically placed throughout the organization’s network, to collect and deliver rich, actionable information on the utilization and conversations from market trading-specific application traffic. Network professionals can then proactively manage faults, monitor application responsiveness, plan network bandwidth changes, and rapidly troubleshoot problems. Network Considerations

In the wake of widespread consolidations, today’s investment services organization’s network landscape typically spans multiple data centers and geographically dispersed sales offices. These locations, all connected by high-speed networks, host applications affecting high-value, financial services activities, such as updated market trading data, verifying executed orders, accessing customer account information, and performing market research. Additionally, given the critical nature of the data involved, redundant and/or disaster recovery sites are well established to protect against natural or man-made disasters.

At the same time, the networks for investment services organizations is evolving and becoming more complicated, implementing SOA (Service-Oriented Architecture) and Web-based applications in support of customer resource management (CRM) and enterprise resource planning (ERP) activities. Many financial service organizations are rolling out VoIP initiatives triggering other IT projects, such as VPNs,

1 “Wall Street’s Quest to Process Data at the Speed of Light,” Information Week, April 23, 2007.

Network Management Challenges

Maintaining the networks critical to their charter and the applications transported across them are key challenges facing market trading IT departments today. Some specific challenges include:

n Troubleshooting network and application response time issues, latency, packet loss, and out of sequence problems that result in performance degradations across the widely distributed network.

n Maintaining the delivery of time-sensitive applications such as real-time stock price updates that depend on TIBCO used with IP Multicast to ensure reliability, FIX Protocol-based applications for electronic trading or Voice over IP to sales offices.

n Supporting disaster recovery and business continuity plans for all aspects of the company’s business data and voice services throughout headquarters, data centers, and branch office facilities.

To meet the myriad of challenges, investment services companies face, a single unified performance management system that assures the efficient delivery of applications across the distributed network, troubleshoots network problems in real-time and provides unprecedented visibility to control the cost of network operations is required. They need the nGenius Performance Management Solution. Meeting the Need

The nGenius Solution meets the needs of market trading companies by providing the real-time and historical information necessary to keep the network running at peak performance with the lowest possible latency. Using nGenius InfiniStream, nGenius Probes and nGenius Collectors which have been strategically placed throughout the organization’s network to collect and deliver rich, actionable information on the utilization, conversations and packet details from

market trading- or exchange-specific application traffic, network professionals can proactively manage faults, monitor application responsiveness, plan network bandwidth changes, and rapidly troubleshoot problems.

The nGenius Performance Manager provides global market trading companies with

n Real-time monitoring and historical reporting analysis (utilization, response time, hosts and conversations) of market trading applications, such as FIX, OPRA, MDP, PGM, etc.

n Ability to identify IP Multicast groups as unique business applications as well as view the interaction between publishers and groups

n Ability to view TIBCO statistics, including error traffic and retransmissions.

n Real-time microburst alarming

n Robust error detection, such as monitoring out-of-sequence packets/ retransmissions in real time, and the ability to generate an alarm when the gap exceeds a pre-defined value

n Response time analysis which distinguishes network flight times versus application processing times

n Integrated application-level monitoring and analysis with continuous packet capture for robust, post-event data mining

n KPI-to-flow-to-packet approach to problem diagnosis for streamlined troubleshooting that lowers mean time to resolution (MTTR)

Broadest Visibility into Revenue-Impacting Market Trading Applications

The nGenius Solution, built on NetScout’s Common Data Model (CDM) architecture, provides extensive visibility into network resources usage and in-depth application identification and analysis. The nGenius Solution recognizes, discovers, and monitors all the applications in your network including more than 200

MPLS networks or Quality of Service (QoS) prioritization to maintain high-quality voice and ensure the delivery of mission-critical market trading applications. In the network core, to support the speed and volume of data, users, and transactions, market trading firms rely on Gigabit EtherChannel and 10 Gigabit Ethernet links.

Communications between headquarters, data centers, and field sales offices, typically occur over a variety of Wide Area Network (WAN) segments. Local sales office connections may be T1 or E1 circuits, while headquarters and data center connections are more likely OC-3/STM-1 or OC-12/STM-4 speeds.

Business Challenges

To generate revenues and remain competitive, it is essential for today’s market trading companies to ensure that all their products and services are available, at all times and across the global organization. Specific challenges include:

n Maintaining competitive advantage by constantly adapting their trading strategies and increasing the speed of trading in the wake of higher market data volume, quoting in penny increments, and new regulatory demands facing the industry.

n Avoiding delays in quotes, customer information, algorithmic trading, or on-line transactions. Because the network transmits funds, delays translate to lost revenue and unhappy customers.

n Assimilating and incorporating all the changes in users and applications that follow mergers, acquisitions and consolidations.

Monitoring Real-Time Trading Networks with the nGenius Solut ion

well-known applications; complex applications that use a range of ports, such as SAP, Citrix or .NET; custom applications designed specifically for your business; web-based applications identified by URLs; and next-generation communication applications, such as VoIP, video, instant messaging, and peer to peer. Market trading applications monitored “out of the box” by the nGenius Solution include Fix Order Single, FIX Other, MDP, PGM, CTS, CQS, OPRA and Multicast Push and Monitoring. (See Figure 1 for a complete list).

FIX Protocol Support

The Financial Information eXchange (FIX) protocol, which specifies the standards-based method of exchanging trade-related messages and real-time electronic communications, is crucial to many order management and trading systems. With broad adoption as the messaging standard, it is an essential service to the day-to-day operation of applications deployed by virtually every major stock exchange and investment bank with electronic trading, as well as by many global mutual funds, money managers, and investment firms.

The FIX Protocol Interface specification distinguishes the different message types into two categories: Session for Administrative messages, such as heartbeat, test request, and rejects; and Application for Business messages, such as Order Single and Execution Reports. The nGenius Solution collects and analyzes FIX traffic and displays it in two categories:

n FIX Order Single displays the Application for Business message details for the actual order transaction traffic flows and conversations - specifically Order Single and Execution Report.

n FIX Other monitors Session for Administrative messages, such as heartbeat, test requests and rejects.

nGenius Newspaper Report for Response Time over Time and Response Time Distribution over Time.

The nGenius Solution identifies FIX Order Single conversations and traffic flows separately from FIX other

messages as illustrated in this response time report analyzing performance levels in this financial services

network.

Monitoring Real-Time Trading Networks with the nGenius Solut ion

Figure 1Market trading applications monitored “out of the box” by the nGenius Solution include:

Multicast and TIBCO n TIBCO Multicast - 64101n TIBCO Message type 2 - 64102n TIBCO NAK - 64103n TIBCO Keep Alive - 64104n TIBCO NULL - 64105n TIBCO Unicast - 64106n TIBCO Message type 7 - 64107n TIBCO Message type 8 - 64108n TIBCO Other - 64120

FIX Protocoln FIX Order Singlen FIX Other

Other Market Trading Applications n MDP (Market Data Platform)n PGM (Pragmatic General Multicast)n CTS (Consolidated Tape System)n CQS (Consolidated Quote System)n OPRA (Options Price Reporting Authority)

The nGenius Solution uses innovative, deep-packet inspection to track and clas-sify FIX protocol application traffic flows, collecting volume, utilization, users, and conversation statistics as FIX Order Single and FIX Other. It also collects response time metrics for FIX Order Single. These metrics provide the basis for ensuring optimal performance of FIX protocol ap-plications, making it possible for IT staff to easily:

n Evaluate FIX protocol application utilization by recognizing and under-standing existing network patterns and identifying FIX protocol traffic flows and applications running over the network.

n Examine traffic activity and maximize the performance for FIX Order Single specific transactions versus all other FIX traffic by pinpointing application degradations and analyzing application response-time metrics

n Analyze FIX Order Single and FIX Other protocol-based applications and all other applications simultaneously by monitoring and trending the patterns of application behavior side-by-side on segments and virtual circuits in the network.

n Balance the delivery and bandwidth requirements of FIX protocol applications by monitoring and trending traffic patterns and demands to particular hosts and locations for efficient capacity planning purposes.

Support for IP Multicast and TIBCO

Market trading companies use IP Multicast to stream large amounts of data, such as stock price updates, to multiple sites simultaneously and to do so efficiently and in real time. TIBCO is used to ensure reliability of IP Multicast by providing congestion control and bandwidth management and error correction.

IP Multicasts’ one-to-many, many-to-many or many-to-one nature creates an environment where it is difficult to distinguish between conversations. To properly manage the network and accommodate IP Multicast applications, network professionals require a clear picture of how publishers and groups converse. Most performance management systems do not monitor or report on IP Multicast activity and have limited ability to recognize or alarm on microbursts that disrupt streaming IP Multicasts. These microbursts occur in a short span of time

and add additional levels of complexity, making them very difficult to isolate. As a result, IT organizations end up devoting a great deal of time ascertaining the location of a problem, rather than focusing on solving the problem itself.

With the ability to view host group class D IP addresses, NetScout’s nGenius Solution can track IP Multicast conversations, whether they are one-to-many, many-to-many or many-to-one. This ability to monitor individual IP Multicast conversations enables the network team to:

n Identify and track IP Multicast groups and TIBCO protocol messages as unique business applications to identify which applications, servers and locations are most active

n Manage real-time IP Multicast applications along side all other networked applications in order to quickly troubleshoot interactions that could impact service delivery

n View and alarm on critical IP Multicast parameters and other network conditions, including microbursts, in order to reduce performance degradations before they impact users

n View the interaction between publishers and groups, as well as the applications used during the particular conversation

n Join IP Multicast sessions and alarm on TIBCO retransmissions and other error types

n Cost-effectively assure delivery of IP Multicast applications across a globally distributed network, troubleshoot problems in real-time, optimize bandwidth and application delivery, and report to executives, business managers and IT staff worldwide.

nGenius Performance Manager can monitor individual multicast conversations revealing top conversations

from source to group. This information is crucial for capacity planning and resource allocation.

Monitoring Real-Time Trading Networks with the nGenius Solut ion

Support for Other Market Trading Applications

In addition to supporting FIX and Multicast applications, the nGenius Solution supports other applications and protocols that are important to the market trading organizations, including:

n MDP (Market Data Platform) - a multicast distribution platform

n PGM (Pragmatic General Multicast) - a multicast transport protocol used for multicast applications with basic reliability requirements. The protocol is intended for multicast applications that require ordered packets from a source device to multiple recipients.

n CTS (Consolidated Tape System) - multicast protocol for processing of Trade information.

n CQS (Consolidated Quote System) - multicast protocol for processing of quote information.

n OPRA (Options Price Reporting Authority) - provides last sale information and current options quotations from a committee of Participant Exchanges designated as the Options Price Reporting Authority.

The nGenius Performance Management Solution incorporates real-time and historical reporting of all applications - including FIX, other market trading applications, those delivered via IP Multicast and the everyday applications necessary for running any business - eliminating the need for separate products to manage each individual application. Displaying all applications side by side as they compete for bandwidth and resources, not just selected applications, provides total visibility and context into the networking environment. Microburst alarming and monitoring allow the nGenius Solution to pinpoint utilization spikes that last mere milliseconds and affect critical market trading applications and IP Multicast conversations. Therefore, network professionals gain the knowledge and insight they need to

ensure service levels, detect and more quickly troubleshoot degradations, support intelligent capacity planning and report performance results to the organization.

Response Time Analysis

Response time is a primary indicator of network or application service quality and often the primary complaint of end users regarding the network. Tracking and isolating the sources of application response time problems are crucial because in trading transactions, milliseconds are worth millions. The nGenius Solution provides application response time metrics specifically for market trading applications listed in Figure 1. The response time analysis is provided with up to one-minute granularity to aid in establishing financial network services at the appropriate quality levels.

Unique KPI-to-Flow-to-Packet Approach NetScout takes a unique approach to diagnosing network issues, which we call “KPI to Flow to Packet.” (See Figure 2)This approach allows users to view only the data they need to diagnose an issue and alleviates the need to sift through mounds of irrelevant data. In each case, a single right click of the mouse enables you to drill-down into progressively deeper levels of detail:

n KPI - Key Performance Indicators (KPIs) often measure aspects of the user experience, such as errors, jitter, response time, etc., and therefore are useful in providing early detection of performance issues.

n Flows - Once detected, the nGenius CDM architecture allows users to view the applications, conversations, hosts and other flows associated with the specific KPI.

n Packets - When needed, the nGenius System can provide visibility into the individual packet details (bounce charts, packet decodes, etc.) associated with each flow, providing sub-second granularity.

Consider this example: An nGenius Solution microburst alarm emails the network manager to inform him that a WAN link exceeded its capacity for 50ms starting at 14:15.606. Since this alarm is in real-time and it provides the start time and end time for the microburst, when you launch link utilization over time, you can easily zone in on the 50 millisecond slice in question. A right click of the mouse allows you to launch a packet decode to determine what was contributing to the burst and if it caused any transactions to be incomplete.

KPIsSession Errors, VoIP QoS ErrorsApplication Response Time, etc.

FLOWSApplications, Services,

Conversations, Utilization,Volume, etc.

PACKETSHeaders and Payload

WARNINGService Health and

Early Problem Indicators

ISOLATIONAll Application Activityand Complex Service

Relationships

EVIDENCEDefinitive Sub-second Session

and Transaction Details

DATA SET FUNCTION

Figure 2. Illustrated here is the unique KPI-to-Flow-to-Packet approach NetScout Systems takes

to diagnose and resolve network issues.

Monitoring Real-Time Trading Networks with the nGenius Solut ion

Proactive Performance Management with Sophisticated Problem Detection and Alarming

Studies show that in 50% to 75% of the cases, network problems are most often discovered by the network users. When dealing with an application where degradations of milliseconds could severely impact the financial outcome, more effective approaches are necessary. Maintaining network performance for critical trading applications means detecting, diagnosing and rectifying issues that may impede their delivery. In market trading firms, this means identifying, in the earliest stages, problems with packet loss, out-of-sequence packets, latency or delay. The nGenius Solution is well positioned to monitor all the traffic in the network to quickly recognize a number of potential signs of network degradations.

Packet Loss or Out-of-Sequence Packets

NetScout’s Sequence Number KPI watches for gaps in the sequence numbers and generates an alarm when the gap exceeds a predefined value. OPRA, MDP and PGM are examples of protocols that use the Sequence Number KPI to track out of sequence packets. This KPI is also used to monitor packet loss for applications such as MDP that use a fixed off-set for the location of the sequence number (for MDP, offset is 0).

The Sequence Number KPI provides/returns the following information:

n Previous sequence number

n Current sequence number

n Gap in sequence numbers

n Multicast address

n Source address

Microburst Alarms

Power Alarms are time-over-threshold alarms as granular as 1 second that can be set on link utilization, as well as on specific applications’ utilization, such as FIX Order Single. A Microburst Alarm is a more granular version of the Power

Alarm that alerts IT when link utilization has exceeded its threshold for as little as 5 milliseconds. It is useful in identifying microbursts of activity that are otherwise difficult to detect and that can slow transaction rates.

Microburst Alarms are set in nGenius Probes and nGenius InfiniStream for instantaneous alarming. This approach differs from most other products that must process the data before sending the alerts, which can delay notification for as much as 15 minutes. The Microburst alerts include evidence of the top applications, hosts and conversations that contributed to the alarm to aid in identifying the source of the problem.

nGenius® K2

nGenius K2 uses NetScout’s patented, statistical behavior modeling to detect abnormal changes in network and application behavior in order to deliver early warning of performance issues. The product automatically learns the network’s behavior patterns and identifies performance anomalies without the manual configuration and guesswork of setting thresholds. By using variously sized

time slices, nGenius K2 is able to detect different classes of problems, including short-term spikes, sustained shifts, and subtle, long-term performance drifts that are virtually impossible to catch manually.

Because nGenius K2 groups related incidences, it generates fewer, more intelligent alerts, allowing network operations to focus in on only the most critical issues. In addition, by linking the Analytics Alarms with detailed diagnostics information, it facilitates faster diagnosis and more accurate interpretation of performance issues, shortening costly downtime. For these reasons, companies that use Enterprise Management Systems (EMS) such as HP Operations Manager or IBM Tivoli NetView, value nGenius K2.

Network forensics for intermittent problem analysis

Traffic problems can severely disrupt the flow of critical market trading applications, and because they can happen in such a narrow time span, they are often hard to isolate. For investigating this type of intermittent performance issue, the nGenius InfiniStream, with up to 8TB of storage, continuously captures and stores

Microburst alarms are available in real-time, provide the application and conversation details for the

Microburst contributors, and allow you to drill down into the specific packets for the alarm time window.

Monitoring Real-Time Trading Networks with the nGenius Solut ion

a complete packet-by-packet audit trail to support at-a-glance identification and quick resolution of difficult, subtle, and intermittent network problems.

In addition to providing post-mortem packet analysis by capturing network traf-fic 24x7 and storing it to disk, the nGenius InfiniStream simultaneously delivers real-time and historical monitoring for revealing key performance metrics such as traffic and application utilization, network talk-ers, conversations, error conditions, and response times. The tight integration of these two functions makes the nGenius Solution the most robust, functional and feature-rich solution for the market trading industry.

Global capacity planningHaving adequate capacity is particularly crucial for avoiding performance degrading bottlenecks for market trading/exchange applications. The nGenius Solution provides the trended, historical information needed to ensure adequate bandwidth and to optimize the delivery of networked electronic trading services. Baseline and forecast reports enable network engineers to fine-tune traffic flows across the enterprise, understand the normal behavior patterns of various critical applications, and make informed decisions on where to invest resources.

Redundancy and Scalability

Lastly, enterprise-class scalability and redundancy is crucial to the investment services industry. They must maintain operations at all costs and under all circumstances. The nGenius Solution provides the following capabilities for continuous performance monitoring:

n Scalability. Capable of being deployed in a distributive manner, the nGenius Servers work in parallel, rolling up data to a designated master server that provides a complete enterprise-wide view of networked application performance.

n Disaster recovery. The nGenius Performance Manager Standby Server functions as a co-located backup with a main server to protect against a hardware or database failure. Alternatively, it can be deployed at a remote site for seamless business continuity in the event of a disaster.

n Third-party EMS integration. By integrating the nGenius Solution with other industry-leading management and business service solutions often installed in investment services networks, such as HP (Operations Manager, Business Availability Center and Network Node Manager) or IBM (Tivoli NetView, Tivoli Netcool/OMNIbus and Tivoli Enterprise Console), performance monitoring capabilities are expanded, enabling better visibility and control over mission-critical networked and application resources.

n Data Export. In addition, NetScout’s CDM information is available to other third-party applications via the Common Data Export (CDE) functionality. The CDE supports a set of interfaces based on industry

standards that facilitates the attachment of a wide variety of higher-level tools that can be used to distribute information, integrate knowledge into various management platforms, and exercise resource control.

Conclusion

Market trading and exchange applications are the lifeblood of investment services organizations, thus their performance lev-els receive a great deal of attention. The nGenius Solution is uniquely positioned to provide comprehensive visibility into a broad range of market trading applica-tions, as well as their associated hosts and conversations, response time perfor-mance, and errors conditions. In addition, the nGenius Solution provides tightly integrated capabilities for packet capture and decode, early warning alarming, and historical reporting and trending. Armed with evidence from their own network’s day-to-day operations, IT organizations are better able to reduce MTTR as they troubleshoot degradations, respond to emerging utilization and performance problems, and plan for future bandwidth changes.

The ability to view 1-second peaks is an important tool for capacity planning

Monitoring Real-Time Trading Networks with the nGenius Solut ion

About NetScout SystemsNetScout Systems provides advanced network and application service assurance solutions that deliver complete visibility into real-time, packet/flow-based operational intelligence. IT operators at the world’s largest enterprises, government agencies, and service providers use the Sniffer and nGenius solutions to troubleshoot service degradations faster and more efficiently in order to reduce MTTR.

Our world-renowned Sniffer and nGenius solutions include:

n Intelligent Data Sources for high capacity, deep-packet recording and monitoring

n Analysis Software for real-time and historical network and application performance management, troubleshooting, capacity planning, and reporting

n Advanced Intelligence for early detection and in-depth analysis of complex or specialized application services

n Comprehensive, global support, consulting and training services

Corporate Headquarters 310 Littleton RoadWestford, MA 01886-4105Phone: 978-614-4000Toll Free: 888-999-5946www.netscout.com

European Headquarters NetScout Systems (UK) Ltd.100 Pall Mall London SW1Y 5HP United Kingdom Phone: +44 (0)20 7321 5660

Asia/Pacific HeadquartersRoom 105, 17F/B, No. 167TunHwa N. RoadTaipei, TaiwanPhone: +886 2 2717 1999www.netscout.cn

©2008 NetScout Systems, Inc. All rights reserved. NetScout, the NetScout logo, Network General, the Network General logo, nGenius, Sniffer, InfiniStream, Business Container, Business Forensics, NetVigil and Quantiva are trademarks or registered trademarks of NetScout Systems, Inc. Other brands, product names and trademarks are property of their respective owners. NetScout reserves the right, at its sole discretion, to make changes at any time in its technical information and specifications, and service and support programs.

WP0908_01revA

Case Studies

Fixing FIX Visibility

A leading U.S. brokerage firm recognized they had a critical need for more visibility into FIX protocol-based application traffic flows soon after they had publicly introduced a new FIX protocol-based, on-line foreign exchange trading platform, which allowed traders to customize currency pairs and settings, set ticket size limits, and provided streaming spot prices in all major currencies via a “one-click” browser-based system for fast market access.

The brokerage needed more robust, automated analysis of FIX protocol application performance than the simple packet decode would provide so NetScout put the development effort for FIX protocol on the fast track. In the end, adding support for FIX protocol-based application utilization, hosts and conversation tracking, and response time analysis was crucial to supporting this new application, as was the combined monitoring and recording of nGenius AFMon for automated analysis and post-event forensics troubleshooting. Fortunately, NetScout’s CDM architecture is designed for easy extensibility so adding new application and protocol support is a quick and painless process.

Abnormal delay with market trading application

An East Coast financial institution was experiencing intermittent delay problems with one of its market data applications and users were starting to complain. Using the nGenius Solution, they investigated the application response time and saw that the delay was attributable to the server, not the network. In a few short clicks, they were able to determine that all the clients were accessing the same remote server. The server was overwhelmed by the sheer number of the client connections and requests for market data and became a bottleneck. Once network operations identified the problem, the support team was able to redirect the clients to the right source for production data.

Misconfigured clients disrupting WAN service

An intermittent service disruption on the WAN link of a U.S. brokerage firm had network operations wondering why there was an increase in traffic volume and a corresponding increase in packet loss. Using the nGenius Solution, the network operations team was able to view all the applications on that link and their associated volume and was thus able to quickly identify the two applications contributing to the load. Further instigation allowed them to identify the clients associated with the traffic. It turns out a couple of the clients were misconfigured to use a remote server instead of the local server, causing the increased load on the WAN link. Network operations made a quick call to the remote support team who was able to redirect the clients to the local server and restore the WAN to normal operations.

Monitoring Real-Time Trading Networks with the nGenius Solution