August 20, 2006

Delivering Low Latency With IBSUSE® Linux Enterprise Real Time

Moiz KohariSuse LabsDirector Real-Time Systems

© Novell Inc. All rights reserved

2

Agenda

• Why Real-Time• What Is Real-Time• Benefits • IB – End To End QoS• Future Development/Roadmap

© Novell Inc. All rights reserved

3

Why Companies Want Real Time

• Faster, predictable data delivery• Eliminate data latency • Applications that must always respond in a finite period of time

– Market data distribution, algorithmic trading, process control, data acquisition...

• Some distributed clusters benefit from highly accurate time synchronization, low-latency, and deterministic I/O

• Some applications, like Visualization, Simulation and VOD, need consistent processing times for smooth graphics displays

© Novell Inc. All rights reserved

4

What is SUSE® Linux Enterprise Real Time?• An enriched distribution of the SUSE Linux kernel from Novell® that provides guaranteed performance in time-critical environments.

• An industry-standard real-time version of Linux for x86 architectures providing the following key features:

– CPU Shielding: Dedicated resources (such as cpu, memory, fabric) for high priority processes

– Kernel Preemption: Fast response to external events, optimized kernel critical sections (aggressive lock breaking)

– Priority Inheritance: Solves priority inversion problem for mission critical processes

– Frequency Based Scheduling: Mechanism for coordinating multiple execution threads within predefined CPU cycles

© Novell Inc. All rights reserved

5

When Is Real-Time The Appropriate Platform?

• Use standard kernel for throughput sensitive workloads and real-time kernel for latency sensitive workloads (for example):

• When fair scheduling algorithms are required across compute resources (educational deployments – time sharing)

• Web farms serving FIFO based requests

• Enterprise mail servers

• When certain mission/business critical jobs demand the fastest response time possible:

•Missile Defence Systems

•Air traffic control

•Trading systems

•Process control applications

Standard Kernel:

Real-Time Kernel:

© Novell Inc. All rights reserved

6

What is Meant by Interrupt Response Time?

Standard Linux Kernel Measured 44,759,417 samplesMax latency: 92.3 ms44374681 samples < 0.1ms (99.140 %)219672 samples < 0.2ms (99.630 %)93496 samples < 1.0ms (99.843 %)44452 samples < 10.ms (99.939 %)16188 samples < 30.ms (99.975 %)10761 samples < 60.ms (99.999 %)166 samples < 90.ms (99.999 %)1 samples < 100ms (100%)Interrupts fire at 490 us /dev/rtc

SUSE Linux Enterprise Real Time Kernel Measured 28,800,882 samplesMax latency: 27 usMin latency: 11 usAverage latency: 11.3 us28808870 samples < 0.02 ms (99.99999 %)12 samples < 0.03 ms (100 %)

Kernel.org 2.4.00 – rcl Interrupt ResponseSUSE Linux Enterprise Real Time

© Novell Inc. All rights reserved

7

SLERT Performance

1 1 1 2 2 2 4 4 4 8 8 8 16 16 160

50

100

150

200

250

300

Ingres Database Performance

RHEL4

SLERT

Threads Generating System Load

Tim

e I

n S

ec

on

ds

© Novell Inc. All rights reserved

8

Leveraging IB To Double Wombat Feed Handler Capacity

Server 1Wombat Feed

Handler

Voltaire 9096Switch

Server 2Wombat Client

Server 3Wombat Client

• 4 Consumer Application Connections

• Subscribing to 100,000 Symbols Each

• 50,000/second per connection

➔ 2 Consumer Applications/Client➔ Handling A Total Of 200,000 msgs/sec

© Novell Inc. All rights reserved

9

Real Time Tests at a Major Investment Bank

3 4 5 6 7 8 9 10 11 12 13 140

100

200

300

400

500

600

Archipelago Data – Using The Flood Test

3 4 5 6 7 8 9 10 11 12 13 14

0

50

100

150

200

250

300

350

400

450

500

99.9th Percentile Latency Implication Flood Test

RHEL4 SLERT

3 4 5 6 7 8 9 10 11 12 13 14

0

100

200

300

400

500

600Inet

© September 08, 2005 Novell Inc, Confidential & Proprietary

10

InfiniBand End-to-end QoS

Port

Schedule Queue (shaper) arbitration

QP to Schedule Queue mapping

SQ0SQ0 SQ1SQ1 SQ2SQ2 SQ3SQ3 SQ4SQ4 SQ5SQ5 SQ6SQ6 SQ7SQ7 SQ8SQ8 SQ9SQ9 SQNSQN…

Link Arbitration(dynamic)

QP0 QP1 QP2 QP3 QP4 QP5 QP6 QP7 QP8 QP9 QPi QPi QPi QPi QPi QPi QPi QPi QPj QPj QPj QPj QPj QPj QPj QPj QP16M…

QP0QP0 QP1QP1 QP2QP2 QP3QP3 QP4QP4 QP5QP5 QP6QP6 QP7QP7 QP8QP8 QP9QP9 QPiQPi QPiQPi QPiQPi QPiQPi QPiQPi QPiQPi QPiQPi QPiQPi QPjQPj QPjQPj QPjQPj QPjQPj QPjQPj QPjQPj QPjQPj QPjQPj QP16MQP16M…

16 Million QPs

Granular

Scalable

End-to-end

Hardware-based

Deterministic

© September 08, 2005 Novell Inc, Confidential & Proprietary

11

Development/Roadmap

Real-Time Kernel

Fabric QoS

Ultra-Responsive, Real-time Clusters

QoS-Based Virtual Compute Platform

© September 08, 2005 Novell Inc, Confidential & Proprietary

12

Xen and IB

●Guest needs the performance of direct access to HCA for data transfer●Guest needs to appear as endpoint on ib fabric●XEN-IB prototype of VMM bypass for guest domains

● only supports Unreliable Datagram service type● service discovery not supported● no support for QP1/0 access at guest

© September 08, 2005 Novell Inc, Confidential & Proprietary

13

Xen and IB

SACM VERBS

mthca driver

ib core

Domain 0 Guest Domain

libmthcalibibverbs

user app

SRP IPoIB

Hardware

XEN

kernel kernel

SACM VERBS

ib core

libmthcalibibverbs

user app

HCA providervendor specific

driverGuest ib

resouces proxy

HCA providerprovider-

frontprovider-

back

madfrontmadbackMAD MAD

SRP IPoIB

Unpublished Work of Novell, Inc. All Rights Reserved.

This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General Disclaimer

This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. Novell, Inc., makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.