1

Optiputer High Performance Transport Protocols & Implementation Plans

Ryan X. Wu, Andrew A. Chien, Eric Weigle, Nut TaesombutCSAG, CSE, UCSD

01/25/2005

2

High Performance Transport Problem

• OptIPuter is Bridging the Gap Between High Speed Link Technologies and Growing Demands of Advanced Applications

• Transport Protocols Are the Weak Link– TCP Has Well-Documented Problems That Militate Against its Achieving High

Speeds– OptIPuter Pursuing Range of Transport Protocols

– Shared, Routed Infrastructure:

– XCP [USC/ISI], – UDT [Gu&Grossman, UIC]

– Provisioned Lambda:

– RBUDP/LambdaStream [He&Xiong&Leigh, EVL, UIC]– GTP [Wu&Chien, UCSD]

– Cluster-based:

– CEP [Weigle&Chien, UCSD]– How to make applications utilize these protocols?

• OptIPuter Transport Protocol Implementation/Integration– Goal: Flexibility of Protocol Choices under Single Application Implementation– Solution: Globus/XIO

3

Optiputer High Performance Transport Protocols

4

RBUDP/LambdaStream

• Suitable for continuous data intensive applications.• Quickly grasp the underlying bandwidth and maintain very low jitter.• More in Eric He’s talk

[He&Xiong&Leigh, EVL/UIC]

5

XCP for High BDP Networks – Shared, Routed Environment

• Systematic Implementation and Evaluation– Build/Test FreeBSD Kernel implementation, Performance Evaluation

– Full protocol specification and mature the protocol

– Work with the community (researchers, applications developers, users, vendors, operators, IETF), To Develop Deployment plan

– Initial Results are Promising: Match Simulations

XCP MeasuredTCP Measured

[Bannister&Falk, USC-ISI]

6

UDT: UDP-based Data Transport Protocol

• UDP-based Data Transfer Protocol -- Successor of SABUL

• Congestion/Flow Control  -- For shared networks  -- Configurable congestion control

• Implementation  -- User space: no need for root privilege or kernel recompilation  -- Socket API (will also provide XIO API soon)  -- Open source: udt.sf.net

[GU&Grossman, UIC]

7

GTP: Group Transport Protocol: Receiver-based Congestion Management

• Request-response for Reliable Data Transfer• Single Flow Adaptation and Capacity Estimation• Receiver-based Flow Scheduling for Fairness and Low Loss Rate

– Balance Concurrent Data Fetching from Multiple Sources – Fair across Varied Sender RTTs– Efficient Transitions under Rapid Changes

R1 R2

Multipoint-to-point contention at receivers

…...Single Flow Control and Monitoring

Centralized Rate Allocation

UDP (data flow) / TCP (control flow)

IP

Applications

GTP

GTP Receiver Architecture

[Wu & Chien, UCSD]

8

CEP: Composite Endpoint Protocol

• We need a High-performance, scalable, robust, flexible, adaptive, easy-to-use mechanism to terminate disproportionately large data transfers.

• User controls transfer through simple sockets-like interface, and transfer constraint specification.

– Can emulate GridFTP, MPI-IO, and parallel filesystem interfaces easily.

• CEP handles actual transfers subject to user constraints via scheduling mechanism

– Uses multiple cooperating nodes.

– Heterogeneous hosts may take part unequally in a connection.

– Scheduler handles failures, fairness, changing resources over time.

– Does the “Heavy Lifting”

– TCP, UDP, FAST, HSTCP, ... used as underlying protocol.

[Weigle & Chien, UCSD]

9

OptIPuter Transport Protocol Implementation/Integration

10

Transport Protocol Deployment Problem

• Issues: – How to make applications utilize OptIPuter transport protocols?– What is the relation between transport protocols and backbone management and

configuration?

• Each transport protocol has its own API and semantics– Different targeted network environments

– Shared, routed network– Private, or QoS-provisioned network

– Different semantics– Blocking/non-blocking– Point-to-point, multipoint-to-point

• Applications need to be modified to use different protocols.– Changes may be complex– Need to maintain multiple versions with different transport protocols

• Provide the same API for all OptIPuter transport protocols– Allow convenient integration with applications– Support further testing/evaluating different transport protocols – Solution: Globus XIO: eXtensive Input Output library

11

Globus/XIO

• Globus XIO user API– Single API/Single Set of semantics.

– Simple open/close/read/write API

• Driver Abstraction– Hides protocol details

– Allows for extensibility

– Drivers can be selected at runtime

• XIO interface for OptIPuter transport protocols

– A simple XIO wrapper (~1000 lines of code) on top of each transport protocol to provide XIO API

– Almost no changes needed to the original protocol implementation

User API

Fram

ework

Driver Stack

Transform

Transform

Transport

XIO Framework

12

XIO Example

• Init the protocol stack– globus_xio_stack_init(&stack, NULL);

• Load drivers– globus_xio_driver_load(“TCP", &txdriver);

– or ‘UDP’, “UDT’’, “GTP”…– globus_xio_stack_push_driver(stack, txdriver);

• Init and set protocol attributes– globus_xio_attr_init(&attr);– globus_xio_attr_cntl(attr, txdriver, GLOBUS_XIO_SET_LISTENING_PORT,

hosts, numhosts);

• Create handle– globus_xio_handle_create(&handle, stack);

• Send/receive data– globus_xio_open(&handle, NULL, target);– globus_xio_write(handle, "hello world\n", 12, 1, &nbytes, NULL);– globus_xio_close(handle, NULL);

13

OptIPuter Communication Software Architecture

• Standard and Custom Communication Protocols are Accessible through a Unique API

• DVC Descriptor Provides Information about Resource Configuration, Naming, and Other Properties

GTP XCP UDTLambda-Stream

DVC Descriptor

High-level IPC Library (e.g., MPI, PVM) DVC. APIs

Socket Wrapper (C and Java)

XIO Interface

DVC Config.

Distributed Applications/ Web Services/Viz. Apps

CEP

14

Implementation Plan

• Current Implementation Status– The TCP, UDP and simple UDT XIO drivers are coming with the

Globus/XIO

– GTP XIO driver is available

• Implementation and Integration Plan– XIO driver for UDT 2.0 will be available early next month

– XIO drivers for LambdaStream and CEP are in progress

– Demonstrate applications running with different protocols via XIO interface

– Further testing and large-scale experiments on different transport protocols

15

For More Information

• A. Falk, T. Faber, J. Bannister, A. Chien, R. Grossman, J. Leigh, Transport protocols for high performance, Communications of the ACM, Volume 46, Number 11, November 2003, pp. 42-49.

• X. Wu and A. Chien, GTP: Group Transport Protocol for Lambda Grids, IEEE Symposium on Cluster Computing and the Grid (CCGrid), April 2004.

• Y. Gu, X. Hong, and R. Grossman, Experiences in Design and Implementation of a High Performance Transport Protocol, (submitted for publication).

• Eric Weigle and Andrew A. Chien, The Composite Endpoint Protocol (CEP): Scalable Endpoints for Terabit Flows Submitted to CCGrid 2005.

• LambdaStream• Globus XIO: http://www.globustookit.org