mellanox scalable hierarchical aggregation and reduction ...€¦ · daemons on systems with...

www.mellanox.com Mellanox Technologies

Mellanox Scalable Hierarchical Aggregation and

Reduction Protocol

(SHARP)™

Release Notes

Rev 1.4.3

Mellanox Technologies 2

Overview

Mellanox Technologies

350 Oakmead Parkway Suite 100

Sunnyvale, CA 94085

U.S.A.

www.mellanox.com

Tel: (408) 970-3400

Fax: (408) 970-3403

© Copyright 2017. Mellanox Technologies Ltd. All Rights Reserved.

Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®,

CoolBox®, CORE-Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®,

GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, Kotura®, Kotura logo, Mellanox CloudRack®, Mellanox

CloudXMellanox®, Mellanox Federal Systems®, Mellanox HostDirect®, Mellanox Multi-Host®, Mellanox Open

Ethernet®, Mellanox OpenCloud®, Mellanox OpenCloud Logo®, Mellanox PeerDirect®, Mellanox ScalableHPC®,

Mellanox StorageX®, Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual Modular

Switch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, Open Ethernet logo, PhyX®, PlatformX®,

PSIPHY®, SiPhy®, StoreX®, SwitchX®, Tilera®, Tilera logo, TestX®, TuneX®, The Generation of Open Ethernet

logo, UFM®, Unbreakable Link®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are registered

trademarks of Mellanox Technologies, Ltd.

All other trademarks are property of their respective owners.

For the most updated list of Mellanox trademarks, visit http://www.mellanox.com/page/trademarks

NOTE:

THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED

DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY

KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT

USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST

ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY

QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES

CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE

HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE

IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND

NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR

ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL

DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF

SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)

HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,

OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE

PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH

DAMAGE.

Rev 1.4.3 Mellanox Technologies 3

Overview

Table of Contents

Document Revision History .................................................................................................................. 5

1 Overview .......................................................................................................................................... 6

1.1 Packages ................................................................................................................................ 6

1.2 Prerequisites ........................................................................................................................... 6

1.3 Supported OS and Platforms .................................................................................................. 6

2 Changes and New Features ........................................................................................................... 8

2.1 Parameters Changes .............................................................................................................. 8

3 Limitations ....................................................................................................................................... 9

4 Change Log History ...................................................................................................................... 11

4.1 Parameters Changes ............................................................................................................ 12

4.2 API Updates .......................................................................................................................... 13

4 Mellanox Technologies Rev 1.4.3

Overview

List of Tables

Table 1: Document Revision History ....................................................................................................... 5

Table 2: Supported OS and Platforms ..................................................................................................... 6

Table 3: Limitations ................................................................................................................................. 9

Table 4: Change Log History .................................................................................................................. 11


Overview

Document Revision History

Table 1: Document Revision History

Revision Date Description

Rev. 1.4.1 Oct 31, 2017 Initial version of this release


Overview

1 Overview

Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™

technology improves upon the performance of MPI operation, by offloading collective

operations from the CPU to the switch network, and eliminating the need to send data

multiple times between endpoints. This innovative approach decreases the amount of data

traversing the network as aggregation nodes are reached, and dramatically reduces the MPI

operations time. Implementing collective communication algorithms in the network also has

additional benefits, such as freeing up valuable CPU resources for computation rather than

using them to process communication.

1.1 Packages

Package Version

MLNX OFED 4.2-x.x.x

HPC-X 2.0.x

UFM (Aggregation Manager v1.4.1 only) 5.9.5

1.2 Prerequisites

Name Version

Externally managed Switch-IB 2 Firmware version: 15.1460.0162 or later

MLNX OS 3.6.4008

Subnet Manager 4.9 (MLNX OFED 4.2-x.x.x ) or later .

4.7-4.8 require additional configuration in Aggregation

Manager.

1.3 Supported OS and Platforms

Table 2: Supported OS and Platforms

Distro Platform Kernel

RHEL 6.1 x86-64 2.6.32-131.0.15

RHEL 6.2 x86-64 2.6.32-220

RHEL 6.3 x86-64 2.6.32-279

RHEL 6.4 x86-64 2.6.32-358

RHEL 6.5 x86-64 2.6.32-431

RHEL 7.0 x86-64 3.10.0-123

RHEL 7.2 x86-64 3.10.0-327

RHEL 7.2 ppcle 3.10.0-327

RHEL 7.3 Arm 4.5.0-15.el7.aarch64

RHEL 7.4 x86-64 3.10.0-693

Fedora14 x86-64 2.6.35.6-45


Overview

Distro Platform Kernel

Fedora16 x86-64 3.1.0-7

Fedora17 x86-64 3.3.4-5

Fedora18 x86-64 3.6.10-4

Fedora24 x86-64 4.5.5-300

Fedora26 x86-64 4.11.8-300

SLES 11 SP1 x86-64 2.6.32.12-0.7

SLES 11 SP2 x86-64 3.0.13-0.27

SLES 11 SP3 x86-64 3.0.76-0.11

Ubuntu12.04 x86-64 3.2.0-37

Ubuntu13.10 x86-64 3.11.0-12

Ubuntu14.4 x86-64 3.13.0-24

Ubuntu15.10 x86-64 4.2.0-16

CentOS6.3 x86-64 2.6.32-279

CentOS6.0 x86-64 2.6.32-71


Changes and New Features

2 Changes and New Features

Table 2: Changes and New Features

Component Category Description

Aggregation

Manager (sharp_am)

Fabric extension Enabled adding/replacing new non-root aggregation nodes

without restarting Aggregation Manager.

Aggregation

Manager (sharp_am)

Fabric extension Optimized root placement on tree topologies (improved the

location of Mellanox SHARP trees roots on the tree topologies).

Aggregation

Manager (sharp_am)

Resiliency Added the option to notify running jobs about Aggregation

Manager (sharp_am) restart.

Mellanox SHARP

Daemon (sharpd)

Out-of-the-box

improvement

Added Systemd support.

Mellanox SHARP

Daemon (sharpd)

Out-of-the-box

improvement

Added Socket-Based-Activation support for Mellanox SHARP

daemons on systems with Systemd.

Mellanox SHARP

Daemon (sharpd)

Out-of-the-box

improvement

Removed static binding to network IP interface in Mellanox

SHARP daemons.

2.1 Parameters Changes

Parameter Component Description

accumulate_log sharp_am/sharpd New Parameter: Accumulates log file over multiple

sessions. If set to FALSE and log rotation is disabled,

the log file is truncated on startup

Note: No support on update runtime.

syslog_verbosity sharp_am/sharpd New Parameter: Syslog verbosity level: 1 - Errors, 2 -

Warnings. Default value is "1".

Note: Supported on update runtime.

persistent_dir Sharp_am New Parameter: Path to persistent data directory.

Note: No support on update runtime.


Limitations

3 Limitations

Table 3: Limitations

Internal Ref. Issues

1179747 Description: Changing smx_sock_interface configuration parameter is not supported.

Workaround: N/A

Keywords: Mellanox SHARP Daemon

1074862 Description: Fabric extension: adding new non-root switches or non-root switch

replacement are currently not supported by the Aggregation Manager for tree topologies.

(Fat-Tree, Quasi-Fat-Tree)

Workaround: Restart Aggregation Manager after the Subnet Manager completes fabric

reconfiguration followed by the fabric changes.

Keywords: Aggregation Manager

- Description: Aggregation Manager High Availability is currently not supported in

HPCX/MLNX OFED packages. Therefore, only a single instance of Aggregation

Manager can run in the IB fabric.

Workaround: Use Aggregation Manager in UFM.


- Description: Aggregation manager should run on the same Host where the Master Subnet

Manager (SM) is running.

Workaround: N/A


- Description: In case of HPCX/MLNX OFED packages, upon Subnet Manager

handover/failover, another instance of Aggregation Manager should be started on the Host

where the new Master SM is running

Workaround: Use Aggregation Manager in UFM.


- Description: Aggregation Manager should be started after completion of fabric

configuration by the Subnet Manager.

Workaround: N/A


- Description: Fabric extension: adding new switches or switch replacement are currently

not supported by the Aggregation Manager for hypercube topologies

Workaround: Restart Aggregation Manager after the Subnet Manager completes fabric

reconfiguration followed by the fabric changes.


- Description: Fat-Tree, Quasi-Fat-Tree and Hypercube topologies are supported by the

Aggregation Manager.

Workaround: N/A

Keywords: Fabric Topology


Limitations

Internal Ref. Issues

- Description: Only IB fabrics where all compute nodes are connected to Mellanox

SHARP capable switches (Switch-IB 2) are supported by the Aggregation Manager.

Workaround: Manually configure mapping between the compute port and the

Aggregation Node.

Keywords: Fabric Topology

- Description: Upon changes in configuration file beyond parameters in 3.3, Aggregation

Manager should be restarted to deploy new configuration.

Workaround: N/A

Keywords: Configuration

1071876 Description: Rebooting the switch the SM port is connected to could affect Aggregation

Trees.

Workaround: Restart Aggregation Manager



Change Log History

4 Change Log History

Table 4: Change Log History

Feature/Change Description

Rev. 1.3

Aggregation Manager (sharp_am)

Out-of-the-box improvement Added support for extended fabric format (SMDB). Note: This

requires Subnet Manager 4.9 or later.

Fabric extension Compute hosts can be added/replaced without Aggregation Manager

restart.

Configuration Added the ability to update some configuration parameters in

runtime without application restart.

Mellanox SHARP Daemon (sharpd)

Out-of-the-box improvement Removed static binding to IB port.

Configuration Added the ability to update some configuration parameters in

runtime without application restart.

Rev. 1.2

Aggregation Manager (sharp_am) Added support for IB fabric events (flapping links, switch/host

reboot)

Resiliency: Mellanox SHARP Tree QP Recovery

Added support for Hyper-cube topology (needs OpenSM 4.8.1 or

later)

HCOLL Added new non-blocking API for Mellanox SHARP collectives

Job Scheduler Added new API for integration with Job Scheduler

UFM Enabled Aggregation Manager integration with UFM

Rev. 1.1

HCOLL Enables UD MCAST result distribution

Enables multiple group leaders per compute nodes

Delivers error to an application

Enables Mellanox SHARP Group trim

Added support for ppcle platform

Rev. 1.0

MPI 2.x Barrier and Allreduce collective operations using Mellanox SHARP

protocol are supported in Open MPI, MPICH, ScalableSHMEM

with HCOLL library.

HCOLL Enables running of Mellanox SHARP collective with the mpirun

utility.

For the complete list of flags that can be used when running

Mellanox SHARP software, please refer to the SHARP Deployment

Guide.


Change Log History

4.1 Parameters Changes


Rev. 1.3

ib_mad_timeout sharp_am Removed

ib_mad_retries sharp_am Removed

hyper_cube_coordinat

es_file

sharp_am Deprecated (with Subnet Manager 4.9

and later).

root_guids_file sharp_am Deprecated (with Subnet Manager 4.9

and later).

ib_dev sharpd Removed

log_verbosity sharp_am /

sharpd

Modified behavior: Added the option to

update on runtime.

lst_file_timeout sharp_am Modified behavior: Added the option to

update on runtime.

lst_file_retries sharp_am Modified behavior: Added the option to

update on runtime.

generate_dump_files sharp_am Modified behavior: Added the option to

update on runtime.

max_quota sharp_am Modified behavior: Added the option to

update on runtime.

default_quota sharp_am Modified behavior: Added the option to

update on runtime.

span_all_agg_nodes sharp_am New Parameter: Generate trees that span

all possible aggregation nodes

Relevant only if "topology_type" is tree.

Rev. 1.2

Environment variable:

SMX_SOCK_PORT

AM / SD Replaced by smx_sock_port

parameter

Environment variable:

SMX_SOCK_INTERFACE

AM / SD Replaced by smx_sock_interface

SHARP_COLL_SHARP_ENA

BLE_MCAST_TARGET

HCOLL Replaced by

SHARP_COLL_ENABLE_MCAST_TARG

ET

smx_sock_interface sharp_am /

sharpd

New Parameter: Network interface to be

used by SMX.

Default: empty string - Use first interface

found in UP state

smx_sock_port sharp_am /

sharpd

New Parameter: The external port to be

used by SMX. Default - 6126

lst_file_timeout sharp_am New Parameter: Length of timeout in

seconds between attempts to load the

LST file. Default - 3 seconds.

lst_file_retries sharp_am New Parameter: Max number of retry

attempts when loading the LST file and


Change Log History


encountering "No such file" errors.

Default - 0 meaning no retries.

log_max_backup_files sharpd New Parameter: Number of backup log

files. Used for log rotation

log_file_max_size sharpd New Parameter: Maximum size of a log

file, in MBs. If value is 0, log rotation

isn't used

mgmt_mode sharpd New Parameter: When running in

managed mode, SHARPD expects

notifications from the Resource manager

(Job scheduler). The possible values are:

0 - Unmanaged mode; 1 - Managed mode

smx_sock_backlog sharpd New Parameter: Defines the maximum

length to which the queue of pending

connections for the SMX listen socket

may grow

group_allocate_timeo

ut

sharpd New Parameter: Maximum time [in

milliseconds] to wait for group allocation

transaction to complete.

4.2 API Updates

API Category Description

Rev. 1.2

sharp_coll_do_allreduce_nb HCOLL Changed

sharp_coll_do_barrier_nb HCOLL Changed

sharp_coll_do_reduce_nb HCOLL Changed

sharp_coll_req_test HCOLL Changed

sharp_coll_req_wait HCOLL Changed

sharp_coll_req_free HCOLL Changed

sharp_job_quota Job Scheduler Added

mellanox scalable hierarchical aggregation and reduction ...€¦ · daemons on systems with...

Documents