high performance cluster computing architectures and systems

36
High Performance Cluster Computing Architectures and Systems Hai Jin Hai Jin Internet and Cluster Computing Center Internet and Cluster Computing Center

Upload: tuan

Post on 12-Jan-2016

29 views

Category:

Documents


0 download

DESCRIPTION

High Performance Cluster Computing Architectures and Systems. Hai Jin. Internet and Cluster Computing Center. Multiple Path Communication. Introduction Heterogeneity in Networks and Applications Multiple Path Communication Case Study Summary and Conclusion. Introduction (1). - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: High Performance Cluster Computing Architectures and Systems

High Performance Cluster Computing

Architectures and Systems

Hai JinHai Jin

Internet and Cluster Computing CenterInternet and Cluster Computing Center

Page 2: High Performance Cluster Computing Architectures and Systems

2

Multiple Path Communication

Introduction Heterogeneity in

Networks and Applications

Multiple Path Communication

Case Study Summary and

Conclusion

Page 3: High Performance Cluster Computing Architectures and Systems

3

Introduction (1)

Communication is major factor of performance in network-based computing

Utilize all available network resources within a system Substantial benefits in simultaneously

exploiting multiple independent communication paths between processors

Heterogeneous network environment Single application execution Different network - different performance

characteristic

Page 4: High Performance Cluster Computing Architectures and Systems

4

Introduction (2)

Performance-based path determination (PBPD) Exploiting multiple physical communication paths &

multiple communication protocols within a single parallel application program to reduce communication overhead

Different physical networks & protocols have different performance characteristics

Different types of communication within an application might be best suited to one of several different communication paths

bulk data transfer (image data): – high-bandwidth communication

short control messages (synch, acknowledgement): - low-latency communication

Page 5: High Performance Cluster Computing Architectures and Systems

5

Introduction (3)

Performance-based path determination (PBPD) Performance-based path selection (PBPS)

Dynamically select the best communication path among several in a given system

Performance-based path aggregation (PBPA) Aggregate multiple communication paths into a single

virtual path for sending an individual message in an application

Each independent path simultaneously carries a fraction of a single message

Useful in bandwidth-limited situations Control is based on message size, message type, or

network traffic load

Page 6: High Performance Cluster Computing Architectures and Systems

6

Heterogeneity in Network and Applications

Heterogeneity inherent interprocessor communication

Different types of messages latency-limited bandwidth-limited

Page 7: High Performance Cluster Computing Architectures and Systems

7

Varieties of Communication Networks

WAN (1 Mbps) MAN (100 Mbps) SAN (more than 1 Gbps) Ethernet (10 Mbps – 1000 Mbps) ATM (155 Mbps) FDDI (100 Mbps) HiPPI (800 Mbps – 1.6 Gbps) Infinitband (10Gbps)

Page 8: High Performance Cluster Computing Architectures and Systems

8

Exploiting Multiple Communication Paths

Messages size communication groups priority level

Different type of network services to support different types of message communication

Network services connection oriented connectionless

Both are logically sufficient for the implementation of any communication pattern

Each offers performance & programming benefits for some classes of applications

Page 9: High Performance Cluster Computing Architectures and Systems

9

Benefits to Support Multiple Communication Paths

Efficient network utilization Most appropriate communication paths to be

used Alternative network protocols

Connection oriented / connectionless Robust communication

Multiple networks provides extra reliability Network load balancing Quality of service

Multiple paths can support diverse QoS requirements

Page 10: High Performance Cluster Computing Architectures and Systems

10

Multiple Path Communication

A single application is likely to require several different types of messages for communication Different types of messages may be better

suited to a different type of communication mechanism

Page 11: High Performance Cluster Computing Architectures and Systems

11

Performance-Based Path Selection (PBPS)

Useful when one provides better performance in one situation while the other is better in another situation

Possible to dynamically select the appropriate communication path for a given communication event

f1(m1) = t1 f2(m2) = t2 fPBPS(m) = Best[fi(m)], where (i= 1.. N)

Page 12: High Performance Cluster Computing Architectures and Systems

12

When sending a message of size mi, performance-based path selection (PBPS) uses the lower latency curve among f1(mi) and f2(mi)

Page 13: High Performance Cluster Computing Architectures and Systems

13

Performance-Based Path Aggregation (PBPA)

Can be applied when different paths show similar characteristics 2 nearly identical networks

aggregate 2 networks into a single virtual network bandwidth will be nearly twice

Divide – transmit - aggregate Important consideration

determine the size of the submessages

fPBPA(m) = fi(mi), where (i= 1.. N), (m = mi)

Page 14: High Performance Cluster Computing Architectures and Systems

14

When using performance-based path aggregation (PBPA) with two networks, a message of size m1+ m2 is split into two submessages

such that messages of size m1 and m2 are sent over networks f1(mi) and f2(mi) simultaneously

Page 15: High Performance Cluster Computing Architectures and Systems

15

PBPD Library

Custom library whose main feature is the support of multiple communication paths in a single application program

Based on common TCP layer Add integer field ‘length’ tells message size

Used by PBPA if a message is too small to segment Handle the multiplexing of different TCP

connections Use UNIX select system call with appropriate table

lookups

Page 16: High Performance Cluster Computing Architectures and Systems

16

Implementation and Protocol Hierarchy of the PBPD

Communication Routines

Page 17: High Performance Cluster Computing Architectures and Systems

17

Case Study: Multiple Path Characteristic

Communication type in parallel application program point-to-point, collective

4 Silicon Graphics Challenge L shared-memory multiprocessors. 4 or 8 R10000 processors at 196MHz per node 10Mbps Ethernet and 266Mbps Fiber Channel

Page 18: High Performance Cluster Computing Architectures and Systems

18

Multiple Heterogeneous Network Configuration used in the

Experiments

Page 19: High Performance Cluster Computing Architectures and Systems

19

Point-to-point and Broadcast Characteristics of Ethernet using the TCP and UDP

Communication Protocols

Page 20: High Performance Cluster Computing Architectures and Systems

20

Example of a Broadcast Operation of a Separate Addressing Method using the PBPA

Technique

Page 21: High Performance Cluster Computing Architectures and Systems

21

Point-to-point Characteristics of Ethernet and Fibre Channel using the TCP Protocol

Page 22: High Performance Cluster Computing Architectures and Systems

22

Broadcast Characteristics of Ethernet and Fibre Channel using the TCP Protocol

Page 23: High Performance Cluster Computing Architectures and Systems

23

Case Study: Communication Patterns of Parallel Applications

Performance of PBPD at application-level depends on the communication patterns of the specific application being executed

Page 24: High Performance Cluster Computing Architectures and Systems

24

Parallel Benchmark Programs Tested

Programs Description

CG Conjugate gradient

MG Multigrid

IS Integer sort

Filter Smoothing (averaging) filter

Gauss Gaussian elimination

Hough Line recognition algorithm

Kirsch Image processing

TRFD Two-electron integral transformation

Warp Spatial domain image restoration

BT Simulated CFD application using Block tridiagonal solver

LU Simulated CFD application using LU solver

SP Simulated CFD application using Pentadiagonal solver

MICOM Miami isopycnic coordinate ocean model

Page 25: High Performance Cluster Computing Architectures and Systems

25

Case Study: Computation Model

Data parallelization Medium to coarse-grained parallelism Similar communication pattern for every

node except when starting application

Each processor tends to alternate computation and communication at the same time communication congestion is inevitable

Page 26: High Performance Cluster Computing Architectures and Systems

26

Relative Times of Communication Events for the IS Benchmark

Page 27: High Performance Cluster Computing Architectures and Systems

27

Relative Times of Communication Events for the MG Benchmark

Page 28: High Performance Cluster Computing Architectures and Systems

28

Case Study: Message Size and Destination Distributions

Distribution of message destinations Uniformly distributed among all nodes Biased to some destinations for each node

Distribution of message sizes

Page 29: High Performance Cluster Computing Architectures and Systems

29

Overall Message Destination Distribution for All of the Test Programs

Page 30: High Performance Cluster Computing Architectures and Systems

30

The Message Destination Distribution for Each Processor in the CG Benchmark

Page 31: High Performance Cluster Computing Architectures and Systems

31

The Cumulative Distribution of Message Sizes in the Test Programs

Page 32: High Performance Cluster Computing Architectures and Systems

32

Experiments Results

Parameterize communication pattern Point-to-point communication

Small message mean size l1, probability to appear is b

Large message mean size l2, probability to appear is 1 - (b+c)

Broadcast communication Message mean size l3, probability to appear is c

Assumption 1 master, p-1 slave processors Message size for each communication follow

Poison distribution with 3 different mean value l1, l2, l3

Page 33: High Performance Cluster Computing Architectures and Systems

33

Parameter Values Used in the Synthetic Benchmark

Applicationtype

A

B

C

pt2pt (small)mean l1

pt2pt (large)mean l2

broadcastmean l3

b 1-(b+c) c

45% 45% 10%

25% 25% 50%

5% 5% 90%

Page 34: High Performance Cluster Computing Architectures and Systems

34

Speedup using PBPS with the TCP and UDP Protocols over Ethernet. The speedups are normalized to the

case when using the TCP protocol alone

Page 35: High Performance Cluster Computing Architectures and Systems

35

Speedup using PBPA Technique with Ethernet and Fiber Channel using the TCP Protocols. The speedups are

normalized to the case when using the Ethernet alone

Page 36: High Performance Cluster Computing Architectures and Systems

36

Summary and Conclusion

Communication overhead can be reduced by exploiting heterogeneity in both communication path and application Reduce communication overhead

PBPD technique can achieve performance improvement based on message type