es mpich2 a message passing interface with enhanced security.bak

ES-MPICH2: A Message PassingInterface with Enhanced Security

Xiaojun Ruan, Member, IEEE, Qing Yang, Member, IEEE, Mohammed I. Alghamdi,

Shu Yin, Student Member, IEEE, and Xiao Qin, Senior Member, IEEE

Abstract—An increasing number of commodity clusters are connected to each other by public networks, which have become a

potential threat to security sensitive parallel applications running on the clusters. To address this security issue, we developed a

Message Passing Interface (MPI) implementation to preserve confidentiality of messages communicated among nodes of clusters in

an unsecured network. We focus on MPI rather than other protocols, because MPI is one of the most popular communication protocols

for parallel computing on clusters. Our MPI implementation—called ES-MPICH2—was built based on MPICH2 developed by the

Argonne National Laboratory. Like MPICH2, ES-MPICH2 aims at supporting a large variety of computation and communication

platforms like commodity clusters and high-speed networks. We integrated encryption and decryption algorithms into the MPICH2

library with the standard MPI interface and; thus, data confidentiality of MPI applications can be readily preserved without a need to

change the source codes of the MPI applications. MPI-application programmers can fully configure any confidentiality services in

MPICHI2, because a secured configuration file in ES-MPICH2 offers the programmers flexibility in choosing any cryptographic

schemes and keys seamlessly incorporated in ES-MPICH2. We used the Sandia Micro Benchmark and Intel MPI Benchmark suites to

evaluate and compare the performance of ES-MPICH2 with the original MPICH2 version. Our experiments show that overhead

incurred by the confidentiality services in ES-MPICH2 is marginal for small messages. The security overhead in ES-MPICH2 becomes

more pronounced with larger messages. Our results also show that security overhead can be significantly reduced in ES-MPICH2 by

high-performance clusters. The executable binaries and source code of the ES-MPICH2 implementation are freely available at http://

www.eng.auburn.edu/~xqin/software/es-mpich2/.

Index Terms—Parallel computing, computer security, message passing interface, encryption.

Ç

1 INTRODUCTION

1.1 Motivation

DUE to the fast development of the internet, an increasingnumber of universities and companies are connecting

their cluster computing systems to public networks toprovide high accessibility. Those clusters connecting to theinternet can be accessed by anyone from anywhere. Forexample, computing nodes in a distributed cluster systemproposed by Sun Microsystems are geographically deployedin various computing sites. Information processed in adistributed cluster is shared among a group of distributedtasks or users by the virtue of message passing protocols(e.g., message passing interface—MPI) or confidential datatransmitted to and from cluster computing nodes.

Preserving data confidentiality in a message passingenvironment over an untrusted network is critical for a widespectrum of security-aware MPI applications, because

unauthorized access to the security-sensitive messages byuntrusted processes can lead to serious security breaches.Hence, it is imperative to protect confidentiality of messagesexchanged among a group of trusted processes.

It is a nontrivial and challenging problem to offerconfidentiality services for large-scale distributed clusters,because there is an open accessible nature of the opennetworks. To address this issue, we enhanced the securityof the MPI protocol by encrypting and decrypting messagessent and received among computing nodes.

In this study, we focus on MPI rather than otherprotocols, because MPI is one of the most popularcommunication protocols for cluster computing environ-ments. Numerous scientific and commercial applicationsrunning on clusters were developed using the MPI protocol.Among a variety of MPI implementations, we pickedMPICH2 developed by the Argonne National Laboratory.The design goal of MPICH2—a widely used MPI imple-mentation—is to combine portability with high perfor-mance [14]. We integrated encryption algorithms into theMPICH2 library. Thus, data confidentiality of MPI applica-tions can be readily preserved without a need to change thesource codes of the MPI applications. Data communicationsof a conventional MPI program can be secured withoutconverting the program into the corresponding secureversion, since we provide a security enhanced MPI-librarywith the standard MPI interface.

1.2 Possible Approaches

There are three possible approaches to improving security ofMPI applications. In first approach, application programmers

IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 9, NO. 3, MAY/JUNE 2012 361

. X. Ruan is with the Department of Computer Science, West ChesterUniversity of Pennsylvania, PA 19383. E-mail: [email protected].

. Q. Yang is with the Department of Computer Science, Montana StateUniversity, MT 59717. E-mail: [email protected].

. M.I. Alghamdi is with the Department of Computer Science, Al-BahaUniversity, Al-Baha City, Kingdom of Saudi Arabia.E-mail: [email protected].

. S. Yin and X. Qin are with the Department of Computer Science andSoftware Engineering, Auburn University, Auburn, AL 36849-5347.E-mail: {szy0004, xqin}@auburn.edu.

Manuscript received 27 Sept. 2010; revised 22 Aug. 2011; accepted 14 Dec.2011; published online 11 Jan. 2012.Recommended for acceptance by M. Singhal.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference IEEECS Log Number TDSC-2010-09-0167.Digital Object Identifier no. 10.1109/TDSC.2012.9.

1545-5971/12/$31.00 � 2012 IEEE Published by the IEEE Computer Society

http://ieeexploreprojects.blogspot.com

can add source code to address the issue of messageconfidentiality. For example, the programmers may rely onexternal libraries (e.g., SEAL [26] and Nexus [11]) toimplement secure applications. Such an application-levelsecurity approach not only makes the MPI applications error-prone, but also reduces the portability and flexibility of theMPI applications. In the second approach, the MPI interfacecan be extended in the way that new security-aware APIs aredesigned and implemented (see, for example, MPISec I/O[22]). This MPI-interface-level solution enables programmersto write secure MPI applications with minimal changes to theinterface. Although the second approach is better than thefirst one, this MPI-interface-level solution typically requiresan extra code to deal with data confidentiality. The thirdapproach—a channel-level solution—is proposed in thisstudy to address the drawbacks of the above two approaches.Our channel-level solution aims at providing messageconfidentiality in a communication channel that implementsthe Channel Interface 3 (CH3) in MPICH2 (see Fig. 1).

1.3 Contributions

In what follows, we summarize the four major contributionsof this study.

. We implemented a standard MPI mechanism calledES-MPICH2 to offer data confidentiality for securenetwork communications in message passing envir-onments. Our proposed security technique incorpo-rated in the MPICH2 library can be very useful forprotecting data transmitted in open networks likethe Internet.

. The ES-MPICH2 mechanism allows applicationprogrammers to easily write secure MPI applica-tions without additional code for data-confidenti-ality protection. We seek a channel-level solution inwhich encryption and decryption functions areincluded into the MPICH2 library. Our ES-MPICH2maintains a standard MPI interface to exchangemessages while preserving data confidentiality.

. The implemented ES-MPICH2 framework provides asecured configuration file that enables applicationprogrammers to selectively choose any cryptographicalgorithm and symmetric-key in ES-MPICH2. Thisfeature makes it possible for programmers to easilyand fully control the security services incorporated inthe MPICHI2 library. To demonstrate this feature, weimplemented the AES and 3DES algorithms in ES-MPICH2. We also show in this paper how to addother cryptographic algorithms into the ES-MPICH2framework.

. We have used ES-MPICH2 to perform a detailedcase study using the Sandia Micro Benchmarks andthe Intel MPI benchmarks. We focus on runtimeperformance overhead introduced by the crypto-graphic algorithms.

1.4 Roadmap

The paper is organized as follows: Section 2 demonstrates thevulnerabilities of existing MPI implementations by describ-ing a security threat model for clusters connected by publicnetworks. Section 3 not only provides a reason for focusingon the confidentiality issue of MPICH2 rather than other MPIimplementations, but also gives an overview of the MPICH2implementation. Section 4 presents the motivation of thiswork by showing why secured MPI is an important issueand also outlines the design of ES-MPICH2—the messagepassing interface with enhanced security. Section 5 describesthe corresponding implementation details of ES-MPICH2.Section 6 discusses some experimental results and comparesthe performance of ES-MPICH2 with that of MPICH2.Section 7 presents previous research related to our project.Finally, Section 8 states the conclusions and future work ofthis study.

2 THREAT MODEL

A geographically distributed cluster system is one in whichcomputing components at local cluster computing plat-forms communicate and coordinate their actions bypassing messages through public networks like the Inter-net. To improve the security of clusters connected to thepublic networks, one may build a private network toconnect an array of local clusters to form a large scalecluster. Building a private network, however, is not a cost-effective way to secure distributed clusters. The Internet—avery large distributed system—can be used to supportlarge-scale cluster computing. Being a public network, theinternet becomes a potential threat to distributed clustercomputing environments.

We first describe the confidentiality aspect of security inclusters followed by three specific attack instances. Webelieve new attacks are likely to emerge, but the confidenti-ality aspect will remain unchanged. Confidentiality attacksattempts to expose messages being transmitted among a setof collaborating processes in a cluster. For example, ifattackers gain network administrator privilege, they canintercept messages and export the messages to a databasefile for further analysis. Even without legitimate privilege,an attacker still can sniff and intercept all messages in acluster on the public network. Such attacks result in theinformation leakage of messages passed among computingnodes in geographically distributed clusters. Cryptographyand access control are widely applied to computer systemsto safeguard against confidentiality attacks.

We identify the following three confidentiality attacks onMPI programs running on distributed clusters:

. Sniffing message traffic. Message traffic of an MPIprogram can be sniffed. For example, when MPCH2is deployed in a cluster connected by a GigabitEthernet network, attackers can sniff plaintextmessages transmitted through the TCP socket.Message sniffing can reveal security-sensitive data,metadata, and information.

. Snooping on message buffer. In an MPI program, buffersare employed to send and receive messages. Regard-less of specific MPI implementations, message buffersare created before the send and receive primitives are

362 IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING, VOL. 9, NO. 3, MAY/JUNE 2012

Fig. 1. Hierarchical Structure of MPICH2 [13].


invoked. Attackers who snoop into the messagebuffers in memory can access data and informationwithout being given specific access privileges.

. Message traffic profiling. Message traffic profilingattacks seek to use message type, time stamps,message size, and other metadata to analyzemessage exchange patterns and types of protocolsbeing used in message transmissions. For example,an attacker can monitor the network connection of acluster running an MPI program. If a message hasbeen regularly transmitted, the attacker can spec-ulate the importance of the message and interceptthe content of the message.

Confidentiality services can effectively counter theaforementioned threats in MPI applications running onclusters connected by a public network. In this research, weencode messages using the Advanced Encryption Standard(AES) and the Triple Data Encryption Standard (Triple-DESor 3DES). It is worth nothing that 3DES is not considered forany modern applications, we investigate 3DES in this studybecause of two reasons. First, 3DES is still used in manylegacy application systems in the industry. Second, 3DES isa good baseline solution used for the purpose of compar-ison. In the case that attackers intercept messages in an MPIprogram, they are unable to transform the ciphertext intothe original plaintext due to the lack of data enciphermentkeys (DEK).

3 MPICH2 OVERVIEW

MPICH—one of the most popular MPI implementations—-were developed at the Argonne National Laboratory [14].The early MPICH version supports the MPI-1 standard.MPICH2—a successor of MPICH—not only providessupport for the MPI-1 standard, but also facilitates thenew MPI-2 standard, which specifies functionalities likeone-sided communication, dynamic process management,and MPI I/O [13]. Compared with the implementation ofMPICH, MPICH2 was completely redesigned and devel-oped to achieve high performance, maximum flexibility,and good portability.

Fig. 1 shows the hierarchical structure of the MPICH2implementation, where there are four distinct layers ofinterfaces to make the MPICH2 design portable andflexible. The four layers, from top to bottom, are themessage passing interface 2 (MPI-2), the abstract deviceinterface (ADI3), the CH3, and the low-level interface.ADI3—the third generation of the abstract device inter-face—in the hierarchical structure (see Fig. 1) allowsMPICH2 to be easily ported from one platform to another.Since it is nontrivial to implement ADI3 as a full-featuredabstract device interface with many functions, the CH3layer simply implements a dozen functions in ADI3 [18].

As shown in Fig. 1, the TCP socket Channel, the sharedmemory access (SHMEM) channel, and the remote directmemory access (RDMA) channel are all implemented in thelayer of CH3 to facilitate the ease of porting MPICH2 onvarious platforms. Note that each one of the aforemen-tioned channels implements the CH3 interface for acorresponding communication architecture like TCP sock-ets, SHMEM, and RDMA. Unlike an ADI3 device, a channelis easy to implement since one only has to implement adozen functions relevant for with the channel interface.

To address the issues of message snooping in themessage passing environments on clusters, we seek toimplement a standard MPI mechanism with confidentialityservices to counter snooping threats in MPI programsrunning on a cluster connected an unsecured network.More specifically, we aim to implement cryptographicalgorithms in the TCP socket channel in the CH3 layer ofMPICH2 (see Fig. 2 and Section 5 for details of how toconstruct a cryptosystem in the channel layer).

4 THE DESIGN OF ES-MPICH2

4.1 Scope of ES-MPICH2

Confidentiality, integrity, availability, and authenticationare four important security issues to be addressed inclusters connected by an unsecured public network. Ratherthan addressing all the security aspects, we pay particularattention to confidentiality services for messages passedamong computing nodes in an unsecured cluster.

Although preserving confidentiality is our primary con-cern, an integrity checking service can be readily incorpo-rated into our security framework by applying a public-keycryptography scheme. In an MPI framework equipped withthe public-key scheme, sending nodes can encode messagesusing their private keys. In the message receiving procedure,any nodes can use public keys corresponding to the privatekeys to decode messages. If one alters the messages, theciphertext cannot be deciphered correctly using public keyscorresponding to the private keys. Thus, the receiving nodescan perform message integrity check without the secureexchange of secret keys. Please refer to Section 5.6 for detailsof how to add integrity checking services in our MPIframework called ES-MPICH2.

4.2 Design Issues

The goal of the development of the ES-MPICH2 mechanismis to enable application programmers to easily implementsecure enhanced MPI applications without additional codefor data-confidentiality protection. With ES-MPICH2 inplace, secure MPI application programmers are able toflexibly choose a cryptographic algorithm, key size, and datablock size for each MPI application that needs dataconfidentiality protection.

ES-MPICH2 offers message confidentiality in an MPIprogramming environment by incorporating MPICH2 withencryption and decryption algorithms. In the process ofdesigning ES-MPICH2, we integrated the AES and 3DESalgorithms into the MPICH2 library.

RUAN ET AL.: ES-MPICH2: A MESSAGE PASSING INTERFACE WITH ENHANCED SECURITY 363

Fig. 2. Message passing implementation structure in MPICH2.


The ES-MPICH2 implementation has the following fourdesign goals:

. Message confidentiality: ES-MPICH2 aims to pre-serve message confidentiality from unauthorizedaccesses by untrusted processes. We leverage theAES to protect the confidentiality of messages,because AES is an encryption standard adopted bythe US government. For comparison purpose, wealso consider 3DES in the design of ES-MPICH2. AESwith 128-bit keys can provide adequate protectionfor classified messages up to the SECRET level. Theimplementation of AES in products intended toprotect national security systems and/or informa-tion must be reviewed and certified by NSA prior totheir acquisition and use [1]. In this study, weintegrated data confidentiality services withMPICH2 by implementing the cryptographic algo-rithms in a CH3 channel.

. Complete transparency: Preserving message con-fidentiality in MPICH2 is entirely transparent toapplication programmers. Such confidentialitytransparency is feasible and the reason is two-fold.First, the encryption and decryption processes canbe built in the MPICH2 library at the channeltransmission layer. Second, we maintain the sameinterface as the APIs of the MPICH2 implementa-tion. Therefore, it is not necessary to modify MPIprograms to adapt ES-MPICH2.

. Compatibility and portability: Ideally, ES-MPICH2needs to be easily ported from one platform toanother with no addition to the application sourcecode. ES-MPICH2 is an extension of MPICH2 and;thus, ES-MPICH2 should have the same level ofportability as MPICH2. However, it is challenging toachieve high portability in ES-MPICH2, because wehave to implement a cryptographic subsystem ineach channel in the CH3 layer in MPICH2.

. Extensibility: ES-MPICH2 must allow applicationprogrammers to selectively choose any cipher tech-niques and keys incorporated in MPICH2. Thisdesign goal makes it possible for programmers toflexibly select any cryptographic algorithm imple-mented in ES-MPICHI2. Although we implementedAES and 3DES in the channel layer of MPICH2, wewill show in the next section how to add othercryptographic algorithms (e.g., Elliptic Curve Cryp-tography (ECC), [2]) to the ES-MPICH2 environment.

5 IMPLEMENTATION DETAILS

During the implementation of ES-MPICH2, we addressedthe following five development questions:

1. Among the multiple layers in the hierarchicalstructure of MPICH2, in which layer should weimplement cryptographic algorithms?

2. Which cryptosystem should we choose to implement?3. How to implement secure key management?4. How to use the implemented ES-MPICH2?5. How to add integrity checking services to ES-

MPICH2?

5.1 Ciphers in the Channel Layer

Fig. 2 outlines the message passing implementationstructure in the original version of MPICH2. In such ahierarchical structure of MPICH2, messages are passedfrom a sending process to a receiving process through theabstract device interface, the CH3, and the TCP socketchannel. Cryptographic subsystems may be implemented inone of the three layers (i.e., ADI3, CH3, or the TCP socketchannel). To achieve the design goal of a completetransparency, we chose to implement cryptographic algo-rithms in the TCP socket channel. Compared with ADI3 andCH3, the TCP socket channel is the lowest layer of theMPICH2 hierarchy. Implementing cryptosystems in thelowest layer can preserve message confidentiality in anyconventional MPI program without adding extra code toprotect messages. Fig. 3 depicts the implementationstructure of ES-MPICH2, where a cryptosystem is imple-mented in the TCP socket layer. Thus, messages areencrypted and decrypted in the TCP socket channel ratherthan the ADI3 and CH3 layers.

Fig. 4 shows that the encryption and decryption func-tions in ES-MPICH2 interact with the TCP socket to providemessage confidentiality protection in the TCP socket layer.Before a message is delivered through the TCP socketchannel, data contained in the message are encrypted by acertain cryptographic algorithm like AES and 3DES. Uponthe arrival of an encrypted message in a receiving node, thenode invokes the corresponding decryption function todecrypt the message. Fig. 4 demonstrates that ES-MPICH2maintains the same application programming interface orAPI as that of MPICH2 by implementing the encryption anddecryption algorithms in the TCP socket level. The con-fidentiality services of ES-MPICH2 were implemented inthe MPICH2 libraries, thereby being totally transparent toMPI application programmers.

5.2 Block Ciphers

We have no intention of reinventing a cryptographiclibrary, because it is very costly to guarantee that thesecurity of your own implementation is higher than that ofexisting tested and audited security libraries. In the ES-MPICH2 framework, we adopted the implementation of theAES and 3DES cryptographic algorithms offered by thePolarSSL library in MPICH2 version 1.0.7. PolarSSL is anopen-source cryptographic library written in C. We focuson block ciphers in the implementation of ES-MPICH2,because a block cipher transforms a fixed-length block ofplaintext into a block of ciphertext of the same length. If the


Fig. 3. Message passing implementation structure in ES-MPICH2 withencryption and decryption processes. A cryptosystem is implemented inthe TCP socket layer to achieve the design goal of completetransparency.


case where ciphertext and plaintext are different in length,MPI applications have to be aware of such a difference inorder to correctly decode ciphers. Keeping in mind thatsecurely passing messages should be transparent to MPIapplication programmers, we advocate the use of blockciphers rather than non-block-ciphers that force program-mers to be aware of the lengths of plaintext and ciphertext.

5.3 Key Management

The goal of key management is to dynamically establishsecure message-passing channels by distributing crypto-graphic keys. ES-MPICH2 maintains two types of keys—dataencipherment keys (a.k.a., session keys) and interchangekeys. In the MPI initialization phase of each MPI application,a data encipherment key is created and shared among all thecommunicating processes.

Fig. 5 presents the key management infrastructure in ES-MPICH2. Public key cryptography employed in ES-MPICH2relies on interchange keys (i.e., public and private keys) tosecurely exchange session keys in an unsecured network.More specifically, when a master node attempts to share newsession keys with other slave nodes, the master node usesthe slave nodes’ public keys to encrypt the session keys. Theslave nodes make use of their private keys to deciphermessages containing the session keys. Then, the master nodeand slave nodes can securely communicate using the MPIframework. Interchange keys also can be stored on physicaldevices like smart card and ROM [7], [9], [20].

5.4 Socket Programming

In socket programming, there is a buffer containing datasent and received through the TCP socket channel. Fig. 6demonstrates the encryption and decryption process in ES-MPICH2. ES-MPICH2 encrypt the plaintext in the buffer ofthe sending node then decrypt it on the receiving node.Because the plaintext and ciphertext are identical in lengthin block cipher algorithms, the sizes of the buffers in boththe sending and receiving nodes remain unchanged afterthe encryption and decryption processes.

5.5 Usage

The security features of ES-MPICH2 can be configuredwithout modifying any MPI application source code. Tosecurely pass messages using ES-MPCH2, the followingconfigurations must be set before an MPI initialization.First, a security option should be enabled or disabled.Second, one has to select a specific cryptographic algorithmimplemented in ES-MPICH2. Third, exchange keys must besecurely stored in a configuration file or a physical device ineach node (see Section 5.3 for details on the key manage-ment issue.) Then, users can run their MPI programs in thesame way as they should run the programs in MPICH2.Thus, if an MPI program can be executed in MPICH2, onecan also run the MPI program in ES-MPICH2 withoutmodifying the source code of the program.

5.6 Incorporating Integrity Services in ES-MPICH2

In addition to confidentiality services, integrity checkingservices can be seamlessly incorporated into the ES-MPICH2 framework. In what follows, we address theimplementation issue of how to integrate integrity checkingservices in ES-MPICH2.

Spreading feature of block encryption algorithms.Block encryption algorithms have a spreading featurewhich means if even 1 bit is changed in ciphertext, thedecrypted text will be completely different from the originalplaintext. Altered messages causing fatal errors cannot beinterpreted. Although using the spreading feature is not areliable solution, the spreading feature does provide anintegrity checking method. Since both AES and 3DES areblock encryption algorithms, ES-MPICH2 may rely on thespreading feature to perform integrity checking.

Public key. An integrity service tailed for small messagescan be added into ES-MPICH2 using a public-key encryp-tion scheme, in which sending nodes encode messages using


Fig. 5. Key management in ES-MPICH2. Public key cryptographyemployed in ES-MPICH2 relies on interchange keys (i.e., public andprivate keys) to exchange data encipherment keys in a secure way.

Fig. 6. ES-MPICH2 socket details.

Fig. 4. The interface between the encryption/decryption processes andthe TCP socket. ES-MPICH2 maintains the same API as that ofMPICH2.


private keys whereas receiving nodes use the correspondingpublic keys to decipher the ciphertext. Before a nodedelivers an encrypted message in ES-MPICH2, the nodeencrypts the message using the private key of the sendingnode. To check the integrity of the cipher message, areceiving node simply needs to decode the cipher messageby applying the public key of the sending node.

Hash functions. When it comes to large messages, hashfunctions are widely used in integrity checking and digitalsignatures. MD5, SHA-1, and SHA-2 can be implemented tocheck the integrity of encrypted messages in ES-MPICH2.The hash is a cryptographic checksum or message integritycode that sending and receiving nodes must compute toverify messages. Specifically, a sending node uses a hashfunction to compute a checksum for a large message. Thechecksum is shorter than the original message. Then, thesending node signs the hash value with a shared key. Indoing so, the integrity of the large message can be checkedin an efficient way.

People use hash and then sign the hash with the key, thesize of hash being much shorter than the message.

6 EXPERIMENTAL EVALUATION

To evaluate the features and performance of ES-MPICH2,we implemented ES-MPICH2 and deployed it on twoclusters with different configurations. The first cluster hassix nodes of 2.2 GHz Intel Celeron processors with 2 GBmemory. The second cluster contains 10 nodes. The masternode has a 3.0 GHz Intel Pentium Core 2 Duo processorwith 1 GB memory, whereas the nine slave nodes have333 MHz Intel Pentium II processors with 64 MB memory.The six nodes in the first cluster are connected by a 1 GbpsEthernet LAN. The 10 nodes in the second cluster areconnected by a 100 Mbps Ethernet LAN. Apparently, theoverall performance of the first cluster is higher than that ofthe second cluster.

We use a fast cluster (i.e., the first one) and a slow (i.e.,the second one) cluster to conduct experiments, because oneof the goals is to illustrate the impact of computing capacityof clusters on the performance of ES-MPICH2.

6.1 A 6-Node Cluster of Intel Celeron Processors

6.1.1 Experimental Testbed

Let us first evaluate the performance of both MPICH2 andES-MPICH2 on a 6-node cluster. Table 1 reports theconfiguration of the first cluster with six identical comput-ing nodes of Intel Celeron processors. The operating systemused in the six nodes is Ubuntu 9.04 Jaunty Jackalope. Thecomputing nodes are connected by a 1 Gbps network. Allthe slave nodes share a disk on the master node through the

network file system (NFS) [25]. The MPI library used in the

6-node cluster is MPICH2 version 1.0.7. We run the Sandia

Micro Benchmarks and the Intel MPI Benchmarks to

evaluate and compare the performance of MPICH2 and

ES-MPICH2. When we test ES-MPICH2 in each experiment,

we set the cryptographic service to AES and 3DES,

respectively. The length of data encipherment keys gener-

ated and distributed in ES-MPICH2 is 192-bit.

6.1.2 SMB: Sandia Micro Benchmark

The Sandia National Laboratory developed the Sandia

Micro Benchmark Suite (a.k.a., SMB) to evaluate and test

high-performance network interfaces and protocols. Table 2

described the four performance metrics used in the SMB

benchmark suite. These metrics include total execution time

(i.e., iter_t), CPU execution time for iterations (i.e., work_t),

message passing overhead (i.e., overhead_t), and message

transfer time calculation threshold (i.e., threshold or base_t).

The detailed information on these metrics can be found at

http://www.cs.sandia.gov/smb. Please note that the mes-

sage passing overhead can be derived by subtracting the

CPU execution time from the total execution time. Each

benchmark has 1,000 iterations.Fig. 7 shows the total execution time of the SMB

benchmark running on the original MPI implementation

(i.e., MPICH2) as well as AES-based ES-MIPCH2 and 3DES-

based ES-MPICH2. We observe from this figure that when

the message size is small (e.g., 1 KB), the performance of ES-

MPICH2 is very close to that of MPICH2. For example, the

encryption modules in ES-MPICH2 only modestly increase

the execution time by less than two percent. These results

indicate that ES-MPICH2 can preserve confidentiality of

small messages with negligible overhead.


TABLE 1The Configuration of a 6-Node Cluster

of Intel Celeron Processors

TABLE 2Performance Metrics Used in theSandia Micro Benchmark Suite

Fig. 7. Sandia Micro Benchmark iter_time.


Figs. 8a, 8b, 8c, and 8d show the total execution time, CPUtime, overhead, and threshold of MPICH2 and ES-MPICH2when the message size is set to 2, 16, 128, 512, and 1,024 KB,respectively. The results plotted in Fig. 8a show that AES-based ES-MPICH2 and MPICH2 have similar performancein the case of small messages. For example, the AES modulein ES-MPICH2 increases the execution times of iter t andwork t by 5.9 and 1.2 percent, respectively. However, when3DES is employed in ES-MPICH2, the security overhead ofES-MPICH2 becomes noticeable even for small messages.For example, let us consider a case where the message size is2 KB. Compare with the 3DES module, the AES module canreduce the execution times of iter t and base t by approxi-mately 56 and 45 percent, respectively. Figs. 8b, 8c, and 8dillustrate that both AES and 3DES in ES-MPICH2 introducemuch overhead that makes ES-MPICH2 performs worsethan MPICH2—the original MPI implementation. Securityoverhead in ES-MPICH2 becomes more pronounced withincreasing message size. Since AES has better performancethan 3DES, AES-based ES-MPICH2 is superior to 3DES-based ES-MPICH2. We recommend the following twoapproaches to lowering overhead caused by encryptionand decryption modules in ES-MPICH2. First, one canreduce the security overhead in ES-MPICH2 by enhancingthe performance of block cipher algorithms. Second, multi-core processors can boost efficiency of the encryption anddecryption modules, thereby benefiting the performance ofES-MPICH2.

6.1.3 IMB: Intel MPI Benchmarks

The Intel MPI benchmark suite or IMB was developed fortesting and evaluating implementations of both MPI-1 [8]and MPI-2 [12] standards. IMB contains approximately10,000 lines of code to measure the performance ofimportant MPI functions [5], [24]. We have evaluated theperformance of ES-MPICH2 and the original MPICH2 byrunning the benchmarks on the 6-node cluster. Table 3 listsall the Intel benchmarks used to measure the performanceof ES-MPI2 and MPICH2. The benchmarks in IMB-MPI1can be categorized in three groups: single transfer, paralleltransfer, and collective benchmarks. Single transfer bench-marks are focusing on a single message transferred betweentwo communicating processes. Unlike single transfer bench-marks, parallel transfer benchmarks aim at testing patternsand activities in a group of communicating processes withconcurrent actions. Collective benchmarks are implementedto test higher level collective functions, which involveprocessors within a defined communicator group. Pleaserefer to http://software.intel.com/en-us/articles/intel-mpi-benchmarks for more information concerning IMB.

Figs. 9a and 9b show the performance of PingPong andPingPing—two single transfer benchmarks in IMB. Sincesingle transfer benchmarks are used to test a pair of twoactive processes, we run PingPong and PingPing on twonodes of the 6-node cluster. The total execution times ofPingPong and PingPing go up when the message size


Fig. 8. Sandia Micro Benchmark, Iteration Time, Work Time, Overhead Time, and Base Time on different message size from 6-node cluster of IntelCeleron.


increases because larger messages give rise to higher

encryption and decryption overheads. Compared with

MPICH2, the execution times of AES-based and 3DES-

based ES-MPICH2 are more sensitive to message size.Now we analyze the performance of Sendrecv and

Exchange—two parallel transfer benchmarks in IMB—

running on ES-MPICH2 and MPICH2 on the 6-node cluster.

Sendrecv, in which the main purpose is to test the

MPI_Sendrecv function, consists of processes forming a

periodic communication chain. Similarly, Exchange is abenchmark focusing on the evaluation of the MPI_ISend,MPI_Waitall, and MPI_Recv functions. Unlike the afore-mentioned single transfer benchmarks, message passingoperations in these two parallel benchmarks are performedin parallel.

Fig. 9c plots the performance results of the SendRecvbenchmark on the cluster, where each node receives datafrom its left neighbor and then sends data to its rightneighbor. The total execution time of the SendRecv bench-mark does not noticeably change when we vary the numberof computing nodes in the cluster. We attribute this trend tothe factor that message passing in multiple nodes arecarried out in parallel rather than serially. Thus, increasingthe number of nodes does not affect SendRecv’s totalexecution time. With respect to parallel transfers, theperformance of AES-based and 3DES-based MPICH2 isclose to that of the original version of MIPCH2 whenmessage size is relatively small. When it comes to largemessages, AES-based ES-MPICH2 has better paralleltransfer performance than 3DES-based MPICH2.

Fig. 9d depicts the total execution time of the Exchangebenchmark. Comparing Fig. 9d with Fig. 9c, we realize thatregardless of the MPI implementations, the execution timeof the Exchange benchmark is longer than that of theSendRecv benchmark under the condition of same messagesize. This is mainly because in Exchange each node transferdata to both left and right neighbors in the communication


TABLE 3Intel MPI Benchmarks

Fig. 9. Intel MPI Benchmarks, PingPong, and PingPing are Single Transfer Benchmarks, SendRecv and Exchange are Parallel Benchmarkson 6-node cluster of Intel Celeron.


chain. Thus, communication time in Exchange is larger thanthat in SendRecv. As a result, the total execution time ofExchange is approximately two times higher than that ofSendrecv when message size is large.

Let us vary message size and evaluate the performance ofcollective benchmarks. We run the benchmark 10 times oneach MPI implementation and report the average executiontimes. Figs. 10a and 10b show the performance of the firstgroup of nine collective benchmarks. We observe from thesefigures that the total execution time of each collectivebenchmark continually increases with increasing messagesize. MPICH2 has better performance than AES-based and3DES-based ES-MPICH2 across all the collective bench-marks, because the confidentiality is preserved at the cost ofmessage passing performance. Figs. 11a and 11b plot theexecution times of the second group of three benchmarks.The performance results of the second benchmark group areconsistent with those of the first benchmark group reportedin Figs. 10a and 10b.

Fig. 12 shows the results of the Window benchmark,which aims to test MPI-2 functions like MPI_Win_create,MPI_Win_fence, and MPI_Win_free. In this benchmark, awindow size message is transferred to each node, which inturn creates a window using a specified size. Fig. 12indicates that the execution time of the benchmark is notsensitive to message size. The results confirm that AES-based ES-MPICH2 improves the security of the Windowbenchmark on MPICH2 with marginal overhead.

6.2 A 10-Node Cluster of Intel Pentium II Processors

6.2.1 Experimental Testbed

Now we evaluate the performance of MPICH2 and ES-MPICH2 on a 10-node cluster of Intel Pentium II processors.

The cluster configuration is summarized in Table 4. The

operating system running on this cluster is Fedora Core

release 4 (Stentz). Although the processors of the nine slave

nodes are 333 MHz Intel Pentium II, the master node

contains a 3.0 GHz Intel Pentium Core 2 Duo processor,

which is almost 10 times faster than the processors in the

slave nodes. Each slave node has only 64 MB memory,

whereas the master node has 1 GB memory. All the 10 nodes

are connected by a 100 Mbps Ethernet network. Like the first

cluster, all nodes in the 10-node cluster share disk space on

the master node through the network file system.

6.2.2 SMB: Sandia Micro Benchmark

Figs. 13a, 13b, and 13c reveal the total execution time, CPU

time, overhead, and threshold of MPICH2 and ES-MPICH2

when the message size is set to 1, 16, and 32 KB,

respectively. The results show that the performance of


Fig. 10. Intel MPI Benchmarks, Collective Benchmarks, Group A, a group of benchmarks from Collective on 6-node Cluster of Intel Celeron.

Fig. 11. Intel MPI Benchmarks, Collective Benchmarks, Group B, a group of benchmarks from Collective on 6-node Cluster of Intel Celeron.

Fig. 12. Intel Micro Benchmark Window six nodes.


AES-based and 3DES-based ES-MPICH2 is noticeably worsethan that of MPICH2, because the encryption and decryp-tion modules in ES-MPICH2 spend significant amount oftime in encrypting and decrypting messages issued from thebenchmarks. This trend is true even when messages aresmall (e.g., see Fig. 13a where message size is 1 KB).

Comparing Fig. 13a and Fig. 8a, we draw the followingthree observations. First, the first 6-node cluster issignificantly faster than the second 10-node cluster.Although the 10-node cluster has more computing nodesthan the second one, the nodes of the 10-node cluster havelower computing capacity than those in the 6-node cluster.This is because the hardware configuration of the 10-nodecluster is worse than that of the 6-node cluster. In otherwords, the processors in the 10-node cluster are Pentium IICPUs; the 6-node cluster relies on Intel Celeron 450 to runthe benchmarks. Second, compared with the 10-node slowcluster, the 6-node fast cluster allows the iter_t and thework_t benchmarks to spend smaller time periods dealingwith the security modules in MPICH2. For example, onthe 6-node fast cluster, iter_t and work_t spend approxi-mately 7.1 and 3.2 percent of the benchmarks’ totalexecution times in the AES-based security modules (SeeFig. 8a). On the 10-node slow cluster, the AES modulesaccount for about 35.5 and 39.3 percent of these twobenchmarks’ total execution times(See Fig. 13a). Third, theperformance of AES-based MPICH2 on the first cluster isvery close to that of MPICH2 when message size issmaller than 2 KB.

The above observations indicate that given message-intensive MPI applications, one can improve the processorcomputing capacity of a cluster to substantially reduce theportion of time (out of the applications’ total executiontime) spent in processing security modules in ES-MPICH2.

6.2.3 IMB: Intel MPI Benchmarks

Figs. 14a, 14b, 14c, and 14d depict the performance of thePingPong, PingPing, SendRecv, and Exchange benchmarksin IMB. The total execution times of the four IMB bench-marks increases with increasing message size. Comparedwith MPICH2, the execution time of ES-MPICH2 is moresensitive to message size. More importantly, Figs. 14a, 14b,14c, and 14d demonstrate that when ES-MPICH2 isdeployed on a slow cluster, ES-MPICH2 preserves messageconfidentiality by substantially degrading the performanceof the original MPICH2. By comparing the Intel benchmarkperformance on both the 6-node cluster (see Figs. 9a, 9b, 9c,and 9d and 10-node clusters (see Figs. 14a, 14b, 14c, and14d)), we observe that the performance gap betweenMPICH2 and ES-MPICH2 on the fast cluster is muchsmaller than the performance gap on the slow cluster. Animplication of this observation is that security overhead inES-MPICH2 can be significantly reduced by deploying ES-MPICH2 in a high-end cluster.

7 RELATED WORK

Message passing interface. The Message Passing Interfacestandard (MPI) is a message passing library standard usedfor the development of message-passing parallel programs[14]. The goal of MPI is to facilitate an efficient, portable, andflexible standard for parallel programs using messagepassing. MPICH2—developed by the Argonne NationalLaboratory—is one of the most popular and widelydeployed MPI implementations in cluster computing envir-onments. MPICH2 provides an implementation of the MPIstandard while supporting a large variety of computationand communication platforms like commodity clusters,high-performance computing systems, and high-speed net-works [13].

As early as 1997, Brightwell et al. from the SandiaNational Laboratory insightfully pointed out barriers tocreating a secure MPI framework [3]. The barriers includecontrol and data in addition to cryptographic issues. In asecure MPI, both control and data messages must beprotected from unauthorized access of attackers andmalicious users. Although there is a wide range ofimplementations of the MPI and MPI-2 standards (e.g.,MPICH and MPICH2 are two freely available implementa-tions from the Argonne National Laboratory), there is a lack


Fig. 13. Sandia Micro Benchmarks on the 10-node cluster of Intel Pentium II.

TABLE 4The Configuration of a 10-Node Cluster

of Intel Pentium II Processors


of secure MPI frameworks developed for large-scaleclusters distributed across wide area networks.

Data confidentiality in MPI-I/O. Prabhakar et al.designed and implemented a secure interface called MPISecI/O for the MPI-I/O framework [22]. MPISec I/O preservesthe advantages of both parallel I/O and data confidentialitywithout significantly impacting performance of MPI appli-cations. It is flexible for MPI programmers to manually setencryption rules in MPISec I/O. Data can be encrypted andwritten onto disks in MPISec I/O, then encrypted data canbe read from the disks before being decrypted. There aretwo interesting features of MPISec I/O. First, MPISec I/Oprogrammers need to carefully set up encryption anddecryption rules in their MPI programs. Otherwise, somedata may be either stored on disks without encryption orread without decryption and as a result, the MPI programsare unable to function properly until the rules are set in acorrect way. Second, MPISec is not completely compatiblewith nonsecure MPI libraries. In other words, preservingdata confidentiality in MPISec I/O is not transparent to MPIapplication programmers. One has to modify the sourcecode of conventional MPI programs to improve security ofthe MPI programs. Apart from updating the source code ofthe MPI programs before MPISec I/O can be used properly,disk-resident data must be marked as encrypted orunencrypted.

Block ciphers. The Data Encryption Standard (DES)provides a relatively simple method of encryption. 3DES

encrypts data three times instead of one using the DESstandard [4]. 3DES is a block and symmetric cipher chosenby the US National Bureau of Standards as an officialFederal Information Processing Standard in 1976. 3DESincreases the key size of DES to protect against brute forceattacks without relying on any new block cipher algorithm.A hardware implementation of 3DES is significantly fasterthan the best software implementations of 3DES [10] [15]. Asoftware implementation of 3DES was integrated in ES-MPICH2. A hardware 3DES can substantially improveperformance of 3DES-based ES-MPICH2.

In November 2001, the symmetric block cipher Rijndaelwas standardized by the National Institute of Standardsand Technology as the Advanced Encryption Standard [6].AES—the successor of the Data Encryption Standard—hasbeen widely employed to prevent confidential data frombeing disclosed by unauthorized attackers. AES can be usedin high-performance servers as well as small and mobileconsumer products. AES is the preferred cryptographicalgorithm to be implemented in ES-MPICH2, which wasbuilt based on symmetric block ciphers. Although AESintroduces overhead due to additional security operationsin ES-MPICH2, the overhead caused by AES in ES-MPICH2can be significantly reduced by AES hardware architectures(see [19] for details of a highly regular and scalable AEShardware architecture).

Security enhancement in clusters. There are severalresearch works focusing on the security enhancement in


Fig. 14. Intel MPI Benchmarks, PingPong, and PingPing are Single Transfer Benchmarks, SendRecv and Exchange are Parallel Benchmarks on 10-node Cluster of Pentium II.


commodity clusters. For example, Lee and Kim developed asecurity framework in the InfiniBand architecture (IBA)[17]. For confidentiality and authentication, Lee and Kimproposed the partition-level and QP-level secret keymanagement schemes. The security in IBA is improvedwith minor modifications to the IBA specification. Ram-surrun and Soyjaudah constructed a highly availabletransparent Linux cluster security model, which offers anew approach to enhancing cluster security [23]. Koenig etal. implemented a tool that monitors processes acrosscomputing nodes in a cluster [16]. The tool delivers real-time alerts when there are immediate threats. Similarly,Pourzandi et al. investigated the security issues of detectingthreats and hazards in distributed clusters [21]. Theaforementioned security solutions developed for clustersare inadequate to directly support security-sensitive MPIprograms, because the existing security solutions generallyrequire application developers to implement securityfunctionality in their MPI programs.

8 CONCLUSIONS AND FUTURE WORK

To address the issue of providing confidentiality servicesfor large-scale clusters connected by an open unsecurednetwork, we aim at improving the security of the messagepassing interface protocol by encrypting and decryptingmessages communicated among computing nodes. In thisstudy, we implemented the ES-MPICH2 framework, whichis based on MPICH2. ES-MPICH2 is a secure, compatible,and portable implementation of the message passinginterface standard. Compared with the original version ofMPICH2, ES-MPICH2 preserves message confidentiality inMPI applications by integrating encryption techniques likeAES and 3DES into the MPICH2 library.

In light of ES-MPICH2, programmers can easily writesecure MPI applications without an additional source codefor data-confidentiality protection in open public networks.The security feature of ES-MPICH2 is entirely transparentto MPI programmers because encryption and decryptionfunctions are implemented at the channel-level in theMPICH2 library. MPI-application programmers can fullyconfigure any confidentiality services in MPICHI2, becausea secured configuration file in ES-MPICH2 offers theprogrammers flexibility in choosing any cryptographicschemes and keys seamlessly incorporated in ES-MPICH2.Besides the implementation of AES and 3DES in ES-MPICH2, other cryptographic algorithms can be readilyintegrated in the ES-MPICH2 framework. We used theSandia Micro Benchmarks and the Intel MPI benchmarks toevaluate and analyze the performance of MPICH2.

Confidentiality services in ES-MPICH2 do introduceadditional overhead because of security operations. Inthe case of small messages, the overhead incurred by thesecurity services is marginal. The security overhead causedby AES and 3DES becomes more pronounced in ES-MPICH2with larger messages (e.g., the message size is larger than256 KB). Our experimental results show that the securityoverhead in ES-MPICH2 can be significantly reduced byhigh-performance clusters. For example, the overheadadded by AES in ES-MPICH2 is reduced by more than halfwhen the 6-node cluster of Intel Celeron is used instead of

the 10-node cluster of Intel Pentium II. In addition to high-end clusters, the following two solutions can be applied tofurther reduced overhead caused by confidentiality servicesin ES-MPICH2. First, AES/3DES hardware implementationscan lower security overhead in ES-MPICH2. Second,security coprocessors can hide the overhead by allowingthe encryption and decryption processes to be executed inparallel with the message passing processes.

We are currently investigating varies means of reducingsecurity overhead in ES-MPICH2. For example, we plan tostudy if multicore processors can substantially lower theoverhead of confidentiality services in ES-MPICH2.

Another interesting direction for future work is toconsider several strong and efficient cryptographic algo-rithms like the Elliptic Curve Cryptography in ES-MPICH2.Since ECC is an efficient and fast cryptographic solution,both the performance and the security of ES-MPICH2 arelikely to be improved by incorporating ECC.

A third promising direction for further work is tointegrate encryption and decryption algorithms in othercommunication channels like SHMEM and InfiniBand inMPICH2 because an increasing number of commodityclusters are built using standalone and advanced networkssuch as Infiniband and Myrinet.

The current version of ES-MPICH2 is focused on securingthe transmission control protocol (TCP) connections on theinternet, because we addressed the data confidentialityissues on geographically distributed cluster computingsystems. In addition to the MPI library, other parallelprogramming libraries will be investigated. Candidatelibraries include the shared memory access library andthe remote direct memory access library. We plan to provideconfidentiality services in the SHMEM and RDMA libraries.

Last but not least, we will quantitatively evaluate theperformance of integrity checking services that are incor-porated into the ES-MPICH2 framework. The goal ofdeveloping the integrity checking services is to reduceoverhead of the services in ES-MPICH2.

9 AVAILABILITY

The executable binaries and source code of ES-MPICH2 arefreely available, along with documentation and benchmarksfor experimentation, at http://www.eng.auburn.edu/~xqin/software/es-mpich2/.

ACKNOWLEDGMENTS

The work reported in this paper was supported by the USNational Science Foundation under Grants CCF-0845257(CAREER), CNS-0757778 (CSR), CCF-0742187 (CPA), CNS-0917137 (CSR), CNS-0831502 (CyberTrust), CNS-0855251(CRI), OCI-0753305 (CI-TEAM), DUE-0837341 (CCLI), andDUE-0830831 (SFS), as well as Auburn University under astartup grant and a gift (Number 2005-04-070) from the IntelCorporation. Mohammed Alghamdi’s research was sup-ported by the King Abdulaziz City for Science andTechnology (KACST) and AL-Baha University.



REFERENCES

[1] Nat’l Security Agency. Nat’l Policy on the Use of the AdvancedEncryption Standard (aes) to Protect Nat’l Security Systems andNat’l Security Information cnss Policy no. 15 Fact Sheet no. 1, June2003.

[2] I.F. Blake, G. Seroussi, and N.P. Smart, Elliptic Curves inCryptography. Cambridge Univ. Press, 1999.

[3] R. Brightwell, D.S. Greenberg, B.J. Matt, and G.I. Davida Barriersto Creating a Secure mpi, 1997.

[4] D. Coppersmith, D.B. Johnson, S.M. Matyas, T.J. Watson, D.B.Johnson, and S.M. Matyas Triple des cipher Block Chaining withOutput Feedback Masking, 1996.

[5] Intel Corporation. Intel mpi Benchmarks User Guide andMethodology Description, 2008.

[6] J. Daemen and V. Rijmen, The Design of Rijndael. Springer, 2002.[7] D.E. Denning, “Secure Personal Computing in an Insecure

Network,” Comm. ACM, vol. 22, no. 8, pp. 476-482, 1979.[8] J.J. Dongarra, S.W. Otto, M. Snir, and D. Walker, “An

Introduction to the Mpi Standard,” technical report, Knoxville,TN, 1995.

[9] W. Ehrsam, S. Matyas, C. Meyer, and W. Tuchman, “ACryptographic Key Management Scheme for Implementing theData Encryption Standard,” IBM Systems J., vol. 17, no. 2, pp. 106-125, 1978.

[10] A.J. Elbirt, W. Yip, B. Chetwynd, and C. Paar, “An fpga-BasedPerformance Evaluation of the aes Block Cipher CandidateAlgorithm Finalists,” IEEE Trans. Very Large Scale IntegrationSystems, vol. 9, no. 4, pp. 545-557, Aug. 2001.

[11] I. Foster, N.T. Karonis, C. Kesselman, G. Koenig, and S. Tuecke,“A Secure Communications Infrastructure for High-PerformanceDistributed Computing,” Proc. IEEE Sixth Symp. High PerformanceDistributed Computing, pp. 125-136, 1996.

[12] A. Geist, W. Gropp, S. Huss-Lederman, A. Lumsdaine, E.L. Lusk,W. Saphir, T. Skjellum, and M. Snir, “Mpi-2: Extending theMessage-Passing Interface,” Proc. Second Int’l Euro-Par Conf.Parallel Processing (Euro-Par ’96), pp. 128-135, 1996.

[13] R. Grabner, F. Mietke, and W. Rehm, “Implementing an mpich-2Channel Device over Vapi on Infiniband,” Proc. 18th Int’l Paralleland Distributed Processing Symp., p. 184, Apr. 2004.

[14] W. Gropp, E. Lusk, N. Doss, and A. Skjellum, “A High-Performance, Portable Implementation of the Mpi MessagePassing Interface Standard,” Parallel Computing, vol. 22, no. 6,pp. 789-828, 1996.

[15] P. Hamalainen, M. Hannikainen, T. Hamalainen, and J. Saarinen,“Configurable Hardware Implementation of Triple-des Encryp-tion Algorithm for Wireless Local Area Network,” Proc. IEEE Int’lConf. Acoustics, Speech, and Signal Processing (ICASSP ’01),pp. 1221-1224, 2001.

[16] G.A. Koenig, X. Meng, A.J. Lee, M. Treaster, N. Kiyanclar, and W.Yurcik, “Cluster Security with Nvisioncc: Process Monitoring byLeveraging Emergent Properties,” Proc. IEEE Int’l Symp. ClusterComputing and Grid (CCGrid ’05), 2005.

[17] M. Lee and E.J. Kim, “A Comprehensive Framework forEnhancing Security in Infiniband Architecture,” IEEE Trans.Parallel Distributed Systems, vol. 18, no. 10, pp. 1393-1406, Oct.2007.

[18] J. Liu, W. Jiang, P. Wyckoff, D.K. Panda, D. Ashton, D. Buntinas,W. Gropp, and B. Toonen, “Design and Implementation of Mpich2over Infiniband with rdma Support,” Proc. 18th Int’l Parallel andDistributed Processing Symp., p. 16, Apr. 2004.

[19] S. Mangard, M. Aigner, and S. Dominikus, “A Highly Regular andScalable aes Hardware Architecture,” IEEE Trans. Computers,vol. 52, no. 4, pp. 483-491, Apr. 2003.

[20] S. Matyas and C. Meyer, “Generation, Distribution, and Installa-tion of Cryptographic Keys,” IBM Systems J., vol. 17, no. 2, pp. 126-137, 1978.

[21] M. Pourzandi, D. Gordon, W. Yurcik, and G.A. Koenig, “Clustersand Security: Distributed Security for Distributed Systems,” Proc.IEEE Int’l Symp. Cluster Computing and the Grid (CCGrid ’05), vol. 1,pp. 96-104, May 2005.

[22] R. Prabhakar, C. Patrick, and M. Kandemir, “MPISec I/O:Providing Data Confidentiality in MPI-I/O,” Proc. IEEE/ACMNinth Int’l Symp. Cluster Computing and the Grid, pp. 388-395, 2009.

[23] V. Ramsurrun and K.M.S. Soyjaudah, “A Highly AvailableTransparent Linux Cluster Security Model,” Proc. IEEE Int’lPerformance, Computing and Comm. Conf. (IPCCC), pp. 69-76, Dec.2008.

[24] S. Saini, R. Ciotti, B.T.N. Gunney, T.E. Spelce, A. Koniges, D.Dossa, P. Adamidis, R. Rabenseifner, S.R. Tiyyagura, and M.Mueller, “Performance Evaluation of Supercomputers using hpccand imb Benchmarks,” J. Computer and System Sciences, vol. 74,no. 6, pp. 965-982, 2008.

[25] S. Shepler, B. Callaghan, D. Robinson, R. Thurlow, C. Beame, M.Eisler, and D. Noveck, “Network File System (nfs) Version 4Protocol,” 2003.

[26] D.S. Wong, H.H. Fuentes, and A.H. Chan, “The PerformanceMeasurement of Cryptographic Primitives on Palm Devices,”Proc. 17th Ann. Computer Security Applications Conf. (ACSAC),pp. 92-101, 2001.

Xiaojun Ruan (S’07-M’09-A’10) received the BSdegree in computer science from ShandongUniversity in 2005, PhD degree in computerscience from Auburn University in 2011. He is anassistant professor of Computer Science atWest Chester University of Pennsylvania. Hisresearch interests include energy efficient sto-rage systems, parallel and distributed systems,computer architecture, operating systems, HDDand SSD technologies, and computer security.He is a member of the IEEE.

Qing Yang (S’08-M’11) received the BE and MEdegrees in computer science and technologyfrom Nankai University and Harbin Institute ofTechnology, China, in 2003 and 2005, respec-tively, and the PhD degree in computer sciencefrom Auburn University in 2011. He is currently aRightNow Technologies assistant professor ofcomputer science at Montana State University.His research interests include wireless vehicularad hoc networks (VANET), wireless sensor

networks, network security and privacy, and distributed systems. He isa member of the IEEE.

Mohammed I. Alghamdi received the BSdegree in computer science from King SaudUniversity, Riyadh, Saudi Arabia in 1997, the MSdegree in software engineering from ColoradoTechnical University, Colorado, the master’s ofscience degree in information technology man-agement from Colorado Technical University,Denver, Colorado, 2003 and the PhD degree incomputer science from New Mexico Institute ofMining and Technology.



Shu Yin (S’09) received the BS degree incommunication engineering from Wuhan Uni-versity of Technology (WUT) in 2006, the MSdegree in signal and information processingfrom WUT in 2008, and is currently, workingtoward the PhD degree in the Department ofComputer Science and Software Engineering atAuburn University. His research interests in-clude storage systems, reliability modeling, faulttolerance, energy-efficient computing, and wire-

less communications. He is a student member of the IEEE.

Xiao Qin (S’00-M’04-SM’09) received the BSand MS degrees in computer science from theHuazhong University of Science and Technol-ogy, Wuhan, China, and the PhD degree incomputer science from the University of Nebras-ka-Lincoln, Lincoln, in 1992, 1999, and 2004,respectively. For three years, he was anassistant professor with the New Mexico Insti-tute of Mining and Technology, Socorro. Cur-rently, he is an associate professor with the

Department of Computer Science and Software Engineering, AuburnUniversity, Alabama. His research interests include parallel anddistributed systems, storage systems, fault tolerance, real-time systems,and performance evaluation. His research is supported by the USNational Science Foundation (NSF), Auburn University, and IntelCorporation. He was a recipient of the US National Science FoundationComputing Processes and Artifacts Award, the NSF Computer SystemResearch Award in 2007, and the NSF CAREER Award in 2009. He wasa subject area editor of the IEEE Distributed System Online in 2000-2001. He has been on the program committees of various internationalconferences, including IEEE Cluster, IEEE International Performance,Computing, and Communications Conference, and International Con-ference on Parallel Processing. He is a senior member of the IEEE.

. For more information on this or any other computing topic,please visit our Digital Library at www.computer.org/publications/dlib.



es mpich2 a message passing interface with enhanced security.bak

Documents

mpi protocol

esmpich2 implementation

aware mpi applications

mpich2 library

standard mpi interface

conventional mpi program

mpiapplication programmers

variety of mpi implementations