Distributed Hybrid Earthquake Engineering Experiments:

Experiences with a Ground-Shaking Grid Application

Laura Pearlman, Carl Kesselman, Sridhar GullapalliUSC Information Sciences Institute

B.F. Spencer, UIUCIan Foster, Paul Hubbard, ANL

Chuck Severance, University of MichiganJoe Futrelle, Kathleen Ricker, NCSA

laura@isi.edu

NEESgrid

• NEES: NSF-funded project in support of earthquake engineering.

• NEESgrid: National earthquake engineering collaboratory– Deployment complete September 30, 2004– Operational through 2014

• This talk is an overview of one application of NEESgrid.

Outline• The Application: Distributed Hybrid

Experiments• Architecture• Experiences: The MOST Experiment• Security Considerations• Related Work• Conclusions

Traditional Methods• Computational Simulations• Physical Experiments

– Build and instrument a specimen.

– Subject it to forces.– Record sensor

measurements throughout the experiment.

Physical System

Physical System

Control / Data Acquisition System

Physical Testing Apparatus

Characteristics of Physical Experiments

• Specimens can be large-scale (50 tons or more)

• Take months to construct• Some steps can’t be “undone”• Require specialized equipment and

facilities

Hybrid Experiments

Computational Simulation

Control / Data Acquisition System

Physical System

Physical System

Pre-NEESgrid Hybrid Experiments

• Physical and computational simulations occur at same site, usually with communication via a shared memory backbone such as SCRAMnet.

• Proprietary communication protocol between the computational and physical system

Structural ExperimentStructural Experiment

Geotechnical ExperimentGeotechnical ExperimentStructural ExperimentStructural Experiment

Geotechnical ComputationGeotechnical ComputationStructural ComputationStructural Computation

Soil-Structure Test

Distributed Hybrid Experiment Characteristics

• Multiple physical experiments– At different geographic sites– Resources owned by different organizations– Heterogeneous control platforms

• Multiple computational simulations– At different geographic sites– Resources owned by different organizations– Heterogeneous simulation platforms– Sometimes used in place of physical

experiments

Experiment Variations

1/51/5thth-scale LBCB-scale LBCB

Full-scale LBCBFull-scale LBCBLBCB simulator (Computer Model)LBCB simulator (Computer Model)

Equipment Site Central Services

NEESgrid Services

NEES-POP

Control System

DAQ System

telecontrol

streamingdata

Files

Data/metadata

Repository,Credential

cache

Web portal

Ingestiontool

Control client

Files

Observers

Physical System

Physical System

Telecontrol Service Requirements

• Uniform interface for heterogeneous systems

• Impose minimal requirements on control systems.

• Support for recovery from transient errors• Performance requirements vary widely (1

ms for some experiments; 10 seconds for others).

NEESgrid Teleoperation Control Protocol (NTCP)

• Transaction-based• State model

– provides for at-most-once execution– guarantees that a control point is doing at

most one thing at a time– guarantees that requests involving any

control point are executed in the order received

• Protocol allows for negotiation of request parameters prior to execution.

NTCP Plugins

Control System B

Control System B

Control System A

Control System A

SimulationSimulation

NTCP NTCP

NTCP

ClientPlugin Plugin

Plugin

Multi-site Online Simulation Test (MOST)

• First distributed hybrid experiment in NEESgrid (and, we believe, in the US).

• Combined physical experiments at the University of Illinois (UIUC) and University of Colorado (CU) with a simulation at NCSA.

• Simulations and experiments created by earthquake engineers at UIUC and CU.

The MOST Structure

SAC Consortium Benchmark Structure

gx

Pinned Connection

Moment Connections

gx

NCSA Computational Model

m1

f1

UIUC

Experimental Model

gx

f1

m1

f2f2

U. Colorado

Experimental Model

gx

The MOST Substructures

Slide courtesy of Bill Spencer and Narutoshi Nakata, UIUC

Computation-Only Test

SimulationCoordinator

NTCPServer

NTCPServer

NTCPServer

Mplugin

Mplugin Mplugin

UIUCSimulation

NCSASimulation

CUSimulation

Colorado

NCSA UIUC

MOST Components

CoordinatingSimulation

NTCPServer

NTCPServer

NTCPServer

Mplugin

MpluginShore-Western

Plugin

NCSASimulation

Matlab(xPC host)

Colorado

NCSA UIUC

Matlab(xPC target)

Control Components in MOST

Globus Toolkit version 3

NTCPServer

MpluginShore-Western

PluginNTCP

Java API

MpluginJava API

NTCP Plugin Interface

MpluginMatlab toolbox

NTCPMatlab toolbox

The MOST Event

Mini-MOST

Security Considerations

NTCP Security Features

• Authentication via GSI• Authorization via gridmap lookups

and a policy plugin.• Control plugin architecture allows for

running control system on a separate host, isolated behind a firewall

• Control plugin also allows for setting of local limits.

Ongoing NEES Experiments

• Fast-MOST: Berkeley, Buffalo, CU, and UIUC are performing a MOST-like experiment with stricter performance requirements.

• MISST: RPI, UIUC, Lehigh: soil-structure interaction.

• UC-Davis: Soil study with remote control of a robot arm.

Other Related Work

• Multi-Site Pseudo-Dynamic Substructure Testing: – Method for dividing structures into

substructures and testing separately– Developed in Japan in 1999– MOST simulations are based on this method

• TeleScience Portal – using NTCP to control electron microscopes.

• NMI Common Instrument Middleware Architecture (CIMA) project

Conclusions• Distributed hybrid earthquake engineering

experiments are a “real” grid application.• NTCP works:

– Earthquake engineers are developing and using their own clients and plugins.

– Its failure-recovery features have allowed experiments to continue after transient network failures

• But there’s still work to do…– NTCP requires some shared knowledge between developers

of clients and plugins. Better use of service data could fix this.

– Some aspects of the protocol need to be “cleaned up”– There’s not much support for recovering from backend

failures.

Acknowledgements

Dan Abrams, Cristina Beldica, Ian Buckle, Ben Clifford, Mike D’Arcy, Amr Elnashai, Tom Finholt, David Gehrig, Tomasz Haupt, Dan Horn, Erik Johnson, Young Suk Kim, Dan Kuchma, Lee Liming, Ravi Madduri, Doru Marcusiu, Gilberto Mosqueda, Narutoshi Nakata, Gokhan Pekcan, Pawel Plaszczak, Tom Prudhomme, Chase Phillips, Andrei Reinhorn, Hatem A Seliem, Benson Shing, Eric Stauffer, Bozidar Stojadinovic, Guangqiang Yang, and Nestor Zaluzec.

For More Information

• NEESgrid web site: http://www.neesgrid.org