April 19, 2023

NSF Middleware InitiativeBuilding for the Future

Kevin Thompson, NSF

Ken Klingenstein, Internet2

Tom Garritano, Grids Center

Mary Fran Yafchak, SURA

Ann West, EDUCAUSE/Internet2

10 April 2003 2

Topics

NMI Overview and New Awards

NMI-EDIT

GRIDS Center

NMI Integration Testbed

NMI and NMI-EDIT Outreach

10 April 2003 3

NSF Middleware Initiative

Purpose

To design, develop, deploy and support a set of reusable, expandable set of middleware functions and services that benefit applications in a networked environment

10 April 2003 4

To allow scientists and engineers the ability to transparently use and share distributed resources, such as computers, data, and instruments

To develop effective collaboration and communications tools such as Grid technologies, desktop video, and other advanced servicesto expedite research and education, and

To develop a working architecture and approach which can be extended

to Internet users around the world.

Middleware is the stuff that makes “transparently use” happen, providing persistency, consistency, security, privacy and capability

A Vision for Middleware

10 April 2003 5

NMI Status

21 Active Awards (Prior to new awards)• 3 Cooperative Agreements• 18 individual research awards

Focus on integration and deployment of grid and middleware for S&E

• Near-term deliverables (working code)• coordination and persuasion rather than standards• Significant effort on interoperability, testing, inclusion

NMI Software Releases, best practices, white papers• NMI Release 1 – May, 2002• NMI Release 2 – Oct, 2002• NMI Release 3 – Apr, 2003• NMI Release 4 - est. Dec, 2003

10 April 2003 6

NMI Organization

• GRIDS Center– ISI, NCSA, U Chicago, UCSD & U Wisconsin

• EDIT Team (Enterprise and Desktop Integration Technologies)

– EDUCAUSE, Internet2 & SURA

• Several additions in 2003

Core NMI Team

Grants for R & D• Year 1 -- 9 grants• Year 2 -- 9 grants• Year 3 (New) -- 10 grants

10 April 2003 7

2003 NSF Middleware Initiative Program Awards

20 awards totaling $9M• 10 “System Integrator” awards

– Focus – to further develop the integration and support infrastructure of middleware for the longer term

– Themes - extending and deepening current activities, and expanding into new areas

• 10 smaller awards focused on near-term capabilities and tool development

– Focus – to encourage the development of additional new middleware components and capabilities for the NMI program

10 April 2003 8

2003 New System Integrator Awards

Butler (UIUC) Disseminating and Supporting Middleware Infrastructure: Engaging and Expanding Scientific Grid Communities

Kesselman (USC/ISI)

Designing and Building a National Middleware Infrastructure (NMI-2)

Klingenstein (UCAID)

Extending Integrated Middleware to Collaborative Environments in Research and Education

Livny (U Wisc)

An Integrative Testing Framework for Grid Middleware and Grid Environments

McMullen (Indiana)

Instruments and Sensors as Network Services: Instruments as First Class Members of the Grid

Pierce, Alameda, Severance, Thomas, and von Laszewski

Collaborative Proposal: Middleware for Grid Portal Development

10 April 2003 9

Other New Awards in 2003

Chase (Duke), Ramakrishnan (MCNC)

Collaborative Research: A Grid Service for Dynamic Virtual Clusters

Gemmil (UAB) NMI-Enabled Open Source Collaboration Tools for Virtual Organizations

Karonis (Northern Illinois) Critical Globus-enabled Implementation of the MPI-2 Standard

Lumsdaine (Indiana) Scalable Fault Tolerance for MPI

Ramachandran (Ga Tech) Exploration of Middleware Technologies for Ubiquitous Computing with Applications to Grid Computing

Saltz (Ohio St. URF) GridDB-Lite: Database Support for Data-Driven Scientific Applications in the Grid

Wright (U Wisc.), Linderoth (Lehigh)

Collaborative Research: MW: Master-Worker Middleware for Grids

Arzberger (UCSD) Pacific Rim Application and Grid Middleware Assembly

10 April 2003 10

Looking Ahead

There will be an NMI solicitation in 2004 Exact funding level not set NMI program is expected to be a primary focus area under

CISE’s new division - Shared CyberInfrastructure October 23 Review at NSF for existing activities among the

Grids Center and EDIT teams

10 April 2003 11

Enterprise and Desktop Integration Technologies (EDIT) Consortium

Ken Klingenstein

Director, Internet2 Middleware Initiative

kjk@internet2.edu

10 April 2003 12

Overview

NMI-EDIT Overview

Research Findings

NMI Release Components

Building on the Future

10 April 2003 13

NMI-EDIT Consortium

Enterprise and Desktop Integration Technologies Consortium

• Internet2 EDUCAUSE, SURA• Almost all funding passed through to campuses for work

Project Goals

• Create a common, persistent and robust core middleware infrastructure for the R&E community

• Provide tools and services in support of inter-institutional and inter-realm collaborations

10 April 2003 14

A Map of Middleware Land

10 April 2003 15

NMI-EDIT Findings

Consensus on inter-institutional middleware standards and maturing architecture to support collaborative applications

Widespread interest in Shibboleth within R&E communities

Credential mapping from core enterprise to Grid service

Grid adoption of SAML in Open Grid Services Architecture (OGSA)

10 April 2003 16

NMI-EDIT Findings (cont.)

Creation and maintenance of a heavily referenced set of design and best practices documents

Effective linkages with International research communities

Discovery and development of campus IT staff, leading to

Influence on both federal and commercial standards

Direct outreach to over 320 institutions

10 April 2003 17

NMI-EDIT Components from Three NMI Releases

Authentication: • WebISO solution, credential mapping from Kerberos to

PKI, policy documents, registry service

Enterprise Directories: • Schemas; operational monitoring and schema analysis

tools; practices in design, groups, and metadirectories

Authorization: • Architecture and related software and libraries for multi-

institution collaboration

10 April 2003 18

NMI-EDIT Components from Three NMI Releases (cont.)

Integration Activities:• Credential mapping from campus to Grid environment,

GLUE schema analysis tool

Education: • Venues for learning about enterprise middleware

including CAMPs and on-line deployment materials for directories

10 April 2003 19

NMI-EDIT: Building for the Future

Authorization infrastructure • Develop authorization architecture to support

–campus applications, both legacy and productivity–network services –inter-realm collaboration efforts–virtual organizations

• Deliverables–An Enterprise Authorization Design and Implementation

Guide, with case histories –A set of tools for use with Authz, including registry

templates for policies, roles, and authorities, a GROUPER group manager, end user interfaces, etc.

10 April 2003 20

NMI-EDIT: Building for the Future (cont.)

Middleware end-to-end diagnostics• How do I resolve a permission denied problem?

–From the user perspective–From the user’s help-desk perspective –From the technical support perspective

• Deliverables first year–Develop an expert group to design an interim

architecture and functionality–Build liaisons to adjacent efforts by other groups–Design standard, extensible logging formats–Create a harvesting tool that can assemble end-end

middleware flows from logs–Work with KX.509, Shibboleth, etc…

10 April 2003 21

New Partnerships

Work with JISC on Virtual Organizations• A key cross-stitch among enterprises for inter-institutional collaborations

• VO’s range from Grids to digital libraries, from earthquake engineering to collaborative curation, from managing observatories to managing rights

Interworkings with Australian, Swiss, French universities

Corporate interactions with MS, Sun, Liberty Alliance, etc

10 April 2003 22

NMI-EDIT:Next Generation Architecture

10 April 2003 23

The pieces fit together…

Campus infrastructure• Developing and encouraging the deployment of identity

management components, tools, and support services

Inter-realm infrastructure• Leveraging the local organizational infrastructure to enable access

to the broader community though–Building on campus identity management infrastructures–Extending them to contain standard schemas and data definitions–Enabling the exchange of access information in a private and secure way–Developing diagnostic tools to make complex middleware interactions

easier to understand

10 April 2003 24

The GRIDS Center: Defining and Deploying

Grid Middleware

presented by

Tom GarritanoUniversity of Chicago and Argonne National Laboratory

10 April 2003 25

NSF Middleware Initiative (NMI)

GRIDS is one of two original teams,the other being EDIT

New NMI teams just announced (Grid portals and instrumentation)

GRIDS releases well-tested, deployed and supported middleware based on common architectures that can be extended to Internet users around the world

NSF support of GRIDS leverages investment by DOE, NASA, DARPA, UK e-Science Program, and private industry

10 April 2003 26

GRIDS Center

GRIDS = Grid Research Integration Development & Support Partnership of leading teams in Grid computing

• University of Chicago and Argonne National Lab• Information Sciences Institute at USC• NCSA at the University of Illinois at Urbana-Champaign• SDSC at the University of California at San Diego• University of Wisconsin at Madison• Plus other software contributors (to date: UC Santa Barbara, U. of

Michigan)

GRIDS develops, tests, deploys and supports standard tools for:• Authentication, authorization, policy• Resource discovery and directory services• Remote access to computers, data, instruments

10 April 2003 27

The Grid: What is it?

“Resource-sharing technology with software and services that let people access computing power, databases, and other tools securely online across corporate, institutional, and geographic boundaries without sacrificing local autonomy.”

Three key Grid criteria:• coordinates distributed resources• using standard, open, general-purpose protocols and

interfaces• to deliver qualities of service not possible with pre-Grid

technologies

10 April 2003 28

R

R

RR

R

R

R

R

R

RR

R

Virtual Organizations

• Distributed resources and people

10 April 2003 29

RR

R

R

R

R

R

RR

R

R

R

Virtual Organizations

• Distributed resources and people• Linked by networks, crossing administrative domains

10 April 2003 30

RR

R

R

R

R

R

RR

R

Virtual Organizations

R

R

• Distributed resources and people• Linked by networks, crossing administrative domains• Sharing resources, common goals

VO-BVO-A

10 April 2003 31

RR

R

R

R

R

R

RR

R

Virtual Organizations

• Distributed resources and people• Linked by networks, crossing administrative domains• Sharing resources, common goals• Dynamic

VO-BVO-A

R

R

10 April 2003 32

R RR

R

R

R

R

R

RRR

R

VO-A VO-B

Virtual Organizations

• Distributed resources and people• Linked by networks, crossing administrative domains• Sharing resources, common goals• Dynamic• Fault tolerant

10 April 2003 33

GRIDS CenterSoftware Suite

Globus Toolkit®. The de facto standard for Grid computing, an open-source "bag of technologies" to simplify collaboration across organizations. Includes tools for authentication, scheduling, file transfer and resource description.

Condor-G. Enhanced version of the core Condor software optimized to work with GT for managing Grid jobs.

Network Weather Service (NWS). Periodically monitors and dynamically forecasts performance of network and computational resources.

Grid Packaging Tools (GPT).  XML-based packaging data format defines complex dependencies between components.

10 April 2003 34

GRIDS CenterSoftware Suite (cont.)

GSI-OpenSSH. Modified version adds support for Grid Security Infrastructure (GSI) authentication and single sign-on capability.

MyProxy.  Repository lets users retrieve a proxy credential on demand, without managing private key and certificate files across sites and applications.

MPICH-G2.  Grid-enabled implementation of the Message Passing Index (MPI) standard, based on the popular MPICH library. 

GridConfig.  Manages the configuration of GRIDS components, letting users regenerate configuration files in native formats and ensure consistency.

KX.509 and KCA. A tool from EDIT that bridges Kerberos and PKI infrastructure.

10 April 2003 35

E-Science Benefits Substantially from GRIDS Components

Large-scale IT deployment projects rely on GRIDS components and architecture for core services• BIRN, the Bioinformatics Research Network• GEON, the Geoscience Network• GriPhyN, Particle Physics Data

Grid, International Virtual Data Grid Laboratory

• NEESgrid, part of the Network for Earthquake Engineering and Simulation

• International projects such as the UK e-Science Program and EU DataGrid

GRIDS standard tools let projects avoid building their own infrastructure• Increases interoperability, efficiency• Prevents “balkanization” of applications

BIRN MRI Data for Brain Imaging

10 April 2003 36

Industrial and International Leaders Move to Grid Services

GRIDS leaders engage a worldwide community in defining specifications for Grid services• Very active working through Global Grid Forum• Over a dozen leading companies (IBM, HP, Platform) have

committed to Globus-based Grid services for their products

NMI-R4 in December will include Globus Toolkit 3.0• GT3 is the first full-scale deployment of new Open Grid Services

Infrastructure (OGSI) spec• Significant contributions from new international partners

(University of Edinburgh and Swedish Royal Institute of Technology) for database access and security

• UK Council for the Central Laboratory of the Research Councils (CCLRC) users rank deployment of GT3 as their #1 priority

10 April 2003 37

Acclaim for GRIDS Components

On July 15, the New York Times noted the “far-sighted simplicity” of the Grid services architecture

The Globus Toolkit has earned:• R&D 100 Award• Federal Laboratory Consortium Award

for Excellence in Technology Transfer

MIT Technology Review named Grid one of “Ten Technologies That Will Change the World”

InfoWorld list of 2003’s top 10 innovators includes two GRIDS PIs

GRIDS co-PI Ian Foster named “Innovator of the Year” for 2003 by R&D Magazine

10 April 2003 38

Future GRIDS Plans

GRIDS is completing its second year in October• Original three-year award, through Fall 2004• Very successful in establishing processes, meeting twice/year release

schedule, defining broadly accepted Grid middleware standards, and increasing public awareness of Grid computing

GRIDS Center 2 plans• Further develop and refine core NMI releases and processes• Deploy tools based on Open Grid Services Architecture • Expand testing capability• Create a federated bug-tracking facility• Public databases: Grid Projects and Deployments System and Grid

Technology Repository• Increase outreach to communities at all levels:

– Existing major Grid projects (e.g., TeraGrid, NEESgrid)– Major projects that should use Grid more (e.g., SEEK, NEON)– New communities not yet using Grid (e.g., Computer-Aided Diagnosis)

10 April 2003 39

Upcoming Tutorials

GRIDS is extremely well-represented at SC03, the supercomputing conference• Tutorials, technical papers, BoFs, demonstrations• Phoenix, AZ, November 15-21• http://www.sc-conference.org/sc2003

GlobusWORLD 2004 conference• Co-sponsored by GRIDS• San Francisco, CA, January 20-23• Academia and Industry both well-represented• http://www.globusworld.org

10 April 2003 40

For More Information

The GRIDS Centerhttp://www.grids-center.org/

NSF Middleware Initiativehttp://www.nsf-middleware.org/

The Globus Alliance http://www.globus.org/

10 April 2003 41

NMI Integration Testbed

NMI Integration Testbed

Mary Fran Yafchak

maryfran@sura.org

NMI Integration Testbed Manager

SURA IT Program Coordinator

Southeastern Universities Research Association

10 April 2003 42

USERS

Implementers

Target Communities

NMI Integration Testbed

About the NMI Testbed

NMI Participation• Developed and managed by SURA

• Evaluate NMI components upon release

• Real life contexts - research projects, enterprise applications and infrastructure

?

futureexpansion

UABUAHUFLFSUGSU

UMichTACCUVA

Sites

NMI Integration Testbed

(USC)

DEVELOPERS

SUPPORTERS

CONTRIBUTORS

http://www.nsf-middleware.org/testbed

10 April 2003 43

Activities this year

1) Evaluation of NMI Releases 2 & 3

2) Continued project & enterprise integration

3) Addition of REU student positions

4) Workshops & Presentations

5) Firing up of intra-Testbed grid

10 April 2003 44

Evaluation of NMI R2 & R3

Once upon a time… NMI R1, completed September 2002. 18 components, 61

reports, focused “across the board”

This past year… NMI R2, completed February 2003. 25 components, 59

reports, focused “across the board” NMI R3, completed August 2003. 30 components, 57 reports,

heavy on grids and newer authn/authz components

Trend towards more “practical” evaluation

10 April 2003 45

Project/Enterprise Integration

Catalyzing advanced infrastructure • Seven sites providing centralized identity & authentication

for multiple active applications– Examples: UAB Faculty/Class emailer, UFL Wireless VPN

control

• Four sites implementing campus grids (various stages)– MGrid, USCGrid, UTGrid (w/TIGRE), GridGroup@GSU

10 April 2003 46

Project/Enterprise Integration

Extending access for existing projects Examples:

UAH: Grid-based applications with NCSA MEAD expedition and NASA Marshall Space Flight Center

UMich, GSU: Access for physicists to DOE experiments (Particle Physics Data Grid, ATLAS, CERN LHC)

Expanded access for new audiences Examples:

GSU: Student access to graphic rendering capability GSU: Distributed muon detector in collaboration with GSU physicists and

Georgia state high schools. NSF REU (Research Experiences for Undergraduates)

10 April 2003 47

NSF REU in NMI

SURA provides admin & management; sites provide experience and mentoring

5 positions at 4 sites: GSU, UMich, TACC/UVa• GSU

– Muon particle detector GRID for K-12– GRID-enabled Applications Catalog

• UMich– NMI components in ATLAS and MGRID

• TACC/UVA– Grid Portals for the NMI Integration Testbed

10 April 2003 48

Intra-Testbed Grid

Rationale - Testbed sites’ interest, expertise and position to contribute

Two foci - Demonstration (showcase) & Research• Showcase lead: Art Vandenberg at GSU• Research leads; Marty Humphrey & Jim Jokl at UVa

Key deliverables (through August 2004) - • Working showcase grid and application catalog• Identification and documentation of cross-campus issues

(barriers to scalability) and recommendations for resolution

10 April 2003 49

NMI Testbed Workshops

1st Testbed Results workshop• April 2003 at Internet2 Spring Meeting

SURA NMI PACS workshops • Small group training in enterprise directories and related

applications (Aug & Sept. 2003)

2nd Testbed Results workshop• Monday, Nov. 3, preceding EDUCAUSE 2003• See Testbed Web site for more details• Registration closes this Friday! (October 17)

10 April 2003 50

NMI Testbed Presentations

At I2 meetings (Spring 2003, Fall 2003)– “Taking Grids out of the Lab and onto the Campus,”

Wednesday, 10/14, 10:30 - 11:45 a.m.– “Advanced Directory Services and Applications,” Thursday,

10/15, 8:45 - 10 a.m.

At EDUCAUSE 2003 annual conference– “Experiences in Middleware Deployment: Teach a Man to

Fish…,” Thursday, 11/6, 3:55 - 4:15 p.m.

At GlobusWorld 2004?– Proposed presentation on the intra-Testbed grid. Waiting to

hear…

Tomorrow

Next

10 April 2003 51

NMI and NMI-EDIT Outreach

Ann West

EDUCAUSE/Internet2

10 April 2003 52

Topics

NMI participation model

NMI-EDIT goals, products, and results

Education opportunities

10 April 2003 53

NMI Outreach Participation Overview

10 April 2003 54

NMI-EDIT Outreach Goals

Informing higher-education constituents about enterprise middleware and available resources.

Guiding a broad selection of campuses interested in implementing enterprise middleware towards deployment and service integration, and assisting them in joining the emerging middleware community to benefit from peer support and developments in that community.

Serving the technical community researching and developing enterprise middleware infrastructures.

Establishing information and support persistence.

10 April 2003 55

NMI-EDIT Outreach Products

Awareness presentations• 22 presentations last year

Workshops and tutorials• 1060 total attendance.• 718 distinct participants • 323 distinct organizations

Content development• Articles• Directory Roadmap

Community work• Minority serving institutions and small colleges• Registrars (AACRAO) and CFOs (NACUBO)• Higher-education systems

Persistence

10 April 2003 56

10 April 2003 57

10 April 2003 58

Education Opportunities

CAMPs• Directory CAMP Feb 3-6, 2004 in Tempe AZ• CAMP/Research CAMP June and November 2004

EDUCAUSE regional and annual meetings

Getting started?• Check out getting started link on www.nmi-edit.org

10 April 2003 59

More Information…

NMI• nsf-middleware.org

NMI-EDIT• www.nmi-edit.org• middleware.internet2.edu

GRIDS Center• www.grids-center.org

10 April 2003 60