The Open Science GridOSG

Ruth PordesFermilab

2

What is OSG? A Consortium of people working together to

Interface Farms and Storage to a Grid and

Researchers using these resources by adapting their applications to run on the Grid and

Software developers providing middleware and

A project that provides the Operations, Support, Training and Help to make it effective.

3

Who is OSG ?

Large global physics collaborations: US ATLAS, US CMS, LIGO, CDF, D0, STAR

Research collaborations such as Mariachi, GROW,

Grid technology groups: Condor, Globus, SRM, NMI

Many DOE Labs and DOE/NSF sponsored University IT facilities and

Partnerships with TeraGrid and EGEE, and Campus Grids such as TACC, GLOW etc.

4

OSG Consortium

5

When is OSG ? Grown from of grass-roots collaboration of

GriPhyN, iVDGL and PPDG participants in 2003.

5 years of funding starting ~9/2006 from DOE SciDAC-II and NSF MPS and OCI

Deliver to US LHC and LIGO scales in 2008 and 2009:— Need to routinely distribute data at 1-5 Gbps over 10-50 sites.— Need to routinely exceed 10,000 running jobs per client— Need to reach 99% success rate for 10,000 jobs per day submission under

heavy load

Active engagement effort centered at RENCI to include new sciences.

6

The OSG Map Aug-2006

7

OSG’s world is flat - a Grid of Grids - from Local to Global

Global ScienceCommunity Systems

e.g. FermiGrid, NWIC

Local Campus And RegionalGrids

NationalCyberInfrastructures

for Science

e.g. OSG - TeraGrid

e.g. CMS , D0

8

From the Local (Campus) Grids to the Wide Area GridWithin an organization a Local Grid provides for

simplicity in sharing and efficiency in purchasing and administration.

However, researchers collaborate outside the bounds of a single campus.

With a uniform environment acting globally is just an extension of acting locally.

Open Science Grid focuses on interoperation of the local and the wide area.

9

A resource can be accessed by a user via the campus, community or national grid.

A user can access a resource with a campus, community or national grid identity.

10

Example Uses

High Energy Physics ATLAS - >15 Million proton collision events simulated at 10

minutes each CMS - >70 Million events simulated, reconstructed and analyzed

Biology - GADU Populates databases from search and analysis of similarities and

differences among thousands of publicly available genome and protein sequences and metabolic pathways.

Gravitational Wave Physics - LIGO Data grid Eases grid tools to ensure that 9 computing sites have a copy of the

interesting data, and researchers at 36 LSC institutions use the LDG to find the data they need."

Math research

Education Grid Summer Workshop teaches students to run jobs on OSG and

TeraGrid.

11

Running (and monitored) “OSG jobs” in 06/06.

12

Example GADU run in 04/06

13

Integration Testing of the System

Multi-site Integration Grid tests new OSG Releases and Configurations.

Software Readiness and Validations occur before deployment on the Integration Grid.

Integration Grid SitesIntegration Grid Sites

14

CMS - US part of a Global Community Grid

GermanyTaiwan UKItaly

Data & jobs moving locally, regionally & globally within CMS grid.

Transparently across grid boundaries from campus to global.

Florida

USA

CERN

Caltech

Wisconsin

UCSD

France

Purdue

MIT

UNL

OSG

EGEE

15

How do People and Organizations Participate? VO Registers with with Operations Center

Signs VO Agreement

User registers with VO User added to VOMS of one or more VOs. VO responsible for users to sign AUP. VO responsible for VOMS service support.

Sites Register with the Operations Center Signs the Service Agreement. Agree on which VOs to support (striving for default admit)

VOs and Sites provide Support Center Contact and join Operations groups

16

Community Documentation..

17

18

OSG Principles

Systems are Virtual Organization scoped -- groups of users working together with a shared, common environment.

Sites maintain control, authority and management of use of their Grid accessible resources.

Opportunistically available resources are beneficial. Priorities are governed by Policies (role based within a VO).

The distributed system is heterogeneous. Information must be provided to allow applications to know which resources they can successfully use.

19

VO software stacks that Rely on the Virtual Data Toolkit

NSF Middleware Initiative (NMI): Condor, Globus, Myproxy

Virtual Data Toolkit (VDT) Common Services NMI + VOMS, CEMon (common EGEE

components), MonaLisa, Clarens, AuthZ

OSG Release Cache: VDT + Configuration, Validation, VO management

LHCServices & Framewor

k

LIGOData Grid

OSG VOFramework

Infr

ast

ruct

ure

Applic

ati

ons …Bio

Services &Framewor

k

20

What is the VDT?

A collection of software Grid software (Condor, Globus and lots more) Virtual Data System (Origin of the name “VDT”) Utilities

An easy installation Goal: Push a button, everything just works Two methods:

Pacman: installs and configures it all RPM: installs some of the software, no configuration

A support infrastructure

21

Who uses the VDT?

Open Science Grid

LIGO Data Grid

LCG LHC Computing Grid, from CERN

EGEE Enabling Grids for E-Science

22

What software is in the VDT?

Security VOMS (VO membership) GUMS (local authorization) mkgridmap (local authorization) MyProxy (proxy management) GSI SSH CA CRL updater

Monitoring MonaLISA gLite CEMon

Accounting OSG Gratia

Job Management Condor (including Condor-G &

Condor-C) Globus GRAM

Data Management GridFTP (data transfer) RLS (replication location) DRM (storage management) Globus RFT

Information Services Globus MDS GLUE schema & providers

23

Client tools Virtual Data System SRM clients (V1 and V2) UberFTP (GridFTP client)

Developer Tools PyGlobus PyGridWare

Testing NMI Build & Test VDT Tests

What software is in the VDT?

Support Apache Tomcat MySQL (with MyODBC) Non-standard Perl modules Wget Squid Logrotate Configuration Scripts

And More!

24

Due diligence to Security

Risk assessment, planning, Service auditing and checking

Incident response, Awareness and Training, Configuration management,

User access Authentication and Revocation, Auditing and analysis. End to end trust in quality of code executed on remote CPU -signatures?

Identity and Authorization: Extended X509 Certificates OSG is a founding member of the US TAGPMA. DOEGrids provides script utilities for bulk requests of Host certs, CRL

checking etc. VOMS extended attributes and infrastructure for Role Based Access

Controls.

25

Operations Model

Real support organizations

often play multiple roles

Lines represent communication paths and, in our model, agreements.

We have not progressed very far with agreements yet.

Gray shading indicates that OSG Operations composed of effort from all the support centers

26

The OSG VO

A VO for individual researchers and users.

Managed by the OSG itself.

Learn how to use the Grid!

27

In Summary…

A production grid is the product of a complex interplay of many forces:

Resource providers Users Software providers Hardware trends Commercial offerings Funding agencies Culture of all parties involved …

28

Where do you learn more?

www.opensciencegrid.org

http://vdt.cs.wisc.edu

http://osg.ivdgl.org/twiki/bin/view