Data reprocessing for DZero on the SAM-Grid

Gabriele Garzoglio for the SAM-Grid TeamFermilab, Computing Division

Mar 15, 2005 Gabriele Garzoglio

Overview

The DZero experiment at FermilabData reprocessing

MotivationThe computing challengesCurrent deployment

The SAM-Grid systemCondor-G and Global Job ManagementLocal Job ManagementGetting more resources submitting to LCG

Mar 15, 2005 Gabriele Garzoglio

Fermilab and DZero

DZero

Mar 15, 2005 Gabriele Garzoglio

Data size for the D0 ExperimentDetector Data

1,000,000 ChannelsEvent size 250KBEvent rate ~50 Hz0.75 PB of data since 1998Past year overall 0.5 PBExpect overall 10 – 20 PB

This meansMove 10s TB / dayProcess PBs / year25% – 50% remote computing

Mar 15, 2005 Gabriele Garzoglio

Overview

The DZero experiment at FermilabData reprocessing

MotivationThe computing challengesCurrent deployment

The SAM-Grid systemCondor-G and Global Job ManagementLocal Job ManagementGetting more resources submitting to LCG

Mar 15, 2005 Gabriele Garzoglio

Motivation for the Reprocessing

Processing: changing the data format from something close to the detector to something close to the physics.As the understanding of the detector improves, the processing algorithms changeSometimes it is worth to “reprocess” all the data to have “better” analysis results.Our understanding of the DZero calorimeter calibration is now based on reality rather then design/plans: we want to reprocess

Mar 15, 2005 Gabriele Garzoglio

The computing task

Events 1 BilionInput 250TB (250kB/Event)Output 70TB (70kB/Event)Time 50s/Event: 20,000monthsIdeally 3400CPUs (1GHz PIII) for 6mths (~2 days/file)Remote processing 100%

A stack of CDs as high as the Eiffel tower

Mar 15, 2005 Gabriele Garzoglio

Data processing model

Input Datasets (n files)

Site 1 Site 2 Site m…

Job 1 Job 2 Job n…

Out 1 Out 2 Out n…

(n batch processes per site)

(stored locally at the site)

Merging

Permanent Storage

(at any site)

(n~100: files produced in 1 day)

Mar 15, 2005 Gabriele Garzoglio

Challenges: Overall scale

A dozen computing clusters in US and EUcommon meta-computing framework: SAM-Gridadministrative independence

Need to submit 1,700 batch jobs / day to meet the dead line (without counting failures)Each site needs to be filled up at all time: locally scale up to 1000 batch nodesTime to completion of the unit of bookkeeping (~100 files): if too long (days) things are bound to failHandle 250+ TB of data

Mar 15, 2005 Gabriele Garzoglio

Challenges: Error Handling / Recovery

Design for random failuresunrecoverable application errors, network outages, file delivery failures, batch system crashes and hangups, worker-node crashes, filesystem corruption...

Book-keeping of succeeded jobs/files: needed to assure completion without duplicated eventsBook-keeping of failed jobs/files: needed for recovery AND to trace problems in order fix bugs and to assure efficiencySimple error recovery to foster smooth operations

Mar 15, 2005 Gabriele Garzoglio

Available ResourcesSITE #CPU 1GHz-eq. STATUSFNAL Farm 1000CPUs used for data-

taking

Westgrid 600CPUs readyLyon 400CPUs readySAR (UTA) 230CPUs readyWisconsin 30CPUs readyGridKa 500CPUs readyPrague 200CPUs readyCMS/OSG 100CPUs under testUK 750CPUs 4 sites being

deployed-------------------------------------------------------------

2800CPUs (1GHz PIII equiv.)

Mar 15, 2005 Gabriele Garzoglio

Overview

The DZero experiment at FermilabData reprocessing

MotivationThe computing challengesCurrent deployment

The SAM-Grid systemCondor-G and Global Job ManagementLocal Job ManagementGetting more resources submitting to LCG

Mar 15, 2005 Gabriele Garzoglio

The SAM-GridSAM-Grid is an integrated job, data, and information management systemGrid-level job management is based on Condor-G and GlobusData handling and book-keeping is based on SAM (Sequential Access via Metadata): transparent data transport, processing history, and book-keeping.…lot of work to achieve scalability at the execution cluster

Mar 15, 2005 Gabriele Garzoglio

SAM-Grid Diagram

SiteSite SiteSite SiteSite

Resource Selector

Info Collector

Info Gatherer

Match Making

User InterfaceUser Interface User InterfaceUser Interface

SubmissionGlobal Job Queue

Grid Client

SubmissionSubmission

User InterfaceUser Interface User InterfaceUser Interface

Global DH ServicesSAM Naming Server

SAM Log Server

Resource Optimizer

SAM DB ServerRC MetaData Catalog

Bookkeeping Service

SAM Stager(s)

SAM Station(+other servs)

Data Handling

Worker Nodes

Grid Gateway

Grid/Fabric Interface

JIM Advertise

Local Job Handling

Cluster

AAA

Dist.FS

Info Manager

XML DB server

Site Conf.Glob/Loc JID map...

Info Providers

MDS

MSS Cache Site

Web ServGrid Monitoring

User Tools

Flow of: job data meta-data

Mar 15, 2005 Gabriele GarzoglioJO

B

Computing Element

User Interface

SubmissionService

Job Management DiagramUser

Interface

User Interface

ResourceSelection

Match Making Service

Information Collector

ExecSite #1

Match Making Service

Match Making Service

Computing Element

Grid Sensors

Execution Site #n

Submission Service

Submission Service

Grid Sensors

Computing Element

GenericService Generic

Service

Information Collector

Information Collector

Grid Sensor

s

Grid Sensor

s

Grid Sensor

s

Grid Sensor

s

Computing Element

Computing Element

GenericService

GenericService

ext.algo

ext.algo

Grid/Fabric

Interface

Grid/Fabric

Interface

Grid/Fabric

Interface

Grid/Fabric

Interface

Mar 15, 2005 Gabriele Garzoglio

Fabric Level Job Management

ExecutionSite

Grid/Fabric InterfaceGrid/Fabric

Interface

Grid/Fabric Interface

JOB

SAM Station

SAM StationSandbox

Facility

Sandbox Facility

SAM Station

XML Monitoring Database

Batch System Adaptor

Sandbox Facility

Batch System

Worker Node

XML Monitoring Database

XML Monitoring Database

Batch System

Batch System

Worker Node

Worker Node

Batch System

Adapter

Batch System

Adapter

Job enters the SiteLocal Sandbox created for job (user input, configuration, SAM client, GridFTP client, user credentials)

Local services notified of jobBatch Job submission details requestedJob submittedJob starts on Worker nodePush of monitoring info startsJob fetches SandboxJob gets dependent products and input dataFramework passes control to applicationGrid monitors status of jobUser requests status of jobJob stores output from applicationStdout, stderr, logs handed to Grid

Mar 15, 2005 Gabriele Garzoglio

How do we get more resources?We are working on forwarding jobs to the LCG GridA “forwarding-node” is the advertised gateway to LCGLCG becomes yet another batch system… well, not quite a batch system

Need to get rid on the assumptions on the locality of the network

SAM-Grid LCGFwd-node

VO-srv

Mar 15, 2005 Gabriele Garzoglio

ConclusionsDZero needs to reprocess 250 TB of data in the next 6-8 monthsIt will produce 70 TB of output, processing data at a dozen computing centers on ~3000 CPUsThe SAM-Grid system will provide the meta-computing infrastructure to handle data, job, and information.

Mar 15, 2005 Gabriele Garzoglio

More info at…

http://www-d0.fnal.gov/computing/reprocessing/

http://www-d0.fnal.gov/computing/grid/

http://samgrid.fnal.gov:8080/

http://d0db.fnal.gov/sam/