osg accounting and visualization - indico...osg accounting and visualization this project is...

OSG Accounting and Visualization

This project is supported by National Science Foundation under Cooperative Agreement OAC-1836650. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Team: Derek Weitzel1, Marian Zvada1, Carl Edquist2

Institutions: 1University of Nebraska—Lincoln, 2University of Wisconsin—Madison

Introduction● OSG Accounting is tasked with recording and visualizing resource utilization on the Open Science Grid● Unique challenges for OSG Accounting include:○ Expansion into new types of resources (public and private clouds, allocation-based HPC resources, and GPU farms),○ Multiple destinations for reporting including WLCG & EGI,○ Record annotation from multiple information sources,○ 100+ accounting sources.

Recent and Future Work● New HTCondor-CE Gratia probe for Hosted CE

● Simplification of visualization interface

● Updated version of ElasticSearch for improvedperformance

● Collect new metrics from data transfers

Total RecordsStored

Job Records Per day

Accounting DataStored

Data Transfer RecordsPer day

Visualization● OSG usage is visualized by hosted Grafana interface

● Data go to WLCG for accounting & pledge calculations

WLCG Integration● Uploads records to APEL accounting message bus

● WLCG normalization (HS06) factors are stored inOSG Topology service

● Collaborate closely with WLCG and U.S. LHC sites for corrections to improve data quality

9.6 Billion 13 TB 1.5 Million 5 Million