WLCG-RUS

An Extensible Solution to Grid Accounting & Usage Monitoring

EGEE 3rd User ForumX. Chen, A. KhanBrunel University

Who am I?

• PhD Candidate

• Current Research Scope– Grid Computing, Accounting;– Middleware Solutions;

• OGF– Co-chair of OGF UR working group;– Member of OGF RUS working group;

Outline

• WLCG-RUS Overview

• Component Architecture

• Design Status

• Conclusion

Overview

• RUS– Resource Usage Service– Resource Management in the context of

OGSA;– Aims at providing Grid resource logging and

tracking interfaces;– OGF RUS and OGF UR spec.– Mainly used to enable Grid accounting &

Usage Monitoring

Grid Accounting

• Many Production Grids provide resource usage logging and tracking facilities– e.g. NorduGrid (SGAS being accepted as one of key

service in Globus), EGEE/WLCG (DGAS), and OSG (Gratia);

• Issues– Heterogeneous and Grid-specific solution;– XML:DB persistence only– Gaps between Custom usage schema and OGF UR;– Lacking of interoperability in multi-Grid environment

Accounting In WLCG

• Multi-Grid Environment– Resources from three Grid peers (OSG, NorduGrid, and EGEE) – Most of EGEE services are reused;– WLCG Resource Broker;

• Interoperability

– Get usage data from individual accounting service;

– Get usage data from heterogeneous usage storages (mostly relational DB);

WLCG-RUS• Standardization

– Full OGF RUS compatible– OGF UR version 1.0 as uniform data representation as message

level;

• Customization– at persistence level– Heterogeneous data persistence (Relational, file and XML);– Flexibly data provision (DGAS, Gratia, SGAS & UR generators);

• Advanced Features– Summarization: aggregate UR in summary format (e.g. Total usage

of a specific VO)

Framework

• Based on JISC funded Project:– “Review of Grid Accounting and Usage Monitoring”

(Manchester & Brunel)– http://www.jisc.ac.uk/media/documents/programmes/einfrast

ructure/jisc_aum_final_report_wth.pdf

• Proposed Framework

Client Side

UR Generator

RUS Client

RUS Service

Access Control

Configuration Manager

UR Mapping Session Management

RUS Operation Logics

SOAP

WLCG-RUS Architecture• Layered Architecture

Providing RUS-compatible SOAP Message and Aggregation Extensions

Delegate client-side requests to appropriate command

RUS operation interfaces allowing custom implementations

Property file-based configuration information for individual components

Provide Access Control interface and default XACML implementation

Enable flexible search, update dialects (e.g. SQL, Xupdate, and Xquery)

Aggregator interface for custom algorithms for summarisation

Data Access Object that allows various usage persistence access and Mapping to Usage Records

Messaging• RUS Core WS-I Rendering (draft)

– https://forge.gridforum.org/sf/go/artf6015– http://forge.ogf.org/sf/go/artf6090

• Extensions– Grouping

• Allowing summarisation by grouping criteria;

• An alternative and easier search criteria;

– Sorting• Sorting usage retrieval results

Example: Extract Request Message• Extract and summarise Disk Usage & CPU Duration where

“VO=cms” and “VO=altas” on this month, sorted by Disk Usage;

• Alternative, usage selection can be specified with RUS request body as defined within RUS core spec.

Example: Extract Request Message (Cont.)

Example: Extract Response Message• Response message should return a single usage record that

represented as a summary usage record

• An extension, <urf:Resource description=“NumberOfRecordAggregated” /> is used to indicated total amount of records taken effect on this aggregation.

UR Modelling• UR Modelling

– Derived From OGF-UR 1.0 for those use custom usage record representation;

– Entity Type• Usage properties are shared by reference;• Nine entities

– Value Type• Usage properties are shared by value;• All other usage properties defined in OGF-UR 1.0

Entity Relationship

Default Mapping Strategies

• To bring relational usage storage into RUS• XRM

– XML-Relational Mapping

– Based on JAXB and Hibernate

– Automatic creation of default relational representation of OGF-UR schema;

– Hibernate file-based configuration for custom relational representation

An Example

• Disk Usage Table– Association to a single record;– i.e. One-to-Many record-disk association– The disk_usage table is generated as

following:

Data Access Framework• Generic Data Access Interface;

– Extensive access to relational DB, XML:DB, file system, other service (OGSA-DAI, DGAS, Gratia, and etc.)

– Default DAO implementation provides access through Hibernate engine;– DAO pattern– Abstraction while customization, flexibility and polymorphism.

Security

• Authentication– TSL and mutual-authentication

• Access Control– Coarse-grain Access Control on invocation

of RUS operations (insert, update, extract, delete and auditing)

– Fine-grain Access control on per usage record basis.

Access Control• Role Based Access Control

– Roles are defined according to URF entities (see session1: WLCG-RUS XOM)

– Four default Roles derived from RUS Entities• User

(UserEntity/urf:UserIdentity/ds:KeyInfo/ds:X509Data/ds:X509SubjectName);

• Machine Manager(MachineEntity/urf:MachineName);

• Host Manager(HostEntity/urf:Host);

• Project Manager (ProjectEntity/urf:ProjectName);

– One Role for administration• RUS administrator (for all permissions)

– Roles derived from URF extension framework– e.g. Site Manager (urf:Resource/@description)

Access Control Rules• RULE 1: A user is only allowed to “extract”

usage records on his/her name – either

/urf:Usage/urf:UserIdentity/urf:GlobalUserName/text()

– or /urf:Usage/urf:UserIdentity/ds:KeyInfo/ds:X509Data/ds:X509SubjectName/text()

– depending on deployment environment

Access Control Rules (cont..) RULE 2: Administrator has full permission on ALL RUS

operations upon ALL persistent usage records

RULE 3: Other roles (project manager, host manager, machine manager, and extensive managers) have permissions on “insert”, “extract” and “audit” operations ONLY upon relevant usage records; e.g. A CMS project manager can only execute “insert”,

“extract” and “audit” operations on usage records where “//urf:ProjectName/text()” value is equal to “CMS”;

e.g. The Brunel Tier 2 site manager can only execute “insert”, “extract” and “audit” operations on usage records where“//urf:Resource/@description” value is equal to “Brunel-Tier2”;

Role Configuration• WLCG-RUS schema;• An example

Authorization Framework• Composed of two abstract components:

– Authorizer• The component that performs access control check;

• For both coarse- and fine-grain access control;

– Authorizer Factory• the factory component that creates an instance of authorizer

implementation;

Default Implementation• Default Authorizer

– Based on XACML RBAC 1.0 profile;

– Configurable for custom role definition according to URF extension framework;

– Auto-generation of XACML policy sets;

– Supporting policy persistence;

– Simplified configuration (ease writing verbose XACML policy files)

– High performance (through XACML policy evaluation ONLY when fine-grained access control);

• Default Authorizer Factory– Creation of default authorizer;

Configuration• Runtime Configuration

– Property-based configuration;– Parameters includes DAO factory, Aggregator factory,

Filter factory;

• UR Mapping Configuration– Default UR mapping is based on hibernate-mapping

configuration file (XML-based)

• Deployment Configuration– WS-Management configuration (see implementation

slide)

Runtime: Insertion Example

Accounting with WLCG-RUS

Interoperability

Implementation

• Based on Sun WS-Man platform– Web Service for Management;– Usage Record as a management resource;– Enabling enumeration for large number of query

results;– Extensible through WS-Management interfaces;

• Hibernate– For Default UR Mapping to relational usage

representation;

Development Status

• Version 1.0– Requirement Analysis (done)– System Design (done)– Implementation (debugging)– First alpha release (expected at Feb. Or

March)– Test (Planned to deployed at Tier-2 site,

Brunel-Tier2?)

Conclusion & Future Work

• WLCG-RUS– Provides extensible solution to RUS;– A development framework;– Default implementation for relational usage

persistence (milestone);– Dynamic XML-Relational mapping;– Aggregation;

• Future work– Version 2 with evolvement of UR 2.0

The End

• Thanks