towards an information system product team

EGEE-III INFSO-RI-222667

Enabling Grids for E-sciencE

www.eu-egee.org

EGEE and gLite are registered trademarks

Towards anInformation System

Product Team

David Horat

EGEE’09, Barcelona, Spain

24th September, 2009

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Objectives

1. Where we are today

2. Where we might be tomorrow

3. Managing the transition

4. Software planning

Towards an Information System Product Team 2

• Today • … • Tomorrow

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Where we are today

Towards an Information System Product Team 3

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Where we are today

Towards an Information System Product Team 4

JRA1 SA3Middleware Engineering Integration Testing

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667 Towards an Information System Product Team 5

Where we are today

gLite CVS

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667 Towards an Information System Product Team 6

Where we are today

Savannah

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667 Towards an Information System Product Team 7

Where we are today

Twiki

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667 Towards an Information System Product Team 8

Where we are today

ETICS

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667 Towards an Information System Product Team 9

Where we are today

SA3

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Where we are today

Towards an Information System Product Team 10

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Where we might be tomorrow

Towards an Information System Product Team 11

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Where we might be tomorrow

1. Self Contained Software Unit– Self sufficient– Interface (What we deliver and how?)

2. Same as now with minor changes– Plus certification– Interface (What we deliver and how?)

Towards an Information System Product Team 12

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Managing the transition

Towards an Information System Product Team 13

Information System Product Team• Areas to consider

– Hosting options– Building system– Certification– Release management

• Current products– BDII– GLUE 2.0– GIP– Etc.

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Hosting options

Towards an Information System Product Team 14

[1] It is really an internal messaging system rather than a standard mailing list[2] You can use Google Sites.[3] You can use Google Groups.[4] You can use Google Groups.[5] CVS and SVN are imported into Bazaar format. No native support.[6] It is really an internal blog rather than a standard mailing list

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Hosting options: CERN SVN

Towards an Information System Product Team 16

• CVS SVN• Savannah + Twiki Trac

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Building system

Towards an Information System Product Team 17

• Best practices in Free Software?

• Get closer to the OS?

• Include components in OS distributions?

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Building system

A possibility …

The Fedora Build System

koji build <repo_tag> <source><repo_tag>: The repository to use for dependencies<source>: CVS URL, SVN URL, source RPM, etc.

• Dependencies resolved using the spec file

Towards an Information System Product Team 18

Koji

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Building system

Towards an Information System Product Team 19

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Building system

• Pros– Community supported– Best practices implementation– Multiple source types– One command– Process driven from the spec file– Tag based approach

• Cons– RPM based– Difficulty to build debs

Already being discussed in the community

– Other possibilities? The Debian Build System

Towards an Information System Product Team 20

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Certification

Towards an Information System Product Team 21

• Improve automatic tests• Virtual machine images• Automatic setup• The testbed

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Certification: Automatic tests

• Bash scripts– Exit value 0: OK– Exit value != 0: ERROR

Towards an Information System Product Team 22

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Certification: VM images

Towards an Information System Product Team 23

• VNode: XEN virtual machines– SL4-32-DH– SL4-64-DH– SL5-64-DH

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Certification: Automatic setup

Towards an Information System Product Team 24

1. Update the system

2. Select repository

3. Install nodes

4. Install patches

5. Configure the system

6. Configure the node using YAIM

7. Run basic tests

And now we can start certifying …

./yaimgen.sh <input file> <log file>

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Certification: The testbed

Towards an Information System Product Team 25

TopBDII

SiteBDII

ResourceBDII

ResourceBDII

SiteBDII

ResourceBDII

ResourceBDII

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Certification: The testbed

Towards an Information System Product Team 26

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Certification: The testbed

Towards an Information System Product Team 27

TopBDII

SiteBDII

ResourceBDII

ResourceBDII

SiteBDII

ResourceBDII

ResourceBDII

BDII

Web Config

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Release management

Towards an Information System Product Team 28

• Announcements• Bug tracking• Project portal• Code browser• XML-RPC API

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Release man.: Announcements

Towards an Information System Product Team 29

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Release man.: Bug tracking

Towards an Information System Product Team 30

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Release man.: Bug tracking

Towards an Information System Product Team 31

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Release man.: Project portal

Towards an Information System Product Team 32

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Release man.: Code browser

Towards an Information System Product Team 33

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Release man.: XML-RPC API

• Web Service• XML-RPC• Doc: http://trac-hacks.org/wiki/XmlRpcPlugin

• Manage• Tickets• Wiki• Search

Towards an Information System Product Team 34

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Release man.: XML-RPC API

import xmlrpclib

url = "https://user:password@svnweb.cern.ch/no_sso/trac/gridinfo/login/xmlrpc"

server = xmlrpclib.ServerProxy(url)multicall = xmlrpclib.MultiCall(server)

for ticket in server.ticket.query("owner=horat"): multicall.ticket.get(ticket) print map(str, multicall())

Towards an Information System Product Team 35

Retrieving my tickets in Python

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Software planning

• Software released– BDII– BDII File Editor– GStat – Deployment statistics

• Software in which we collaborate– GLUE Schema– Yaimgen

Towards an Information System Product Team 36

Enabling Grids for E-sciencE

EGEE-III INFSO-RI-222667

Software planning

• To come …– GLUE 2.0 roll out– BDII maintenance

Performance and scalability improvement Investigate a more decentralized model

– GStat 2.0 release Validation probes Web application

– Yaimgen More and more automation

Towards an Information System Product Team 37

EGEE-III INFSO-RI-222667

Enabling Grids for E-sciencE

www.eu-egee.org

EGEE and gLite are registered trademarks

David Horat

david.horat@cern.ch

http://cern.ch/horat

Thank you!