CernVM NewsPredrag Buncic (CERN/PH-SFT)

CernVM ProjectApproved in 2007 as R&D activity under extra funding (White Paper)Portable Analysis Environment using Virtualization Technology (WP9)Led by SFT Group in CERN PH DepartmentStarted in January 2008Funding for 4 (2+2) yearsFellow, student + 0.5 staff

MotivationEmerging technologiesMulti and many core CPUsMaking laptop/desktop ever more powerful and underutilisedHardware support for virtualizationMinimal performance penalty for application running in Virtual MachineVirtualization software is quickly becoming integral part of OS or given as a free add-onKVM, Xen, Vmware Player & Server, Suns VirtualBox

Old problemsMillions of lines of codeComplicated software installation/update/configuration procedure, different from experiment to experimentOnly tiny portion of it is really used at runtime in most of the casesOften incompatible or lagging behind OS version on desktop/laptop

Project GoalsUse virtualization and extra cores to get extra comfortHoping that this will come at acceptable cost in terms of performanceDecouple application lifecycle from evolution of system infrastructureClose to zero effort to install, maintain and keep up to date the experiment softwareReduce the cost of software development by reducing the number of compiler-platform combinations

How do we want to do that?Build a thin Virtual Software Appliance for use by LHC experiments

This appliance should provide a complete, portable and easy to configure user environment for developing and running LHC data analysis locally and on the Gridbe independent of physical software and hardware platform (Linux, Windows, MacOS)

Key Building BlocksrPath Linux 1 (www.rpath.org)Slim Linux OS binary compatible with RH/SLC4rAA - rPath Linux Appliance AgentWeb user interfaceXMLRPC API CVMFS - CernVM file systemRead only file system optimized for software distributionAggressive cachingOperational in offline modeFor as long as you stay within the cacherBulderBuild VM images for various platforms

CernVM componentgroup-cernvm(core packages)group-cernvm-devel(development tools)group-slc4(SLC4 compatibility libs)group-groups and extra packages required by experiment100 MBcompat-db4compat-opensslcompat-linstdc++slc3compat-libxml2compat-readlinecompat-tclcompat-tkgroup--desktopLightweight X working environemnt

Thin Software ApplianceCacheHTTPDSoftwareRepository10 GB1 GB0.1 GBLAN/WAN(HTTP)

CernVM File SystemAppOn same host:/opt/-> /mnt/cmfs/open(/opt/lcg)On File (Web) Server /opt/CacheKernelNFSLFSFUSECernVMFuseHTTP /opt/aliceRSYNCV.O. ADMINUSERS /opt/alice /opt/alice /opt/alice

Web UI

XML-RPC APIimport xmlrpclibimport osurl = 'http://' + user + ':' + password + '@' + host + ':8004/rAA/xmlrpc'server = xmlrpclib.ServerProxy(url)r = server.cernvm.Config.configUpdate('http://cern.ch/cernvm/config', 'ALICE', # (can be None,ATLAS, ALICE, LHCB, CMS) 'off', # (on/off - if on it will make cvmfs locally writeable) 'off', # (on/off - enable/disable grid UI) 'off') # (on/off - enable VO profile)print r["message"]##### Here we create initial user 'test' with the same password as Web admin####r = server.cernvm.User.userUpdate('test', password,password)print r["message"]##### Select grid UI version####r = server.cernvm.Config.configGridUIVersion("")print r["message"]

Developing CernVM

2-4 developersMonthly releasesBi-weekly meetings with experimentsGood collaboration with experiments ATLAS, ALICE, LHCb CMS is joining nowInitial feedback from the expertsFrom version 1.0 (end of October) it will be offered to wider publicThanks to Yves for constructive comments Hope to get soon more input from ALICE

CernVM Release TimelineJANFEBMARAPRMAYJUNJULAUGSEPOCTNOVDEC-Preparation Release 0.5- Release 0.6On time!Release 1.0 Scheduled for end of OctoberKickoff Workshop- Release 0.8- Release 0.7- Release 0.91 (RC1)-Release 0.92 (RC2)

0.92 Release (RC2)Available for download fromhttp://rbuilder.cern.ch/project/cernvm-devel/releasesCan be run on Linux (KVM, Xen,VMware Player, VirtualBox)Windows(WMware Player, VirtualBox) Mac (Fusion, Parallels, VirtualBox) Release Notes http://cernvm.web.cern.ch/cernvm/index.cgi?page=ReleaseNotesHowTohttp://cernvm.web.cern.ch/cernvm/?page=HowTo

Appliance can be configured and used with ALICE, LHCb, ATLAS (and CMS) software frameworks

Content distribution networkCVMFSFile ServerCVMFSFile ServerCVMFSFile ServerHTTPProxy

More than a User Interface..BOINC Open-source software for volunteer computing and grid computingCernVM is now extended to include BOINC clientSummer student project in OpenLab based on LHC@HOME experience and CernVM image to run ATLAS simulation using BOINC infrastructurehttp://boinc.berkeley.edu/ Successfully run PanDA job in CernVMGlobus Virtual Workspace ServiceAn open source infrastructure for the deployment and management of virtual machinesGoogle Summer School (hosted at ANL) project to deploy a one-click, auto-configuring virtual Grid overlay for Alice/AliEnhttp://workspace.globus.org/Successfully created virtual AliEn site with one command

CernVM CoPilot0. Send host JDL (free disk space, free memory, available packages)1. Append framework specific information (e.g. CE name) and request a job3. Send input files and commands for execution(packages are already there) 5. Register output files4. When the job is done send back the output files (and the result of validation)2. Send user job JDL from Task QueeAliEn/DIRAC/PanDA

Mailing listscernvm-talk@cern.ch (Open list to discuss about design, user experience and related issues with the CernVM project)cernvm.support@cern.ch (End-user support for the CernVM project) Savannah PortalPlease submit bugs and feature requests to Savannah athttp://savannah.cern.ch/projects/cernvm

Web site(s) http://cernvm.cern.chhttp://rbuilder.cern.ch

Links

Extra slides

http://cernvm.cern.ch

http://rbuilder.cern.ch/project/cernvm-devel/

NOTE - initial settings are:

User Name: adminPassword: password

Please change Web admin password!

Select experiment for which VM will be configured

Status page

Select your top level group and migrate appliance configuration to get the latest version

Create local user account. First account created using this interface will be allowed to to executed sudo comand

Hands onpbuncic$ ssh -X pbuncic@192.168.197.129pbuncic@cernvm's password: Last login: Fri Jul 11 10:15:48 2008 from 192.168.197.1

[pbuncic@pb-d-128-141-224-235 ~]$ alienv listAliRoot/v4-13-Rev-02GEANT3/v1-9ROOT/v5-19-04

[pbuncic@cernvm ~]$ alienv use AliRoot/v4-13-Rev-02.---------------------------------------------------------------Great! Your compiler setup is ok!---------------------------------------------------------------******************************************************Attention: You don't have a valid grid proxy - doing grid-proxy-init******************************************************Your identity: /DC=ch/DC=cern/OU=.Enter GRID pass phrase for this identity:Creating proxy ....................................... DoneYour identity: pbuncicCreating token ..................................... DoneYour token is valid until: Sat Jul 12 10:18:52 2008

bash-3.00$ which aliroot/opt/alice/packages/AliRoot/v4-13-Rev-02/bin/tgt_linux/alirootbash-3.00$ aliroot -b ******************************************* * W E L C O M E to R O O T * * Version 5.19/04 14 May 2008 * * You are welcome to visit our Web site * * http://root.cern.ch * *******************************************ROOT 5.19/04 (tags/v5-19-04@23850, Jun 11 2008, 02:19:19 on linux)CINT/ROOT C/C++ Interpreter version 5.16.29, Jan 08, 2008Type ? for help. Commands must be C++ statements.Enclose multiple statements between { }.root [0]

Cache: ~200 MB

bash-3.00$ cernvm-cache -i

CONFIGURATION VARIABLES CERNVM_CONFIG_URL=http://cern.ch/cernvm/config CERNVM_ORGANISATION=ALICE CERNVM_PLATFORM=i686-pc-linux-gnu

CURRENT LOCATION Public Hostname: pb-d-128-141-224-235.cern.ch Domain: cern.ch Time zone: Europe/Zurich Location: Geneva

WEB FILE SYSTEM Server: cernvm-webfs.cern.ch Protocol: grow Cashe directory: /tmp/parrot.500 Cache size: 191 MB

*