boss development environment boss--besiii offline software system ma qiumei 2006.03.01
TRANSCRIPT
BOSS Development Environment BOSS--BESIII Offline Software System
Ma Qiumei
2006.03.01
2
Outline
Simple introduction of BOSS How to use CMT in BOSS Simple introduction of CVS Common questions and solutions
3
History of BesIII Offline Software System Besf/v1 : The original version completely based on belle Besf/v2 : The version improving with other frameworks like babar, gaudi, etc. Offline/3.0.0 : The version with the new building tools—CMT Boss/1.0.0 : The testing version based on Gaudi Boss/2.0.0 : The modified version based on Gaudi Boss/2.1.0 、 2.2.0 、 2.3.0 、 3.0.0 、 4.0.0 、 4.1.0 、 4.2.
0 5.0.0 、 5.1.0 、 6.0.0
4
Boss development environment(Now)
Machine: lxplus.ihep.ac.cn OS : Linux-7.3 Compiler : gcc-3.2 Framework :Based on GAUDI-v12r0 External Libs: CERNLIB 、 ROOT 、 CLHEP 、 Geant4 、 and etc. Developing language : C++ , some FORTRAN Database : PostgreSql,MySql Version management tool : CVS Browsing files in CVS repository : ViewCVS Configuration management tool : CMT
5
Boss development environment(future)
Machine: lxplus.ihep.ac.cn OS : Scientific Linux Cern (SLC) Compiler : gcc-3.2.3 Framework :Based on GAUDI_v16r4 External lib: lcg/external; geant4 and etc. Developing language : C++ , some FORTRAN Database : PostgreSql,MySql Version management tool : CVS Browsing files in CVS repository : ViewCVS Configuration management tool : CMT
6
BOSS Structure
Analysis -- Analysis software and some analysis toolsBesExamples -- Examples for using some packagesBesPolicy -- define some standard patterns of Bes BesCxxPolicy -- define some standard patterns only for c++ compilier BesFortranPolicy -- define some standard patterns only for fortran compilier BesRelease -- for all the packages release BesROOT -- convert data from ASCII to DSTCalibration -- for calibration framework and common too
lsControl -- Common tools and framework of bossDatabase -- the geometry data IO
7
DetectorDescription -- Detector Description packages Emc -- for emcEvent -- different event modelEventDisplay -- the interface for event displayEventFilter -- External -- interfaces of external libsGenerator -- for generators MagneticField --Mdc -- for mdc
8
Muc -- for MuonCounterPhyTool -- some tools for physics analysis Reconstruction -- all reconsrtuction algorithmsRootPolicy -- Simulation -- simulationTestRelease -- for some test releaseTof -- tofUtilities -- some common tools More details please see the website: http://koala.ihep.ac.cn/cgi-bin/viewcvs.cgi/BossC
vs
9
How to use CMT in BOSS
Simple introduction of CMT
Setting up your CMT environment Working on a package
Working with boss
10
What is CMT ( Configuration management tool )
A set of tools and conventions structures software development or
production concepts of areas, packages, versions, constituents
organises software into packages describes package properties describes package constituents operates the software production
(management, build, import/export, etc...) by transparently configuring and driving the various
conventional tools (CVS, make, MSDev, Web, tar, compilers, linkers, archivers, etc...)
11
The goals
Organise software development from a single person up to teams in large projects (scalability)
Organise relationships between entire software bases or between simple packages (scalability)
Manage site, platform or product specific properties
Automate complex production tools for non software experts (simplicity)
Factor out know-how from project to project, from package to package or from team to team (process improvement)
12
Who are using CMT
The physics research experiments using CMT as the following:
a.. Virgo b.. LHCb c.. GLAST d.. Atlas
e.. Nemo f.. Opera g.. Planck h.. Auger i.. GANIL j.. GAM
13
Types of Package
Packages may have particular semantics
Normal packages provide software or documents
Container packages manage other packages
Policy packages hold bes specific conventions, policies, patterns
Interface convert conventions and policies of external packages or libs
14
Structure of a package /src source files (*.cxx) /<package-name> header files (*.h) /cmt requirements file and some scripts /share jobOptions and other running scripts /test some app’s source files /i386_linux24 Automatically created at build time
(libs. Applications and etc. under this
directory)
/ChangeLog a log file
15
Global software organization
use
use useuse
use
use
use use
Release area
User area MDCGeomSvcMDCGeomSvc-01-00-01
BesReleaseBesRelease-00-00-01
MDCGeomSvcMDCGeomSvc-00-00-01
BesGeoMdc BesGeoMdc-00-*
External/CLHEPCLHEP-00-01-*
BesPolicyBesPolicy-01-*
CLHEP
External area
use
16
几个概念 Release (发布;软件版本):某个 releas
e 对应于所有软件包的某一个固定版本,它是稳定的,一旦发布基本上就不再更改。
nightly_build :不需要发布一个新的 release ,但是用户希望使用某些软件包的新改变,就建立了这么一个 release : nightly_build 。这个 release 是不稳定的,经常更新。
TestRelease :运行 boss 的接口程序,可以选择需要运行的程序。
17
Setting up your CMT environment
首先,连接 CMT $mkdir cmthome $cp /afs/ihep.ac.cn/bes3/offline/template/cmthome/setupCMT.csh ~/cmthome 注:如果你使用的是 bash ,则 copy setupCMT.sh 到你的 cmthome 目录下。
然后,修改 login requirements ,设置符合自己要求的配置 $cp /afs/ihep.ac.cn/bes3/offline/template/cmthome/requirements ~/cmthome 编辑 requirements 文件,使其符合你的要求 $vi requirements注:需要修改的可能有两个地方: set BES_RELEASE 5.0.0 macro WorkArea "/ihepbatch/bes/maqm/workarea"
18
最后,设置环境完成 source setupCMT.csh cmt config source setup.csh
注:最好把以下语句放到 .tcshrc 文件中: source /ihepbatch/bes/maqm/cmthome/setupCMT.csh source /ihepbatch/bes/maqm/cmthome/setupCVS.csh source /ihepbatch/bes/maqm/cmthome/setup.csh source /ihepbatch/bes/maqm/workarea/TestRelease/TestReleas
e-00-00-14/cmt/setup.csh
19
Working on a package Basic scenarios
Creating a package Checking out a package alone Modifying the package
More: working on several packages Apply same set of commands via cmt
broadcast
20
Creating a new package
1 、 Choosing a location i.e. Which container? … e.g.
Database2 、 Choosing a name
Must be a unique package name in boss (should not either just differentiate by case of letters)
3 、 Choosing a version4 、 The command cmt create BesGeoDB BesGeoDB-00-00-01
21
Examples… The simplest one
One new application with (some) source file(s)
>cmt create A A-00-00-01>cd A/A-00-00-01/cmt>vi requirements>vi A.cxx >gmake>source setup.csh>cd ../i386-linux24>./A.exe
package Ause BesPolicy BesPolicy-01-*application A A.cxx
1) Create a new packageand move to it
2) Describe the package
3) Create the package items
and build them
4) set the environment
and run
22
The requirements file
The requirements file
The requirements file centralizes all configuration parametersAuthor(s),
manager(s)
Structural information •specialized directory structure •used packages•links to external packages
Constituents•Libraries•Applications•generated documents
Make macros and environment variables
and their possible values on various platforms, sites, environments Customization for
new languages, or document generators
Definition of conventional behavioural
patterns
Sufficient for transparently generating all required makefiles and setup scripts, or for
launching test procedures. They are the sources of all queries.
23
Example of requirements
package BesGeoDBauthor maqiumei [email protected] version BesGeoDB-00-00-01branches cmt src BesGeoDB use BesPolicy BesPolicy-*use PGSQL PGSQL-* External library BesGeoDB *.cxx
apply_pattern package_linkopts
24
Checking out a package Getting a package out from CVS to modify it
Go to your private development area Checkout the package (installing the proper
CMT directory structure)cmt co TestRelease
Get the HEAD revision, and make use of the most recent tag
cmt co –r TestRelease-00-00-15 TestRelease Get the required revision, and make use of that tag
The directory version is created with the appropriate CVS tag applied to the package
25
Modifying and building a package
The basic development cycle Modify
> vi *** Build
> source setup.csh
> gmake Test (if available)
26
Working on several packages
Checking out a complete containercmt co Databasecmt co –requirements Database/*/cmt/requirements
Checking out all packages cmt co BesRelease cmt co –requirements
BesRelease/*/cmt/requirements
Applying a command to all checked out packages
cmt broadcast <shell command> Cmt broadcast cmt config Cmt broadcast gmake
27
Working with boss The principle of the TestRelease package
This package is the basic interface to run a job Running boss only requires to “select” a set of
already built components and to locally install a set of runtime files Checking out TestRelease
Customizing it : selecting components Before running : install runtime files
This generally creates symlinks towards runtime files (shared libraries, jobOptions, data files, …)
Running boss
28
Running boss #running a simple example -- ExHelloWorld cd <yourworkarea> cmt co TestRelease cd Testrelease/*/cmt vi requirements Add “use ExHelloWorld ExHelloWorld-* BesExamples” to requiremen
ts cmt broadcast cmt config cmt broadcast gmake source setup.csh cd ../run boss.exe HelloWorldOptions.txt boss.exe jobOptions_rec.txt
29
Simple introduction of jobOptions
////####################################// Boss job options file for MdcRec//======================================
ApplicationMgr.DLLs += {"MdcTrkRecon"}; ApplicationMgr.TopAlg += { "MdcTrkRecon" }; ApplicationMgr.ExtSvc += { "MdcGeomSvc" }; ApplicationMgr.DLLs += { "RootHistCnv" };
//--------------------------------------------------------------// Set output level threshold (2=DEBUG, 3=INFO, 4=WARNING, 5=ERROR, 6=FATAL )// MessageSvc.OutputLevel = 2;
30
// Number of events to be processed (default is 10)ApplicationMgr.EvtMax = 200;
EventSelector.InputFiles ="AsciiDmp_out.dat";
ApplicationMgr.HistogramPersistency = "ROOT";
NTupleSvc.Output = { "FILE101 DATAFILE='Mdc.root' OPT='NEW' TYP='ROOT'“};
//===============================// End of job options file// ===============================
31
Running the reconstruction algorithms
#include "Ascii.txt“ #include "PartPropSvc.txt" //zhangy #include "$MDCTRKRECONROOT/share/jobOptions_MdcPatRec.txt“ //zangsl //#include "$TRKRECOROOT/share/jobOptions_TrkReco.txt" #include "$MDCDEDXALGROOT/share/job_dedx_all.txt" #include "$EMCRECROOT/share/EmcRecOptions.txt" #include "$TRKEXTALGROOT/share/TrkExtAlgOption.txt" #include "$TOFRECROOT/share/jobOptions_TofRec.txt" #include "$MUCRECALGROOT/share/jobOptions_MucRecTrkExt.txt" #include "$ROOTIOROOT/share/jobOptions_Save2Root.txt"
32
// Set output level threshold (2=DEBUG, 3=INFO, 4=WARNING, 5=ERROR, 6=FATAL ) MessageSvc.OutputLevel = 5; // Number of events to be processed (default is 10)ApplicationMgr.EvtMax = 200;EventSelector.InputFiles="AsciiDmp_out.dat";ApplicationMgr.HistogramPersistency = "ROOT";RootCnvSvc.digiRootOutputFile ="digi.root";
NTupleSvc.Output = { "FILE101 DATAFILE='Mdc.root' OPT='NEW' TYP='ROOT'", "FILE104 DATAFILE='kalfit.root' OPT='NEW' TYP='ROOT'", "FILE103 DATAFILE='DeDx.root' OPT='NEW' TYP='ROOT'", "FILE201 DATAFILE='Tof.root' OPT='NEW' TYP='ROOT'", "FILE301 DATAFILE='Emc.root' OPT='NEW' TYP='ROOT'", "FILE401 DATAFILE='Muc.root' OPT='NEW' TYP='ROOT'" };
33
CVS 是什么 ? 1989 年 4 月, Brian Berliner 设计了 CVS 并
编写了代码,之后 Jeff Polk 帮助 Brian 设计了 CVS 模块发行分支。
CVS(Concurrent Versions System ) 版本控制系统是一种 GNU 软件包,主要用于在多人开发环境下的源码的维护。 它可以记录源文件开发的历史。
34
CVS and CMT
The BESIII Offline Software sources are stored in CVS and can be accessed using the CVS command.
The BESIII Offline Software libraries have been built using the CMT. Therefore, using the CMT tool is the recommended way to modify existing packages or re-build the examples included in the release.
35
如何使用 CVS ? 首先,你需要申请一个 CVS 账号 然后,需要设置变量: export CVSROOT=':pserver:[email protected]:/bes/bes'
或 setenv CVSROOT ‘:pserver:[email protected]:/bes/bes
‘ 最后,你需要知道一些常用的 CVS 命令(见下
页)
36
常用的 CVS 命令 (1) Checking in to CVS cvs import –m “**” test Bes test-00-00-01 查看源程序的变化 cvs –n update 添加一个源文件 cvs add 源文件名(路径要正确) 删除一个源文件 cvs remove 源文件名(路径要正确)
37
常用的 CVS 命令 (2) 提交所作的修改 cvs commit –m “**” 给一个包做 tag cvs tag 版本号 查看当前工作的源文件与 cvs库里源文件的区别 cvs diff
38
Viewcvs简介 ViewCVS 是一个使用 Python书写的查看 CVS 代
码库中的 所有数据 的软件。它已经被很多的配置管理系统所使用, 也是现在开放源代码产品中与 cvsweb 一样可以选择的 CVS辅助工具。
ViewCVS架设了一个友好的 WEB界面,通过WEB界面可以非常直观的浏览代码 , 查看图形化的修改记录。
http://koala.ihep.ac.cn/cgi-bin/viewcvs.cgi/BossCvs/
39
常见问题及解决方法 编译的时候:
$ cmt config > Warning : package CERNLIB CERNLIB External not
found (requested by tester) $ gmake >ihepbatch/bes/maqm/workarea/Calibration/CalibDat
a/CalibData-00-00-04/CalibData/CalibTime.h: 14:34: facilities/Timestamp.h: No such file or directory
>/usr/bin/ld: cannot find -lxerces-c
40
运行的时候: boss.exe jobOptions_sim.txt
libGaudiSvc.so: cannot open shared object file: No such file or directory
boss.exe jobOptions_sim.txt
undefined symbol: ****
41
Documents
http://www.cmtsite.org http://atlas.web.cern.ch
/Atlas/Groups/software/oo/tools/cmt http://boss.ihep.ac.cn http://koala.ihep.ac.cn/cgi-bin/viewc
vs.cgi/BossCvs/
42
Thank you!