xmm data reduction: part ii sas command-line analysis and scripting andy read
TRANSCRIPT
XMM data reduction: part IISAS command-line analysis and scripting
Andy Read
Weds 14th Dec 2005 XMM data reduction: part II 2
Overview Command-line examples/help Why use command-line and scripts? Example scripts : SASmakecalev & SASprepareXMM Imaging scripts & spectral scripts Scripts within scripts Final remarks
Weds 14th Dec 2005 XMM data reduction: part II 3
Setting up the user environment To run on XROA system:> sas-setup-6.5.0 initialises software
> setenv SAS_ODF (path_to_ODF_directory)
e.g. /data/71/tro/xmm_data/0106860101/odf/
> setenv SAS_CCFPATH /data/rb2/sas_ccf
> cifbuild fullpath=yes >& cifbuild.log
builds ccf.cif – calibration index file
> setenv SAS_CCF (path_to_ccf.cif_file)
> odfingest odfdir=$SAS_ODF outdir=$SAS_ODF >& odfingest.log
builds ***SUM.SAS file in ODF directory – ODF summary file necessary for reprocessing
(from Tim’s talk)Command line examples
Weds 14th Dec 2005 XMM data reduction: part II 4
Command line help
- Simple help – parameters and default values
Weds 14th Dec 2005 XMM data reduction: part II 5
- More - Parameters, parameter types, default values and ‘one-line’ help
Weds 14th Dec 2005 XMM data reduction: part II 6
- Full help – Links to SAS web pages
- Cut and paste link into browser
Weds 14th Dec 2005 XMM data reduction: part II 7
Why use scripts?
Useful scripting cookbook at http://starlink.rl.ac.uk/star/docs/sc4.htx/sc4.html
- Scripts are just a list of command-line calls/commands run sequentially- There are many powerful command line calls
- Can run intensive analysis on many datasets, sources etc.- Can be sure of doing exactly the same thing every time, e.g. running
exactly the same procedures on many sources/datasets- No need for interactive use
- can run overnight (no need to be there)- can run as cron jobs (no need to be even logged in)
- Often need to redo entire analysis, e.g.- New calibration files- New ODF files
- Mistake, change in e.g. source positions, detection threshold etc.
Weds 14th Dec 2005 XMM data reduction: part II 8
Example script : SASmakecalev
Weds 14th Dec 2005 XMM data reduction: part II 9
Executable t-shell scriptComment lines
Runs at low priority
On-screen help – appears whenever incorrect number of arguments is given (e.g. zero)
Number of arguments
If correct number of arguments is given (7) then input arguments (1-7) converted to parameters used in script, and we move into the main body of the script
Weds 14th Dec 2005 XMM data reduction: part II 10
Check that ODF exists (otherwise exit)
Get SAS version (for directory naming)
Create (if necessary) output directory, and (if allowed) enter
Weds 14th Dec 2005 XMM data reduction: part II 11
Check ODF
Set SAS_ODF environment variable
Check for ***SUM.SAS file
(Check for ***SUM.ASC file)
Create ccf.cif – calibration index file Set SAS_CCF environment variable
Create ***SUM.SAS file in ODF directory – ODF summary file necessary for reprocessing
Output all SAS environment variables to screen – useful check
Weds 14th Dec 2005 XMM data reduction: part II 12
Run emproc (if required)
Run emchain (if required)
Run epproc (if required)
Run epchain (if required)
List created final calibrated event lists and exit
Weds 14th Dec 2005 XMM data reduction: part II 13
Run script
Weds 14th Dec 2005 XMM data reduction: part II 14
Example script : SASprepareXMM
Weds 14th Dec 2005 XMM data reduction: part II 15
Soft Proton flaring in an observation of a Galaxy Group
Weds 14th Dec 2005 XMM data reduction: part II 16
Executable t-shell scriptComment lines
Runs at low priority
On-screen help – appears whenever incorrect number of arguments is given (e.g. none)
Number of arguments
Weds 14th Dec 2005 XMM data reduction: part II 17
If correct number of arguments is given (9) then input arguments (1-9) converted to parameters used in script
If requested :
Check ODF
Check for ***SUM.SAS file
Create ***SUM.SAS file in ODF
Set SAS_ODF environment variable
Weds 14th Dec 2005 XMM data reduction: part II 18
If requested :
Create ccf.cif – calibration index file
Set SAS_CCF environment variable
Output all SAS environment variables to screen – useful check
Weds 14th Dec 2005 XMM data reduction: part II 19
Convert patcode parameter into an expression for evselect
Weds 14th Dec 2005 XMM data reduction: part II 20
MOS1 example (comment)
Weds 14th Dec 2005 XMM data reduction: part II 21
If $m1 does not equal 0, OK – continue to next line
If $m1 equals 0, ignore this whole section
Weds 14th Dec 2005 XMM data reduction: part II 22
If $m1 equals F, then we attempt to find the file
Else, the file name is given by the parameters
Weds 14th Dec 2005 XMM data reduction: part II 23
Remove a temporary file if it already exists
Weds 14th Dec 2005 XMM data reduction: part II 24
Long list all the MOS1 event files in the directory to temporary file
Weds 14th Dec 2005 XMM data reduction: part II 25
Count the number of entries (MOS1 event files) in the temporary file
Weds 14th Dec 2005 XMM data reduction: part II 26
If only one MOS1 event file, use this
Weds 14th Dec 2005 XMM data reduction: part II 27
If more than one MOS1 event file, try to find the largest
Weds 14th Dec 2005 XMM data reduction: part II 28
Initiate loop through ‘i’ (i=0)
Weds 14th Dec 2005 XMM data reduction: part II 29
loop through ‘i’ until ‘i’ equals number of event files, then end
Weds 14th Dec 2005 XMM data reduction: part II 30
Increment ‘i’ by +1 (i=1)
Weds 14th Dec 2005 XMM data reduction: part II 31
Get size of ‘i’th file :
Type the file, extract the top (head) ‘i’ lines, then extract the bottom (tail) line – file size is the 5th string in long listing
Weds 14th Dec 2005 XMM data reduction: part II 32
If size of ‘i’th file is larger than largest recorded, note ‘i’ (‘ibig’) and size as largest recorded
Weds 14th Dec 2005 XMM data reduction: part II 33
Loop round to next ‘i’
Weds 14th Dec 2005 XMM data reduction: part II 34
Get name of largest file (i.e. file corresponding to ‘ibig’) :
Type the file, extract the top (head) ‘ibig’ lines, then extract the bottom (tail) line – file name is the 9th string in long listing
Set $m1 to this filename
Weds 14th Dec 2005 XMM data reduction: part II 35
Create (evselect) lightcurve :
High-energy, single events, ‘good’ flags, 100s bins
Weds 14th Dec 2005 XMM data reduction: part II 36
Output help to screen: What is being done, what the user should do
Weds 14th Dec 2005 XMM data reduction: part II 37
Plot (dsplot) the light curve to the screen for user to examine
Weds 14th Dec 2005 XMM data reduction: part II 38
Request upper and lower threshold values from user
Weds 14th Dec 2005 XMM data reduction: part II 39
Use threshold values to create (tabgtigen) good times GTI file
Weds 14th Dec 2005 XMM data reduction: part II 40
Create (evselect) intermediate event file : filter on low-E threshold, pattern, flags, etc.
Weds 14th Dec 2005 XMM data reduction: part II 41
Create (evselect) final event file : filter intermediate file for good times using GTI file
Weds 14th Dec 2005 XMM data reduction: part II 42
Same for MOS2
Weds 14th Dec 2005 XMM data reduction: part II 43
Same for pn
Weds 14th Dec 2005 XMM data reduction: part II 44
Weds 14th Dec 2005 XMM data reduction: part II 45
Enter MOS1 thresholds
Same for MOS2
Same for pn
New filtered event files (r1***EV***)
smaller than originals
Weds 14th Dec 2005 XMM data reduction: part II 46
Already seen xmmselect
GUI - clean data, and produce science products
Only handles one dataset at a time
Essentially a handy GUI ‘wrapper’ to the SAS-task ‘evselect’
evselect is the command-line workhorse of SAS analysis tasks
Many scripts are command-line ‘wrappers’ of evselect and other SAS-tasks
Weds 14th Dec 2005 XMM data reduction: part II 47
Script to create images
- Essentially an evselect wrapper, plus other tasks (e.g. asmooth)
Weds 14th Dec 2005 XMM data reduction: part II 48
MOS1
MOS1 smoothed
MOS2
MOS2 smoothed
pn
pn smoothed
e.g. 300-2000eV sky (X/Y) 2′′ images (raw and smoothed) [MOS-sdtq, pn-sd]
Weds 14th Dec 2005 XMM data reduction: part II 49
Script to create spectra + ARF, RMF and do grouping etc.
Again, evselect wrapper, plus arfgen, rmfgen, grppha etc…
SAS-task equivalent especget now much improved
Weds 14th Dec 2005 XMM data reduction: part II 50
createspectrum to create BG spectrum
Note: can set parameters (here, co-ordinates from Tim’s region files) on the command lineRun script (using parameters)
Weds 14th Dec 2005 XMM data reduction: part II 51
createspectrum to create source spectrum, ARF, RMF and group final spectrum
Weds 14th Dec 2005 XMM data reduction: part II 52
Weds 14th Dec 2005 XMM data reduction: part II 53
Can put each step/script into a file (e.g. analysis) – A script to call other scripts
Can then run whole ‘analysis’:
> source analysis
Weds 14th Dec 2005 XMM data reduction: part II 54
Homework (2005) Extract NGC 1313 data (Oct 2000) from XSA Set up analysis environment Filter data for background flares Extract 0.3 – 10 keV images for all three
detectors Extract spectra of NGC 1313 X-1 Compare your results to
/data/71/tro/xmm_data/0106860101/work/
Homework (2006) – use scripts (and write a script of your scripts) to do all the analysis
Weds 14th Dec 2005 XMM data reduction: part II 55
Resources This talk (and others):
http://www.star.le.ac.uk/~sav2/stats/ The scripts (and others):
http://www.star.le.ac.uk/~amr30/Scripts/ XMM-Newton SAS web pages
Via http://xmm.vilspa.esa.es Particularly useful documentation:
HEASARC ABC guide SAS user’s guide
Experienced users!