CUC Oct 2005

Science Data Systems

Jonathan McDowell

SDS – Oct 2005

● CIAO Evolution Overview● PSFs● Chandra Source Catalog

Staffing

● Shrinking science team:

– Martin Elvis back to full time research (2004)

– Mike Wise leaving for Amsterdam (Dec 2005)

– Eric Schlegel to U.Tex/San Antonio (Aug 2005)

– Dan Harris to “retirement” (aka science) (Sep 05)

– No replacements

CIAO Releases

● CIAO 3.2.1. - Feb 05 bugfix

– acis_build_badpix header problems, support for

parameter block input● CIAO 3.2.2 – Jun 05 – Repro3 support

– Asphist fixes for multiple observations

– Mkacisrmf bugfixes and support for new CALDB

– New algorithm for HRC degap: will improve HETG

wavelengths

– Change to bad pixel handling in VFAINT mode

CIAO Schedule

● CIAO 3.3 – November (Tool release)

– New science tools and bug fixes

– reproject_image, reproject_aspect● CIAO 4.0 – 2006 (Infrastructure release)

– Sherpa, Chips: major functionality improvements

– Data Model improvements● CIAO script releases – TBD● SAOSAC release - TBD

CIAO 3.3

● reproject_image, reproject_aspect● DM ability to handle data cubes (filter on regions,

make slices, etc.)● Proposal tool support for Cycle 8● Fixes to region area bugs● Fix problem with FTOOLS/CIAO clash (renamed

cfitsio library)

CIAO Future

● Areas needing improvement for users:

– PSFs

– Merging observations

– Background modelling

– Easy fluxes and upper limits

– Publication quality plots

– Robust, fast and programmable spectral and spatial

fitting (improved Sherpa)

PSFs: User needs and our plans● User need:

– Ability to make custom PSFs● Plan: Distribute portable SAOSAC/MARX

combo● User need:

– Estimate PSF fraction easily● Plan: encircled-energy fraction tables in CALDB

and software to use them; some numerical issues at present

PSFs: ECF table status● Normalization issue in wings: tracking down

some discrepancies● HRC-only (flat plane) values: characterizing error

introduced wrt ACIS tilted-chip, appears significant

PSFs: SAOSAC status● SAOSAC (CXC Optics Group) consists of a

heterogeneous group of 40 programs and associated scripts: C,C++, Fortran, Perl, Lua. Designed as engineering tool

● Available to users via CHART web interface● Moving to portable version isolated from

references to local disks, new build system ● Modifying to handle numerical differences

between Sparc and x86 architecture

PSFs: SAOSAC and users● SAOSAC outputs ray files. Need to project to

focal plane and intersect with instrument● Will model instrument effects on PSF (but

should be optional, since PSU has its own detector model)

● Will use MARX for both projection and default instrument model (may replace/enhance instrument model later)

PSFs: SAOSAC/MARX driver script● We will write a driver script with a simple

interface. For a first release:

– Single point source only, no grating

– Specify spectrum (via PHA file or Sherpa model syntax?)

– Specify off-axis angle and azimuth

– Specify SIM position and fiducial correction

– Specify date? (for time-dependent instrument cal?)

– Specify instrument model (MARX or none)

Implementation Plan● CXC Optics to complete SAOSAC portable

infrastructure by December● Perform SDS testing and resolve Linux issues● Identify resources for driver script work● Documentation effort to guide users● Identify release schedule by end January● Release asynchronously with respect to CIAO?

Chandra Source Catalog● Chandra Level 3 Pipeline● Make a definitive catalog of sources useful for

individual and statistical studies● “Blind”, automatic processing capable of

handling different instrument configurations and different kinds of field (crowded, extended, etc.)

● Runs over full field (handle large off-axis PSFs)● Good for stars, galaxies, AGN, clusters, etc.. not

just optimized for one kind of source

Chandra Source Catalog● Not just a static catalog:● Access to extracted data products for each source

via the catalog● Dynamic interaction to produce user-configured

views with on-the-fly processing● Continuous updating as mission continues and

data becomes public

Chandra Source Catalog ● In early years of mission, steadily improved L1

and L2 calibrated products (event lists)● We ran detect and made source lists, but these

products were provided as a guide only.● Our understanding of the instruments is now at

the point where automatic processing can go further along the analysis chain – this was always part of our plan.

Chandra Source Catalog

● New standard data products: extracted spectra, ARF, RMF

and 4-band postage-stamp images for each source.

● New standard data product: high quality source list

● New standard data product: source properties table for each

source, with accurate positions and fluxes from 2D PSF

fitting as well as extent and variability flags

Chandra Source Catalog

● Energy bands for ACIS:

– Broad 0.2-7.5 keV

– Soft 0.2-0.5 keV

– Medium 0.5-2.0 keV

– Hard 2.0 – 7.5 keV

● Only one broad band for HRC

● Bands selected after comparison with those in use in the

community; compatibility with XMM

Chandra Source Pipeline - 1

● Reprocess event lists with latest cal, and perform high

background (flare) cleaning

● Make exposure maps in each band for false edge source

suppression and construction of sensitivity map

● Run multi-scale WAVDETECT in each band

– Scales 1,2,4,8,16

● Identify sources across scales and bands

● Define source and (annular) background regions for each

source

Overlapping sources

Chandra Source Pipeline - 2

● Make postage stamp fluxed image in each band

● Use SAOSAC to make PSF for source

● 2D gaussian (PSF-convolved) fits to image, attempt to

identify source as single, double or complex

● Derive source fluxes and position(s). Use power law spectral

fit with alpha=1 and galactic NH. (Free alpha if enough

counts, currently n=300).

● Correct with ECF (PSF fraction) tables

● Variability test (currently KS, soon Gregory-Loredo)

Near term enhancements

● Merge pipeline: combine source data from multiple

observations and instruments, get best fluxes and long-term

variability estimate

● Remove readout streaks (“out of time events”)

● Revise wavdetect parameters based on pilot studies and

characterize completeness, sensitivity, false source rate

● Complication: 2D fitting can discover extra sources, so

threshold is not simple to define.

Proposed enhancements

● PSF fraction correction (as fn. of energy) for sources. Plan to

use fraction within given contour, rather than PSU method of

selecting contour based on given fraction. (Will be useful

user tool). Method more appropriate from blind pipeline with

avoidance of nearby sources (interactive region tweaking not

possible in pipeline)

● Background maps

Longer term enhancements

● Handle very extended sources (SNRs, clusters) using

Voronoi tesselation detect

● Mosaic regions (M31, Orion..)

● Gratings data (some day...)

Reviews of other processing pipes● ACIS_EXTRACT reviewed by M. McCullogh, F. Primini for

functionality; further tests planned. We are adopting the

important approach of energy-dependent PSF fraction

corrections to the ARF.

● AE Currently uses mkpsf for PSF generation, which we don't

have confidence in.

● XASSIST (A. Ptak) also under review; has no PSF fitting yet

● XMM Catalog: Plan is to do simple celldetect with no

position refinement, and bright threshold (fewer sources)

● CHAMP: Used in initial evaluation; no HRC support.

Operations plan

● Initial evaluation based on 6 test sequences

● Run larger number of sequences for science testing over a

period of months, while merge pipeline in development

● Start production processing circa Fall '06

● Release initial public product when about 1/5 of mission has

been processed, catalog will evolve as more is processed.

● 1 year to process first 5 years

● Access to “live” evolving catalog and to fixed snapshots

User interface

● Web-based GUI interface

● Links to data, source and background events, images, spectra

and responses; searches on all columns

● VO-aware interface with ability to define on-the-fly catalogs

in spreadsheet-like way; iterative searches

● Later enhance to allow workflows (run queries which involve

processing on the actual data)