National Radio Astronomy ObservatoryMay 10, 2006 – EVLA Advisory Committee Meeting

Post-processing Overview

Bryan Butler (& Joe McMullin)(NRAO-AOC)

National Radio Astronomy Observatory May 10, 2006

Science Software Group

Mission• Primary purpose is to provide

facilities for post-observational scientific reduction of ALMA/EVLA data

• Requirement documents: • ALMA Offline Data Processing

Requirements• AIPS++ Audit• EVLA Data Post-Processing

Software requirements• Uses the legacy AIPS++ code

base for development• AIPS used for validation,

algorithm development (e.g., automated flagging)

Requirement Key Areas

• General Requirements and Interaction

• Interface• Data Handling• Calibration and Editing• Imaging• Data Analysis• Visualization• Special Features

• Spreadsheet/tracking on the combined projects’ requirements (common to both projects and the deltas between projects)

National Radio Astronomy Observatory May 10, 2006

Organization

Management• IPT Organization

• G. van Moorsel ((e)VLA/VLBA)• B. Glendenning (ALMA), • Michael Rupen (EVLA) & Debra

Shepherd (ALMA) – Project Scientists for Computing

• J. McMullin, K. Golap (Project Management)

• S. Myers (SSG Project Scientist)• Priorities (ALMA/EVLA)

• Scientific Completeness/Correctness, Usability

• Robustness, Performance

Resources

• NRAO: 6.0 FTEs (including management; split 50/50 with ALMA)

• ALMA: 5.5 FTEs (Application development, Data Model, Data Model Interfaces, Data Capture process)

• NSF grant: 1.0 FTE (visualization; concludes 2007.8)

National Radio Astronomy Observatory May 10, 2006

Resources: DetailsFTEs Comments

NRAO (SSG) 4.75 [5.75]

Project Scientist: Steve Myers

Sanjay Bhatnagar 0.75 Astronomer Algorithm Development (EVLA)

Kumar Golap 0.75 Astronomer Application Code Application code is common to both ALMA&EVLA

Joe McMullin 0.75 Astronomer Management

George Moellenbrock 0.75 Astronomer Application Code

David DeBonis 1.0 Software Engineer System Began 2004-05; depart:2006-01

Darrell Scheibel 1.0 Software Engineer System

Wes Young 0.75 Software Engineer System

ALMA 4.3 [5.3] 2.3 FTE in-kind

Gary Li 1.0 Software Engineer Application Code 0.5 FTE in-kind; Began 2004-03

Raymond Rusk 1.0 Astronomer Application Code

Tak Tsutsumi 1.0 Astronomer Application Code 1.0 FTE in-kind; Began 2005-03 (ACA)

Honglin Ye 0.5 Software Engineer Application Code Began 2005-07 (visualization)

Michel Caillat 0.8 Software Engineer ALMA Data model, simulation 0.3 FTE in-kind

Francois Viallefond 0.5 Astronomer ALMA Data model, simulation 0.5 FTE in-kind

Heiko Hafok 0.5 Astronomer ALMA Data Capture Began 2003-12

NSF grant 1.0

David King 1.0 Software Engineer visualization (concludes 2006-05)

Other 0.5

Urvashi Rao 0.5 Astronomer (student) Application code/Algorithm Dev (EVLA)

Began 2004-09 (9 month appt)

Total 10.05 [12.55]

National Radio Astronomy Observatory May 10, 2006

SSG Development

Status

• Multi-year plan for development of all Priority 1 ALMA science software requirements (2007.5)

• Project Office Page for development planning, accounting, transparency

• http://projectoffice.aips2.nrao.edu • 18 two-month development cycles

completed • Established strong feedback loop

with internal NRAO scientists• Established external scientist

testing/verification • Infrastructure

• RPM distribution• Framework (slide)

Milestones• Yearly NRAO VC and UC meetings• ASAC, EVLA ACM meetings

• ALMA CDR1 (Jun 03) - Passed• ALMA TST1 (Jan 04) – Passed

• Single Field Interferometry• ALMA CDR2 (Jul 04) - Passed• ALMA R2 (Oct 04) - Complete• ALMA TST2 (Nov 04) – Passed

• +Mosaics• ALMA TST3 (Apr 05) – Passed

• +synthesis/single dish combo• ALMA R2.1 (Apr 05) - Complete• EVLA TST1 (Jun 05) – Passed

• +Wide Field Imaging• ALMA CDR3 (Jul 05) - Passed• ALMA R3 (Oct 05) - Complete

• +ATF simulation (1st CASA test)• ALMA R3.1 (Mar 06) - Complete

• +User Interface Charette• Provided testing/detailed requirements

National Radio Astronomy Observatory May 10, 2006

Long-term Schedule (Presented at CDR2 in July 2004)

• 2003.5• Long-term plan

• 2004• Science: Imaging performance enhancements• Science: Flexible continuum subtraction• Framework: CVS code management• Data Interfaces: Draft AEDF document

• 2004.5• Science: SF reduction (TST1)• Science: Calibrater performance enhancements• Science: Calibrater flexibility (interpolations, transfer,

etc).• Framework: Framework requirements, plan• Data Interfaces: AEDF submitted for review, data

capture design• 2005

• Science: Mosaic reduction (TST2)• Science: Automated, interactive data editing• Framework: Library re-organization• Data Interfaces: Data Capture process

• 2005.5• Science: Single dish/interferometric (TST3)• Science: Simulator (telescope modeler, sampler,

visibility simulator)• Framework: Beta, subset, distribution with ACS• Data Interfaces: DRP filler

• 2006• Science: Complex mosaics (TST4) – changed to

ATF (single-baseline) test• Science: Full polarized primary beam imaging• Framework: Initial CLI, Python interface – added

a working group evaluation of the interface• 2006.5

• Science: Single Dish (TST5)• Science: Flexible line fitting• Framework: Testing with new framework – 1st

done in first half 2006• 2007

• Science: Simulations I (TST6)• Science: Multi-frequency synthesis• Framework: GUI development

National Radio Astronomy Observatory May 10, 2006

Coarse Timeline & Staging of Community Usage/User Support of CASA2006

• NRAO: NAUG testing (AIPS++)• Community: Project tests• ALMA: external science testing:

+single-baseline commissioning test) (CASA)

• ALMA: external science testing: +single dish reduction (CASA)

• ALMA: commissioning support (CASA)

• ALMA/EVLA: user interface review (CASA)

• EVLA: external science testing: +full polarization imaging; antenna pointing calibration

• 2006.5 AIPS++ frozen• ALMA: Pipeline Heuristics Use of

CASA2007

• NRAO: User support (CASA)• ALMA: commissiong support

(CASA)

• EVLA: external science testing: + RFI/automated flagging (CASA)

• ALMA: P1 SSRs complete2008

• EVLA: external science testing: + wide band calibration imaging (CASA)

• NRAO: user support (CASA)• ALMA: commissioning support• EVLA: P1 SSRs complete

2009-2010 • ALMA/EVLA: commissioning

support• Community: CASA

released/distributed for early ALMA/EVLA science

2011 • Community: CASA

released/distributed; full user support (ALMA/EVLA: P1/P2s)

National Radio Astronomy Observatory May 10, 2006

Framework Migration

• AIPS++• Glish interface; unknown,

unsupported outside of NRAO• Tasking system based on Glish• GUI system based on Glish/Tk;

limited widgets, not robust!• Difficult for external developers

to contribute• Multi-CD binary distribution• Large monolithic libraries with

cross dependencies• No namespace

• Freeze 2nd half of 2006

• CASA• Python interface (community

standard);IPython • Binding to Python, ACS; other

frameworks readily possible• Hierarchical set of small

libraries with clearly defined dependencies

• Namespace protection for integration with other code

• RPM distribution mechanism; auto-updates possible

• Robust• Inherits all application code

improvements in robustness and performance.

• Smaller memory footprint/startup time

National Radio Astronomy Observatory May 10, 2006

CASA usage status:

• Has been tested internally by NRAO scientists in preparation for the 1st ALMA test.

• Early demos provided bi-monthly to the NAUG at:• http://casa.nrao.edu/gettingstarted.shtml

• 6 scientists used and reported on it – enabled deployment for the ALMA test.

• Was deployed and reviewed by four ALMA testers• http://projectoffice.aips2.nrao.edu/ALMA2006.01/ALMA2006.01.html

• The user interface was reviewed and commented on by 8 NRAO scientists

• User Interface is being refined/further developed based on the user interface report:

• http://projectoffice.aips2.nrao.edu/uiwg_report.pdf• Will be used exclusively by the ALMA Pipeline Heuristics Team (second

half of 2006)• CASA will have replaced AIPS++ within NRAO (developers and NAUG

testers) this year.

National Radio Astronomy Observatory May 10, 2006

User Interface

• User Interface review• IPython interface to functionality, prototype parameter

setting interface and in-line help are in the right direction but:• Full needs are documented at:

• http://projectoffice.aips2.nrao.edu/uiwg_report.pdf

• Revised interface prototype (similar to IRAF epar environment); high priority

• Much work needed to provide astronomer-level documentation (collaboration with NAUG to develop this); high priority

• Development progress will be reviewed with the Fall EVLA test.

National Radio Astronomy Observatory May 10, 2006

Risks • Staffing

• Mitigation strategy: efficient use of existing resources; leverage off of ALMA development as much as sensible

• User Support• Mitigation strategy: Work with group of NAUG scientists such that they can

provide the front-line help.• User Interface

• Mitigation strategy: Cyclic feedback on progress and prototypes (revisited in June-July 2006; will be a review aspect of every formal/informal test)

• Requirement rot• Mitigation strategy: Update of EVLA requirements document to reflect current

planning/time tables.

• ASDM• Mitigation strategy: Multiple evaluations, early testing of needed modifications.

• Algorithm development/data rates – SB talk

National Radio Astronomy Observatory May 10, 2006

Overall Risk Mitigation

• Backup plan: AIPS• Requires:

• Continued support and targeted development• Support current VLA/VLBA users• Develop and deploy automated flagging algorithms to

community for testing/review