evla software - overall architecture, science & online domains bryan butler nrao

EVLA Software - Overall Architecture,

Science & Online Domains

Bryan ButlerNRAO

2006May08 EVLA Advisory Committee Meeting

2

Telescope Data Model

Export Data Format

Science Data Model

Feedback to telescope

Proposal SubmissionAnd Handling

Observation Preparation

EVLA VLBA ALMA GBT

EVLASched

EVLAControl

ALMASched

ALMAControl

GBTSched

GBTControl

DataCapture

Archive

Telcal

Offline VO

ObserverDomain

Mostly Telescope-Independent

Common Software

VLBASched

VLBAControl

Quick Look

Pipeline

GBT Postproc

TelescopeDomain

ScienceDomain

Mostly Telescope-Specific

Project Software

Mostly Telescope-Independent

Common Software

NRAO-wide Design


3

NRAO-wide Design

The EVLA software system needs to conform to the NRAO-wide design to the extent that it has been developed. Unfortunately, the NRAO-wide effort has languished for the past two years (this is changing - a new “e2e Project Manager” [Nicole Radziwill] and “e2e Project Scientist” [Ed Fomalont] have been hired in CV).


4

Domains & ModelsThe NRAO-wide design introduced the concept of “domains” - different (large) pieces of the system that can be grouped sensibly. It presented three such domains:• Observer• Telescope• ScienceIt also presented a number of “models” - descriptions of different (smaller) pieces of the system:• Observatory• Project• Observing• Archive• Telescope Data• Science Data


5

EVLA Overall Design

In the spring of 2004, the EVLA software group undertook an effort to develop and document a high level design (led by Morgan, Ryan, Sowinski, and Waters). This culminated in a completed design, which was reviewed by the NRAO eOC in June 2004. The design presented in the next few slides is based mostly on that effort, with extension and modification in the past two years.


6

EVLA Software Requirements

The software design and implementation is driven by a number of requirements documents:• e2e Science Software Requirements• Engineering Software Requirements• Real-time System Software Requirements• Operations Software Requirements

These do not have everything in them (for instance Proposal Handling and User Database, which are covered in separate [less “requirementy”] documents), but are fairly complete, and notably, the e2e requirements document has extensive requirements for PST and OPT.


7

The main flow of information (and processes; the “workflow” or “dataflow”) is:

Major Elements (“Models”)

Proposal

Project(s)

Program(s) Schedule(s)

Commands

A Scheduling Block is an atomic unit of observing. It is made up of a sequence of scans; a scan is made up of source(s), resource(s) (hardware definition - both FE and BE), timing information, and a “mode”. The mode defines the subscan(s), which are comprised of a single source, resource, and timing information.

Data


8

EVLA High Level Architecture

DATAFLOW


9

EVLA High Level Architecture

Note that most of the major subsystems have a direct counterpart in current VLA software - we have a significant amount of experience in what is needed there. What has been lacking is the electronic storage and passage of information between subsystems, and therefore the ability to do much of this automatically.


10

ObservationPreparationTool (OPT)

Ast

rono

mer

or

Staf

f

Project EVLA ObservingHeuristics

Program Block(Set of Scheduling Blocks for one Program)

Proposal SubmissionTool (PST)

To Observation Scheduling Tool

EVLA Design Detail, Pre-Observing Science Domain

Portal

Proposal HandlingTool (PHT)

Proposal

Au

then

tica

ted

Ast

rono

mer

or

Staf

f


11

EVLA Design Detail, Pre-Observing Online Domain

ObservationSchedulingTool (OST)

Executor

Next SBExecutionState

Equipment State

Metadata to DCAF

Operator

Environment

From OPT

Results from TelCal

Sequence of ConfigurationsAntenna Delays

Archive

Archive

Operator

Heuristics

Metadata to DCAF

To AMCS& CMCS

From AMCS& CMCS


12

EVLA Design Detail, Real-Time Domain

Hardware M&C

AMCS

CMCS

RF EVLA Antennas

FOTS Receiver

Station, Baseline Boards Lag Frames

CBE

State Counts

Raw VisEquipment State,

Data Addressing Info, Messages, Alerts, etc.

From Executor

FF

To Archive & TelCal

To DCAFTo DCAF


13

EVLA Design Detail, Post-Observing Online Domain

SDM

Data Capture AndFormat (DCAF)

From CMCS

TelCal

SDM

From AMCS& CMCS

To ExecutorAnd Archive

To Archive

Quick LookPipeline

(QLP)

Astronomer orOperator

ObservationStatus Monitor

Tool (OSMT)

M&CArchive

Portal

AuthenticatedAstronomer or

OperatorM&CArchive

To Archive (?)

TelCalResults


14

From DCAF

Post-Processing

ImageCubes

VO Astronomer

Default ImagePipeline (DIP)

Cubes (?)

From CMCS

EVLA Design Detail, Post-Observing Science Domain

Archive

Archive Search andRetrieval Tool

(ASRT)

Astronomer

Portal

AuthenticatedAstronomerReprocessed

ProprietaryProducts

Existing ProprietaryProducts

OpenProducts

OpenProducts

Trigger


15

Detailed Subsystem Designs & Prototypes

For most of the major subsystems, we either have a very advanced prototype (Portal, PST, Executor, OSMT), an early prototype (PHT, OPT, OST, ASRT, TelCal), or at least a much more detailed design (DCAF). The areas where we have done little work and in fact have only preliminary (high level) designs are for the pipelines (QLP, DIP).


16

Portal, User Database, Authentication

We know that we need some way to restrict access to the various tools, and the information available within them. We do this with a “portal”, through which users access the various tools. It authenticates users, and generates a unique token which is then used to verify a user’s login status. We have our own implementation of this, but recognize that we may need to migrate to the VO method (or have a translation layer).


17



18


Users enter their own User Database information, except:

• Institution information - they can only select an institution from a pre-set list (we want to use this to automatically generate reports to the NSF, which require precise information on institutions) - if the institute is not there, they indicate so, and we (well, I) add it.

• We allow so-called “3rd party” user registrations during proposal preparation and submission, adding significant complexity (a sore spot with us, but demanded by operations).


19

Proposal Submission Tool (PST)

• Mainly a tool to collect form data (web browser)

• Mostly telescope independent, with “resources” the exception

• Used to enforce telescope “policy”• Coupled to other EVLA tools only

loosely, via Portal and databases.• Currently also supports GBT.


20

PST - Main ProcessesMain Processes:• Model - retrieve and write data to database• Controller - business logic to map user

input (from browser) into objects which are then written to database

• View - the look-and-feel of the interface (done in browser)

• Validation of various fields - an important and significant part of the tool

• Help system


21

PST - ModelThe Model drives everything, so what’s in there?• science information - title, category, “mode”,

abstract, scientific justification, and some misc. info.• Authors, including which is the PI and “contact

author”• Sources• Resources (telescope hardware setup)• “Sessions” (a guide to SB setup)• Student SupportEssentially, everything that is necessary to:• Referee the proposal• Assign telescope time (and money)• Automatically generate SBs (mostly for novice users,

but experienced users will use this too!)


22

PST - “The View”

Although the logic is driven by the model, most of the programming complexity here is in the view. How do we do this? 6 main “tabs” in the browser, to represent the 6 main parts of the model. There is also some superstructure, to allow for higher level functions (edit/create, submit, print, choose telescope, etc.), but, again, most of the complexity is in the view for these 6 tabs.


23

PST - “The View”

Let’s look at the actual tool…


24

PST - SubmitWe must support both normal “deadline” submissions, as well as “Rapid Response” submissions (this is fundamentally a policy issue). Upon submission, the proposal handling process is initiated. What is stored is a “proposal” (as represented by a Proposal Data Model). This is NOT the Project Data Model, but rather is a guide to the creation of the initial PDM. This is an important distinction, and something we came to a recent agreement on with ALMA (for them this is the Science View of the PDM).


25

Proposal Handling Tool (PHT)

• Presently, only very initial “wrangling” (view, print), and then splits into GB and VLA specific handling

• GB and VLA separately support:– Adding additional data (edit XML by hand or script)– Fixing ‘problems’– Pretty-printing, for referees and scheduling committee

• Future:– Referee process– Scheduling committee details

• Points of interest:– Requirements for VLA are in hand, but not in the form of the detailed

requirements for other areas, rather more like a “User Story”. We need to transform this to real requirements, including the GB process (which have not been written down to our knowledge).

– Does this go in the PST (with maybe part in the OPT) or as a separate tool?

• The fundamental output from the PHT is Projects, as represented by a Project Data Model.


26

Observation Preparation Tool (OPT)

This is the process that takes a more generic view of a Project, and turns it into something that can actually be used to command the hardware of the telescope (and run ancillary tasks).

The fundamental output of the OPT is Program Blocks (PBs) (remember that a PB is a collection of SBs, with some extra information - mostly “contingencies”)

It needs to support 3 “levels” of user:• Novice (automatic generation of PBs for “standard

modes”)• Intermediate (this is the tough one!)• Expert (allow for script level editing)


27

OPT - commonality with PST

Objects in common with PST:• Sources• Resources (hardware configuration)Things not in common:• Things not in OPT:

– Front page stuff– Authors– Sessions (well, kind of - since they “represent” SBs)– Student Support

• Things not in PST– Scan builder and organizer– PB organizer– Detailed correlator setup tool– Calibrator tool


28

OPT - Detailed Design


29


Modify PB


30


Modify SB


31

OPT - Current StatusWe currently have an early prototype of the GUI (duplicating the

look-and-feel of the PST) in place which will authenticate users and has the most minimal PB functions (create, delete, etc.).

Much of our work here, however, has been on the specification of the details of the objects (the “data models”) for the various parts of the system. We are starting here, knowing that many of the elements in the system will be needed here and will be common. These include the definitions of Proposal, Project, Program Block, and Scheduling Block, and as parts of that, Sources, Resources, and Scans. We start out with an XML document provided by a domain expert, and then turn that into objects. We are currently having detailed discussions with ALMA to attempt to have as much as common in these objects. It is clear that early parts of the process (Proposal) can be common, and that general concepts (major elements of SBs, for instance) can be common; it is not yet clear the level to which true commonality will be achieved.


32

OPT - Plan

Our plan is to have new major functionality available within the OPT roughly every 3 months. Major milestones are:

• 30Apr06: framework with minimal functionality• 31Jul06: Add VLA calibrator database access, initial

spectral setup• 31Oct06: Full calibrator setup, more observation setup• 31Jan07: VLA mostly supported. Some validation/PB

creation• 30Apr07: Beginning prototype WIDAR support• 31Jul07: VLA fully supported; prototype WIDAR mostly

supported• 31Oct07: Prototype WIDAR fully supported


33

Observation Scheduling Tool (OST)

We (well, Barry) have been conducting tests of dynamic scheduling with the VLA during the past 2 (3?) reconfigurations. Again, we consider these as prototypes for the final EVLA subsystem - giving us invaluable information on the practical aspects of dynamic scheduling of a many-element radio interferometer.


34

OST - Block Diagram


35

OST - Lessons Learned

Here are the lessons learned - I need the full list from Barry. A few things I know:• ALMA system didn’t work well (as expected, per Allen Farris’ email)• System is inordinately fond of short SBs• Currently effort-intensive (but getting better)


36

OST - Plan

Here is the plan. I need to get with Barry on this, but my understanding is that he wants the VLA fully dynamically scheduled by the end of ‘06 (yes, he has indeed said that!).

For the EVLA-specific part, we are assigning some effort beginning summer ‘06 to this.

evla software - overall architecture, science & online domains bryan butler nrao

Documents

software design

cmcs evla design

evla software system

evla software group

dcaf slide

amcs cmcs slide

staff project evla

completed design