reflex workflow developement - eso · eso pipeline workshop, oct 24 2014 reflex workflow...
TRANSCRIPT
![Page 1: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/1.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex workflow developement
Enrique GarciaESO Pipeline Systems Group
![Page 2: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/2.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex Canvas
![Page 3: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/3.jpg)
ESO Pipeline Workshop, Oct 24 2014
Workflow Overview
![Page 4: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/4.jpg)
ESO Pipeline Workshop, Oct 24 2014
Calibration cascade
![Page 5: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/5.jpg)
ESO Pipeline Workshop, Oct 24 2014
Designing a workflow● Steps previous to use Reflex:
● Design modular pipelines: i.e. with recipes performing granular tasks. Avoid cases where I/O of the recipes depend on recipe parameters.
● Think about the supported observing modes. Optional calibrations sometimes imply quite different reduction cascades.
● Start with an stable version of the pipeline I/O and parameters.● Carefully design the FITS file categories.● Design the workflow layout in paper
● Steps using Reflex tool:● Create Data Organisation rules (OCA rules) which mimic the workflow
layout as closely as possible● Use a workflow template (iiinstrument)● Create the graphical workflow layout● Skip interactivity development until the end.
![Page 6: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/6.jpg)
ESO Pipeline Workshop, Oct 24 2014
Documentation
● Reflex User Manual ftp://ftp.eso.org/pub/dfs/reflex/ReflexUserManual-3.5.pdf
● Reflex Workflow Developers Guide: ftp://ftp.eso.org/pub/dfs/reflex/reflex_dev_guide-1.0.pdf
●Reflex paper: http://eso.org/sci/software/pipelines/documents/reflex_aa.pdf
![Page 7: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/7.jpg)
ESO Pipeline Workshop, Oct 24 2014
Data Organisation: OCA Rules
● OCA rules are the mechanism used to perform Data organisation in a Reflex workflow● OCA rules and workflow graphical layout are very closely related.● OCA rules are written in a text file with a given syntax.● There are three types of rules:
● Classification. Based on FITS keywords, it specifies the category of the file we have (“I am a raw flat”)
● Organization. Files are grouped by keywords and each group triggers an action (“These flats create a master flat”)
● Association. Each action can associate static calibrations or products created by other actions (“To create a master flat I need a master bias“)
![Page 8: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/8.jpg)
ESO Pipeline Workshop, Oct 24 2014
File Purpose
● Category is not enough to determine the use of a file.
Example: bias for a science can be different as bias for the flat● The Purpose is the role a file will have in the reduction cascade => Path in the association tree● A given file can have a list of purposes.
![Page 9: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/9.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: FitsRouter
● The FitsRouter will split the input by
category.There are two modes:● Routing by category explicitly. Just create a port
with the name of the category.● Configuring a port. Create a configuration
parameter with the name PORT_config and list the desired categories.
![Page 10: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/10.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: DataSetChooser● It allows to select the Datasets to reduce● Datasets can be inspected and the calibration cascade will be shown.● Purposes can be seen as the path to the leaf in the tree● If a Dataset is incomplete, it will show which are the missing files.
![Page 11: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/11.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: RecipeExecuter
● The RecipeExecuter has to be instantiated explicitly:
Tools -> Instantiate Component. ● The list of recipes that will appear there is the result of the command esorex –recipes.● For each recipe parameters, an actor parameter is created with the name “recipe_param_nn”, where nn corresponds to the parameter order. If recipe parameters change, the RecipeExecuter should be reinstantiated.
![Page 12: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/12.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: SofSplitter & SofAcummulator
● In order to handle properly Purposes, the RecipeExecuter's have to be enclosed between SofSplitter/SofAccumulator.● SofSplitter will create as many groups as different purposes are in the input and will trigger the RecipeExecuter for each of the purpose groups.● SofAccumulator simply combines back all the products in a single channel.
![Page 13: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/13.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: ProvenanceExplorer
It displays the effective reduction cascade used to create a given product. It will be renamed to ProductExplorer
![Page 14: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/14.jpg)
ESO Pipeline Workshop, Oct 24 2014
RecipeLooper
![Page 15: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/15.jpg)
ESO Pipeline Workshop, Oct 24 2014
PythonActor I
● It is able to execute generic python code.● To translate from/to Reflex ports to/from python script arguments a special syntax is used, with the help of a Python module● To create a python actor, use the menu Tools->Instantiate Component and type org.eso.PythonActor.
![Page 16: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/16.jpg)
ESO Pipeline Workshop, Oct 24 2014
Python Actor II
A python script can be “reflexed” using the following syntax:from reflex import *
parser = ReflexIOParser()
#Define inputs
parser.add_option("-i", "--in_sof", dest="in_sof")
#Define outputs
parser.add_output("-o", "--out_sof", dest="out_sof")
(inputs, args) = parser.parse_args()
outputs = parser.get_outputs()
#Set the output
outputs.out_sof = inputs.in_sof
parser.print_outputs()
sys.exit()
Importing Reflex
Define Inputs/Outputs
Getting inputs
Setting outputs
![Page 17: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/17.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex Python Framework I
● Reflex is delivered with a python library that helps to create interactive user interface rather easy using matplotlib and wxPython.● The reflex_interactive_app module is a framework to create interactive windows.● The pipeline_product module eases the reading of FITS images, spectra and tables, using pyfits.● The pipeline_display module produces scatter plots, image displays and spectra plots using matplotlib.● The reflex_plot_widgets module allows to add extra interactivity within the matplotlib plots.● These modules are accessible directly when using a python script inside the PythonActor.
![Page 18: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/18.jpg)
ESO Pipeline Workshop, Oct 24 2014
Examples of interactive windows
![Page 19: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/19.jpg)
ESO Pipeline Workshop, Oct 24 2014
Examples of interactive windows
![Page 20: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/20.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex Python Framework II
Example of using reflex_interactive_app framework: def setInteractiveParameters(self):
return [reflex.RecipeParameter(recipe=rec_name, displayName="par1", group="Limits"]
def readFitsData(self, fitsFiles):
self.frames = dict()
for f in fitsFiles:
self.frames[f.category] = PipelineProduct(f)
pro_raw = self.frames[“MY_CAT”]
pro_raw.readImage()
def addSubplots(self, figure):
self.raw_plot = figure.add_subplot(111)
def plotProductsGraphics(self):
img_raw_disp = pipeline_display.ImageDisplay()
img_raw_disp.setLabels('X', 'Y')
tooltip = paragraph("Raw image")
img_raw_disp.display(self.raw_plot, "Raw image", tooltip, pro_raw.image)
Defining the recipeparameters to show
Reading the FITSData from the input
Defining the plotlayout
Plotting the data
![Page 21: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/21.jpg)
ESO Pipeline Workshop, Oct 24 2014
Directories setup● There are 6 directories:
– CALIB_DATA_DIR: Extra directory with static calibrations provided by the pipeline. Used by the DataOrganizer
– RAWDATA_DIR: The dir where the user data resides. Used by the DataOrganizer
– BOOKKEEPING_DIR: Reflex stores here all the information about previous executions (I/O sof, parameters, etc...). Used by the DataOrganizer, RecipeExecuter, DataSetChooser, PythonActor and ProvenanceExplorer
– LOGS_DIR: The output of esorex are stored here. Used by the RecipeExecuter.– TMP_PRODUCTS_DIR: The products created by each execution of a recipe are
saved here. Used by the RecipeExecuter– END_PRODUCTS_DIR: The science products from each dataset are stored
here. Used by the ProductRenamer, ProvenanceExplorer● The installation script of Reflex takes care of setting up all the workflow directories for the demo data● Workflows usually setup a ROOT_DATA_DIR and define all the other dirs from this one
![Page 22: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/22.jpg)
ESO Pipeline Workshop, Oct 24 2014
Saving workflows
● There are two formats to save a workflow:● KAR. This is the default format.
Binary format● XML. This has to be exported.
“Human-readable”● The paths to the python scripts and
OCA rules file are saved together with the workflow.
![Page 23: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/23.jpg)
ESO Pipeline Workshop, Oct 24 2014
Lazy Mode● Lazy mode for RecipeExecuter.
– It works by comparing the input of the current execution with all the previous recipe executions:
• All files must be the same
• All files must have the same checksum
• All files must have the same date
• All recipe parameters must be the same
– If a recipe at the beginning of the workflow is set to Not-Lazy mode, the input of the next recipes will be new and lazy mode will not be triggered.
● Lazy mode for DataOrganizer.– It avoids the organization of all the data in subsequent workflow
runs. – It works similar to lazy mode for recipes
![Page 24: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/24.jpg)
ESO Pipeline Workshop, Oct 24 2014
Questions?
![Page 25: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/25.jpg)
ESO Pipeline Workshop, Oct 24 2014
Backup slides...
![Page 26: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/26.jpg)
ESO Pipeline Workshop, Oct 24 2014
Data Organisation: OCA Rules II
Classification:
if DPR.CATG like "%SCIENCE%" and DPR.TYPE like "%OBJECT%" then
{
REFLEX.CATG = "RRRECIPE_DOCATG_RAW";
REFLEX.TARGET = "T";
}
if DPR.CATG like "%CALIB%" and DPR.TECH like "%IMAGE%"
and DPR.TYPE like "%STD%" then
{
REFLEX.CATG = "RRRECIPE_CALIB_DOCATG_RAW";
}
![Page 27: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/27.jpg)
ESO Pipeline Workshop, Oct 24 2014
Data Organisation: OCA Rules III
Organisation:
select execute(CALIB_IMG) from inputFiles where REFLEX.CATG == “RAW”
group by INS.FILT1.NAME, OBS.ID, OBS.TARG.NAME, TPL.START;
![Page 28: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/28.jpg)
ESO Pipeline Workshop, Oct 24 2014
Data Organisation: OCA Rules IV
Association:
action CALIB_IMG
{
minRet = 0; maxRet = 1;
select file as STATIC_MASK from calibFiles where REFLEX.CATG == "STATIC_MASK";
recipe rrrecipe_calib;
product IMG_CALIBRATED { REFLEX.CATG = "IMG_CALIBRATED"; PRO.CATG = "IMG_CALIBRATED"; PRO.EXT="tpl_0000.fits";}
}
action COMBINE_IMG
{
minRet = 1; maxRet = 1;
select file as IMG_CALIBRATED from calibFiles where PRO.CATG == "IMG_CALIBRATED";
recipe rrrecipe;
product IMG_OBJ_COMBINED { PRO.CATG = "IMG_OBJ_COMBINED";PRO.EXT="tpl_0001.fits";}
}
![Page 29: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/29.jpg)
ESO Pipeline Workshop, Oct 24 2014
How to configure an actor
● Each actor has input and output ports.● To edit them, right click on “Configure ports”
● An input multiport can receive several tokens which are combined together● An output multiport distributes the token to several actors● Many output ports are single ports and cannot be changed.● Each actor has some parameters that can be configured by right click “Configure actor”.
![Page 30: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/30.jpg)
ESO Pipeline Workshop, Oct 24 2014
Reflex Actors: ProductRenamer
● Rename_keywords specifies the pattern used to create the file name, based on the available FITS keywords.
● The directory to be used will be FinalProductDirectory/SubDir. Subdir is usually defined by the workflow to be the execution timestamp.
● The actor is actually a jython script which can be modified by double-clicking it. Its code is part of its configuration, and therefore it is saved with the workflow. To use a more recent version, just copy and paste.
![Page 31: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/31.jpg)
ESO Pipeline Workshop, Oct 24 2014
Python Actor Example
An example with some data processing: #Retrieve input
in_sof = inputs.in_sof
#Get the input files
files, dataset = parseSof(in_sof)
#Do the stuff
for file in files:
hdulist = pyfits.open(file.name,mode='update')
newdata = hdulist[0].data / 2
hdulist[0].data = newdata
hdulist.flush()
Parsing the Reflex format
Using pyfits
Updating the file
![Page 32: Reflex workflow developement - ESO · ESO Pipeline Workshop, Oct 24 2014 Reflex workflow developement Enrique Garcia ESO Pipeline Systems Group](https://reader030.vdocuments.net/reader030/viewer/2022040311/5d6226da88c993e75e8b9e11/html5/thumbnails/32.jpg)
ESO Pipeline Workshop, Oct 24 2014
Saving workflows in SVN●.● If you are going to save the workflow in SVN, eport as KML and do the following:
● Set ROOT_DATA_DIR to ROOT_DATA_PATH_TO_REPLACE
● Set CALIB_DATA_DIR to 1CALIB_DATA_PATH_TO_REPLACE/ins-@VERSION@
● Set the paths to OCA rules to @prefix@/share/esopipes/ins-@VERSION@/reflex/ins_wkf.oca
● Set the paths to Python actors to @prefix@/share/esopipes/ins-@VERSION@/reflex/ins_interact.py