november 2010deltares configuration course1 day 2

November 2010Deltares Configuration Course 1

Day 2


TimeSeriesSets


FEWS Time Series

Time series are available from two source:

external

simulated

Time series are in two categories in relation to time:

historical (continuous in time)

forecasting (characterised by its start time)

Time series can be in four formats:

0D – scalar

1D – vector or longitudinalprofile

2D – grid

2D – polygon

Time Series are handled in the form of Time Series Sets


FEWS Time Series – external historical

00:00

04:00

07:00

10:00

Complete series

. Added incrementally

. Can be edited by the user

. Only new and changed values are stored

Complete series

Manual edit

Time of import


FEWS Time Series – external forecasting

Time of Forecast

. Added and stored individually

. Can be edited by the user

. Usage of series depends on T0

00:00

04:00

08:00

Series at 08:00

Series at 06:00

00:00 04:00 08:0006:00


FEWS Time Series – simulated historical

System Time

. Continuous in time

. Referenced by the forecast model that produced them

. Approved series shown automatically (including history)

. Most often linked by model states

04:00

08:00

12:00

Series at 16:00

Series at 08:00

00:00 04:00 08:00 12:00

16:00

16:00


FEWS Time Series – simulated forecasting

System Time

. Continuous in time (in combination with simulated historical)

. Referenced by the forecast model that produced them

. Approved and selected series can be viewed

04:00

08:00

12:00

Series at 12:00

00:00 04:00 08:00 12:00 16:00


FEWS Time Series – temporary

temporary time series

time series used in pre- / post-processing not stored in database


Time Series Sets

Properties of time series:

• location Id – or – locationSetId• parameter Id• timestep• relativeViewPeriod

• valueType (scalar, grid, profile, ..)• timeSeriesType (external historical, external forecast, ..)• moduleInstanceId (import, PDM, KW, ISIS, ..)• readWriteMode

• expiryTime (when to delete from database)• synchLevel

• simple transformations (delay, multiplier, divider, incrementer)• ensembleId


Time Series Sets

• All data handling is done through Time Series Sets

• Some elements are only used when retreiving time series, other elements only when storing data


Time Series Sets - Examples

Import Run General Adapter Run


FEWS Workflows and Modules


FEWS Workflows

• Workflows are logical sequences of running forecast modules• Workflows must be registered in the workflow descriptors

– Module Instance ID– Sub-workflows– Run independent– Fallback activity– Ensemble


FEWS Modules and Module Instances

. Modules

are FEWS plug-ins that can be used in a workflow are registered in moduleDescriptor file as Java Classes are part of the system configuration

. Module Instances

are configured modules of a particular module are registered in moduleInstanceDescriptors file are part of the regional configuration


Sub workflows

Workflows can be nested:• Workflow “DO_ALL”

– (sub)Workflow “DO_Step1”– (sub)Workflow “DO_Step2”

Where:• (sub)Workflow “DO_Step1”

– ModuleInstance “DO_Step1a”– ModuleInstance “DO_Step1b”– ModuleInstance “DO_Step1c”

• (sub)Workflow “DO_Step2”– ModuleInstance “DO_Step2a”– ModuleInstance “DO_Step2b”– ModuleInstance “DO_Step2c”

FEWS Explorer:

. Manual Forecast Display Workflow “DO_ALL” (sub)Workflow “DO_Step1” (sub)Workflow “DO_Step2”


Introduction to some available modules

. Import

. Export

. Transformation

. General Adapter

. Lookup

. ErrorModel

. Correlation

. PerformanceIndicator

. ThresholdEvent

. Report


FEWS Workflows

FEWS Workflows can be run (and debugged) on module basis (via F12)


Importing data


Flood Forecasting processing steps

Create evaporation profiles for hydrological model

Calculate catchment average precipitation based on rain

gauge data

Import data from external sources

Combine precipitation sources (e.g. Rain gauge, Radar data,

Meteorological Forecast models)

Run hydrological model to calculate discharge for

catchments

Run hydrodynamic model to calculate water level for

forecast points

Prepare input for hydrodynamic model based on hydrological

models, gauging stations, tidal boundaries

Run error correction model to update output form

hydrodynamic model


Importing data

Import Modules allows data from external sources to be imported

. Various Scalars – e.g. PI XML and Wiski

. ASCII grid time series

. Binary GRIB (grid) and BIL time series

. Currently around 50 data import types

FEWS imports files from a directory specified in the global.properties


Import Module

Key Features:

. ID mapping

. Flag mapping

. Units mapping

. Time zone mapping

. Tolerance

. Validate


EA Import Module


Import Module – ID mapping

• Matching ‘outside world’ to the ‘FEWS world’– locations (ids)– parameter (ids)

• Three levels– direct mapping location names AND parameter names differ– indirect either location names or parameter names differ– 1:1 location and parameter names do NOT differ

• Used when importing and exporting time series


Id Mapping

• Direct

• Indirect

• 1:1


Import Module – Unit conversion

• Map internal units to external units• Used when importing and exporting time series• Extra option to convert datum of level recordings (external unit mAOD)


Import Module – Flag conversion

• Map internal quality flags to external quality flags• Used when importing and exporting time series


Import Module – other elements

Time zone mapping

database stores all data in GMT, specify timezone when importing

Tolerance

when imported data is not exactly observed at the cardinal time step

Validate

Imported files can be validated against their own schema


Import Module - Exercise

Import Time series from EA-XML format (specific Environment Agency format)

• Analyse the 5 time series– 2 * Observed_waterlevel for Hydro1 and Hydro2– 2 * Rainfall for Raingauge and Meteostation– 1 * Temperature for Meteostation

• Add elements to ID mapping table• Run import workflow• Analyse data with the Time Series Display


Validating Data

• Allow quality checking of all scalar time series data

• Difference in hard limit and soft limit checks– Hard limit checks mark data as unreliable– Soft limit checks mark data as doubtful


Validating Data

• Option to enter monthly limits• Rule can be specified for one time series or for a complete set


Validating Data - Exercise

• Check the data just imported• Define validation rules• Enter the rules in the ValidationRuleSets XML file• Delete the data store• Run import workflow• View data• Check the difference!


Thresholds


Example (Map)

Willington Cableway (Midlands, UK)

!

T0


Example (Graph)

Willington Cableway (Midlands, UK)


Thresholds Utility

The thresholds utility checks when time series cross thresholds

Observed and simulated time series Icons are shown on the map Graphs/lines cross threshold lines Events can be triggered (MC) Up events and down events

Three configuration files

thresholdWarningLevels.xml definition of (unique) thresholds + details thresholds.xml the grouping of the (selected) thresholds thresholdValueSets.xml timeseries (loc/par.) and actual levels info


Thresholds - thresholdWarningLevels

• Unique threshold information– colors: displays, reports– icons

• normal/warning• historic colour• forecast colour !


Thresholds Utility - Thresholds

• Thresholds usage– groups– selection(s)


Thresholds Utility – Thresholds Values Sets

• Detailed threshold information– per location/parameter– actual values to be crossed for the different thresholds– ‘triggers’ (action events for up- or downcrossings)


Thresholds – Exercise

Add thresholds for the hydro locations


Presentation of data in FEWS


Display of data - Filters



Filters are used to:– define the locations that are displayed on the map– define the locations that are displayed in the list box– define the parameters that are displayed in the list box



Filters • organised in folders with child filters• arrange time series sets in logical groups• import elements in the time series set

– extra options• view permissions• validation icons visible• link to map extent



Allow/Disallow editing

Relative View Period for Icon

on main map

synchLevel – for live system

(5=edited data)



child filter organised in sub-folders (childs)


Time Series Display

Time Series Display is used to show data in a graph and in a table

• Editing and viewing data• Pre-configured displays• Thresholds• Historic Events


Time Series Display

Time Series Display uses 4 configuration files

– TimeSeriesDisplayConfig– Thresholds and ThresholdValueSets– DisplayGroups– HistoricalEvents


Time Series Display - TimeSeriesDisplayConfig

• default view period• time markers• parameters• module instance mappings


Time Series Display - DisplayGroups

Display groups• are pre-defined displays• are used to create folder structure of displays• can be nested


Time Series Display - DisplayGroups

display element contains time series to display

multiple sub plots

for each subplot – time series

Options– minimum– maximum– convertDatum– relativeViewPeriod– nrOfRecentForecasts


Time Series Display - Exercise

Exercise: Add pre-configured displays

• Add a new display group for the new locations• Add a new display for the hydro locations• Add a new display for the meteo locations


FEWS Data manipulation modules





gauge data





catchments


forecast points




hydrodynamic model


Interpolation/Transformation Module

Interpolation Module was separate, but now integrated with Transformation Module

The Interpolation module generates data at desired locations or at desired points in time by means of

Serial Interpolation methods

linear interpolation

block interpolation

default value

extrapolation

Spatial Interpolation methods

Kriging

Inverse Distance

bi-linear interpolation

grid cell averaging over catchments


Interpolation Module – serial interpolation


Interpolation Module – Exercise

• Analyse module that fills gaps in the water level time series• Add the new module instance to the module instance descriptors• Configure a workflow that executes the interpolation module instance• Check results!


Interpolation

Possible problems / issues of note

Time Series Set to Interpolate

ModuleInstanceId of series!!! (ImportTelemetry)

Relative View Period

When interpolating to fill gaps

Input time series same as output time series

Using LocationSets and/or Locations

Linear interpolation – can be done using a locationSet

Default – take care as default value may vary per location


Transformation Module

. Concepts of transformations function

. User defined function

. Rating curve

. Catchment averaging

. Data Hierarchy

. Typical Profiles

. Aggregation / dis-aggregation

. Equidistant to non-equidistant

. Exercise: Adding rating curves


Transformation - Configuration


Transformation – Input/Output

Inputs and Outputs are defined as TimeSeriesSet4


TransformationUser defined functions and rating curves

Different function / parameters per segment / range

Define ranges in rating curves file / coefficientsets file


Transformation – User defined functionSchema


Transformation – User defined functionExample


Transformation – User defined functionFunctions

Generic function parser used

“Any” function can be defined


Transformation – Rating curve

. Same approach as other transformations using stageDischarge class

. Convention on inputId / ouputId & parameter names

. Rating curves are contained in the CoefficientSetFiles (RatingCurves.xml)


Transformation – stageDischarge (Rating curve)

Rating curves are linked to locationsOne location orlocationSet

Use coefficients from CoefficientSetFile


Transformation – stageDischarge (Rating curve)

• StageDischargeCoefficientSets.xml (name not fixed) • In CoefficientSetsFiles folder• Referenced in Transformation ModuleInstance• Rating curve defined only once

Referenced by locationId


Transformation – Weighted averaging

Convention on inputId / ouputId & parameter names formula: (X1 * a + X2 * b + X3 * c) / (a + b + c) number of locations is flexible, missing locations will be left out


Data Hierarchy

Fallback options to complete a series

used first

used second

used third


Transformation – Data Hierarchy / Merge

used first

used second

result


Transformation – Typical profile

Examples of typical profile use:

Evaporation series (one value per calendar day) Temperature variation across day


Transformation – Typical profileFixed data/time


Transformation Module - Exercise

Add rating curve for location Hydro1

Analyse transformation file

register module instance in descriptors file add module instance to the exercise workflow


FEWS External Modules/Models





gauge data





catchments


forecast points




hydrodynamic model


FEWS External Modules

Available Model Adapters

CEH Adapters (SNOWP, SNOW, PDM, KW, ARMA, TCM) HR (ISIS) PlanB Adapters (TRITON & PRTF) DHI Adapters (Mike11, NAM) Midlands Region (DODO, MCRM) Southern Region (STF) Northwest Region (NW TF – Common Adapter) Wales (SW Overtopping module - Common Adapter) SouthWest (Bruton/Holbeam Dam module – Common Adapter) Delft (SOBEK, RIBASIM, HYMOS, Sacramento) SMHI (HBV) University of Karlsruhe (PRMS) JRC (Lisflood - PCRaster)

http://public.wldelft.nl/display/FEWSDOC/Models+linked+to+Delft-Fews


General Adapter

General adapter concepts State handling ID mapping


General Adapter -Concepts

Delft-FEWS

General Adapter

Module Adapter

Modulemoduleinput data

moduleoutput data

published interface

input data(pi xml)

Pre Post

output data(pi xml)


General Adapter – Configuration

From global.properties

General Adapter keywords

DELFT-FEWS reserved keyword


General Adapter – Export Activities

Export Activities

Time Series – scalar

Time Series – grids (map stacks)

Time Series – profile

States

ModuleDataSet

ModuleParameters


Export Activities – Scalar/longitudinal time Series

. Exports time series from database to a PI-XML file

. Assign File name to export data to

. Assign Time Series Sets to export

. Written to exportDir

. LocationID’s & ParameterID’s will be translated on export (as indicated in IdMapping)


Export activities - grid time series

. Same principle as exporting scalar time series

. Define name of export file

. Export file contains reference to grid

. Data exported as a “mapstack”

Files:

pcr00000.000

pcr00000.001

..

..

pcr00000.023

pcr00000.024


Export activities - grid time series

Supported grid file formats

ASC: ArcView/ARC-INFO grid interchange file

Time Series as map stack

PCR: PCRaster file format

Time Series as map stack

BIL: USGS standard grid format

Time series in one file


Export Activities – Module parameters

. DELFT-FEWS administers module parameters

. Exports to external module as PI-XML file

. Held in Config\ModuleParFiles


Export Activities – Module dataset

DELFT-FEWS administers module dataset

Dataset: native module file Held in Config\ModuleDataSets by DELFT-FEWS as a ZIP file

Exports to external module by extracting data in ZIP file

Exported to a “root”directory


Exporting States

DELFT-FEWS can manage module states

What is a state?

Module initial conditions

Handled in “native”module format

Type of state;

Cold State – default initial conditions

Warm State – initial conditions from a previous run

For each module requiring management of states;

ZIP file of cold state MUST be available in ColdStates dir.


Exporting States

Defining “how” to export state

moduleInstanceId

stateExportDir

stateConfigFile

stateLocations

read write

From the perspective

of the model!!


Exporting States

stateConfigFile

For export: Written by DELFT-FEWS ; read by adapter

For import:Written by adapter ; read by DELFT-FEWS


Exporting States

. Defining “what” to state export; stateSelection

. State selection determines length of module run!!!!!

. Cold State: Always use a cold state to initiate module run

. Warm State: Use most suitable/recent state in search period


04:00 16:00 04:00 16:00 04:00

• Start 02:00 Jan 05. T0 16:00 04 Jan

• Start 7:30 and T0 07:00 Jan 05• Start 19:30 and T0 19:00 Jan 05

• Start 07:30 and T0 07:00 Jan 06

• Start 02:00 Jan 06. T0 16:00 05 Jan

States

02:00 02:0007:00 19:00 07:00

State Search period


General Adapter – Execute Activities

. Running modules

. Executables or Java classes

. Arguments can be passed

. Timout is set in case module “hangs”

. If Diagnostic file indicates error – or non zero return code

Executable considered to have failed


General Adapter – Import Activities

Import Time Series

Scalar Longitudinal Grid

Import States



. Importing time series

. Same principle as exporting

. File name & Time Series Set to save data to

. IdMapping used to translate locationId’s and parameterId’s

. Grids imported as mapstack + XML file



. Importing states

. Simple instruction where to find stateConfigFile


Burn-in profiles

. Avoid “abrupt” shock to model on startup

. Mainly relevant to HD modules (stability)

Only applied when starting from a cold state

Identify start value in cold state Gradual “climb” to actual value


Burn-in profiles

Burn-in at Temple Sowerby


Startup/Shutdown activities

Clean up files from module

Startup – before running module Shutdown – after completing module


Module diagnostics/debugging

PI-XML includes module diagnostics exchange

To be written by module/adapter

General Adapter Defines where diagnostics file is expected

Not all logging passed – use native module log files

External module log inserted into NFFS log

Most messages DEBUG level

Error/Fatal: System messages

Detecting module failure

. Error in diagnostics log

. non-zero return code


General Adaptor - Exercise

Add unit hydrograph model


FEWS Gridded Data


FEWS Gridded Data

Fews supports the display of both irregular and regular grids

Available (common) Import types

ArcInfoAscii BIL BUFR CSV PI HDF5 NetCDF


FEWS Gridded Data

Commonly used data types

Precipitation Soil Moisture Pressure Wind Temperature Wave height Surge


FEWS Gridded Data

Data can be manipulated using transformation and interpolation modules

Cookie cutter Interpolation Transformation Data hierachy


FEWS Gridded Data


Gridded Data - Exercise

Import Gridded Data

november 2010deltares configuration course1 day 2

Documents