![Page 1: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/1.jpg)
OPeNDAP HyraxBack-End Server (BES)
Customization
ESIP Federation Workshop
15 July 2008Patrick West
James Gallagher
![Page 2: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/2.jpg)
OPeNDAP’s Hyrax Architecture
OLFS
Java Servlet Engine
BES
Unix Daemon
BES Commands
XML- encapsulated object
File system with data files,SQL Database, …
DAP2
THREDDS
HTML
Optional THREDDScatalogs
Module
Module
Module
Module
![Page 3: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/3.jpg)
In what ways can you extend the BES?• Extensions are dynamically loaded from a shared object
module
• Containers and Definitions, and ways of storing them
• Dataset Catalogs
• Interface
• Initialization/Termination
• New commands (like our hello world example)
• New response objects
• New response handlers
• New request handlers (data handlers like hdf4, hdf5, …)
• Aggregation engines
• Methods of returning your data (return as netcdf)
• Reporters
• Exception Handlers
• Debugging
![Page 4: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/4.jpg)
Modules are dynamically loaded into the BES
• Within the BES configuration you specify the modules that you want to load into the BES.
• The module is derived from the C++ abstract class BESAbstractModule.
• The class impelements a C-style function called maker that instantiates the module class
• The class implements an initialization and termination method to set up and clean up the module.
• The module can add named handlers to the different pieces of the BES.
• The module can extend already existing modules, for example extending the DAP2 module.
• The module can modify already existing modules, such as changing the way certain responses are built.
![Page 5: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/5.jpg)
BES Configuration File/usr/local/etc/bes/bes.conf
![Page 6: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/6.jpg)
Running what we have• Let’s see what we have already installed
• besctl start (if it’s not already running)
• bescmdln -h localhost -p 10002
• show help;
• show version;
• show keys;
• set container in catalog values c,data/nc/fnoc1.nc;
• define d as c;
• get dds for d;
• define d as c with c.constraint=“u”;
• get dds for d;
• get ascii for d;
![Page 7: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/7.jpg)
Or use the OLFS
• ${CATALINA_HOME}/bin/startup.sh• Start your browser• Localhost:8080/opendap/• Select data, then nc, then fnoc1.nc• Click the check box next to the variable u• Click the button ‘Get ASCII’
![Page 8: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/8.jpg)
The CSV Data Handler Module
• /root/src/bes-3.6.2/csv-handler• …more ‘data/temperature.csv’
"Station<String>","latitude<Float32>","longitude<Float32>","temperature_K<Float32>","Notes<String>”"CMWM",-34.7,23.7,264.3,"BWWJ",-34.2,21.5,262.1,"Foo”"CWQK",-32.7,22.3,268.4,"CRLM",-33.8,22.1,270.2,"Blah”"FOOB",-32.9,23.4,269.69,"FOOBAR"
![Page 9: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/9.jpg)
Setting up the example
• cd /root/src/bes-3.6.2/csv-handler• autoreconf• ./configure• make install
– This will build and install the module library and the sample data
• make bes-config– This will make the changes to the bes.conf
file
![Page 10: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/10.jpg)
BES Configuration File/usr/local/etc/bes/bes.conf
![Page 11: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/11.jpg)
Restart it all
• ${CATALINA_HOME}/bin/shutdown.sh• besctl stop• besctl start• ${CATALINA_HOME}/bin/startup.sh• Back in the browser
– localhost:8080/opendap/– Select data, then csv, then
temperature.csv– Click the button ‘Get ASCII’
![Page 12: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/12.jpg)
CSVModule class
• Example Module class:– see /root/src/bes-3.6.2/csv-handler/CSVModule.cc
• Implements initialize and terminate methods (and dump)
• Adds: (typical for data handlers)– new request handler– new container– new catalog
![Page 13: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/13.jpg)
Containers• A container represents data
– Typically a data file– Can represent a request to a remote system,
such as WCS
• Has three main parts:1. Symbolic name (‘cool_data’)2. Real name (e.g., a filename)3. Type (nc, ff, csv, ...)
• matches the name of the data/request handler
• Derive from the abstraction BESContainer• The BESFileContainer is built into the BES• You might add ... WCSContainer
![Page 14: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/14.jpg)
Accessing the container
• Implement the access method:
virtual string access() ;
• Returns the data container to be accessed, typically full path to a file.– BESFileContainer returns the stored file name– A WCSContainer might make a WCS request to
the remote server, store the resulting data file, and returns the path to that resulting, cached, data file.
![Page 15: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/15.jpg)
Definitions• Analogous to a view of data in a RDB• Can encompass many containers (available soon)
• Can specify constraints for each container• Can specify data attributes from each container• Specify aggregation engine and command
define d as <c1>[,…,<cn>] [with <c1>.constraint=“…”,<c1>.attributes=“…”, <cn>.constraint=“…”,<cn>.attributes=“…” aggregate using <engine> by <command>];
![Page 16: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/16.jpg)
Dataset Catalogs• OLFS uses BES and THREDDS catalogs for
dataset catalog traversal• Can have multiple catalogs in a single BES
representing any type of catalog (coming soon)
– Filesystem– Database catalog
• Each Catalog is configurable• Catalogs inherit from the abstraction
BESCatalog• Add to the list of catalogs in the module
class:BESCatalogList::TheCatalogList()->add_catalog( catalog ) ;
![Page 17: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/17.jpg)
Typical Catalog Configuration
• Data handlers, e.g. csv-handler, add a BESCatalogDirectory representing a file system.
• In the BES configuration file:– BES.Catalog.catalog.RootDirectory - data root
directory of the catalog named “catalog”– BES.Catalog.catalog.TypeMatch - data types
matched by file extensions– BES.Catalog.catalog.Include - what nodes to
include in the catalog list (usually everything)– BES.Catalog.catalog.Exclude - what nodes to
exclude (e.g. dot files)• Let’s look at /usr/local/etc/bes/bes.conf
![Page 18: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/18.jpg)
Catalogs and Containers
• Can create containers within the context of catalogs using BESContainerStorageCatalog
• Uses BES.Catalog.catalog.TypeMatch to determine data type.
set container in catalog values c,<catalog_file>;
![Page 19: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/19.jpg)
Request Handlers
• Derived from BESRequestHandler• Fills in response objects• Register functions that say “I know how
to fill in some response”
![Page 20: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/20.jpg)
Registering a Request Handler
• Inside the Module class the request handler is registered with the BES
BESRequestHandlerList::TheList()->add_handler( name, handler )
• CSVModule::initialize()
BESRequestHandlerList::TheList()->
add_handler( modname,
new CSVRequestHandler( modname ) ) ;
![Page 21: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/21.jpg)
I know how
• Inside the constructor for the request handler you register functions that know how to fill in responses
add_handler( “name”, function )• CSVRequestHandler::CSVRequestHandler
add_handler( DAS_RESPONSE, CSVRequestHandler::csv_build_das ) ;
add_handler( DDS_RESPONSE, CSVRequestHandler::csv_build_dds ) ;
add_handler( DATA_RESPONSE, CSVRequestHandler::csv_build_data ) ;
add_handler( VERS_RESPONSE, CSVRequestHandler::csv_build_vers ) ;
add_handler( HELP_RESPONSE, CSVRequestHandler::csv_build_help ) ;
![Page 22: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/22.jpg)
How does it work
Definitions def as c1, c2
get das for def
Containers c1,file1,csv c2,file2,csv ce,file3,nc c4,file4,ffdef
def
command
Response Handlers das, BESDASResponseHandler help, BESHelpResponseHandler
Request Handlers csv, CSVRequestHandler das, fill_das_func help, fill_help_func
das
das
csv
csv
das
c1
das
c1
c1
c2
c2c2
das
show help
help
help
help
csv
csv
![Page 23: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/23.jpg)
Putting it together so far
![Page 24: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/24.jpg)
Summary: CSV Data Handler Module
• Look at CSVModule– Adds CSVRequestHandler– Adds BESCatalogDirectory– Adds BESContainerStorageCatalog
• Look at CSVRequestHandler– Adds functions to fill in DAS, DDS, DataDDS,
Version and Help responses
• These functions fill in the respective response objects
![Page 25: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/25.jpg)
25
More information…
• The Following slides provide more information on BES– Interfaces– Callbacks– Request plans– Commands– Response objects– Response handlers
![Page 26: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/26.jpg)
The Interface• There is a single interface into the BES.• Inherits from the abstraction BESInterface
– Installed version uses BESCmdInterface– CEDAR uses BESApacheInterface– Creating BESXMLInterface
• Interface runs a set of steps:– initialize the BES environment– build the execution plan– execute the request plan building the response object– transmit the response object– log the status of the request– report on the request– end the request
![Page 27: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/27.jpg)
BESDataHandlerInterface
• BESDataHandlerInterface structure is carried throughout the execution of the commands– Holds on to the response handler– Stores the list of containers– Holds the action being taken– Holds map of name/value pairs to be used during
execution– Holds an error object if an error/exception occurs
![Page 28: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/28.jpg)
Init/End callbacks
• Can register callbacks during the initialization and end steps
static void BESInterface::add_init_callback( p_bes_init init ) ;
static void BESInterface::add_end_callback( p_bes_end end )
typedef bool (*p_bes_init)( BESDataHandlerInterface &dhi ) ;
typedef void (*p_bes_end)( BESDataHandlerInterface &dhi ) ;
• Examples:– Authentication/authorization
– Initialize database connections
– Clean up database connections and files
![Page 29: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/29.jpg)
Building the request plan
• Derived classes of BESInterface implement:virtual void build_data_request_plan() ;
– BESCmdInterface parses the incoming command string
– BESApacheInterface translates information from apache and parses the incoming command string
– BESXMLInterface parses the XML document
• Creates a response handler to execute the command
![Page 30: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/30.jpg)
Commands• String commands sent from client to server• Built in commands:
– show help; (version, process, status, keys)– set container …– show containers;– delete container <container_name>;– delete containers;– define …– show definitions;– delete definition <def_name>;– delete definitions;– set context <name> to <value>;– show context;
![Page 31: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/31.jpg)
DAP Commands• DAPCommandModule adds:
– get das for <def_name> [return as type];– get dds …– get ddx …– get dods …– show catalog [for “node”]; (info)
• dap-server modules (www,usage,ascii) add:– get info_page …– get ascii …– get html_form …
• Data handlers (nc, ff, csv) don’t add any new commands.
![Page 32: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/32.jpg)
Response Objects
• Derived from BESResponseObject.– BESDASResponseObject– BESInfo
• No methods required to be implemented
• Created by a BESResponseHandler• Filled in by BESResponseHandler or
delegated to BESRequestHandler
![Page 33: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/33.jpg)
Informational Response Objects• Built in informational response objects derived from
BESInfo:1. BESTextInfo
2. BESHTMLInfo
3. BESXMLInfo
4. BESSilentInfo
• Each one formats an informational response according to its type
• Can add new informational response objects.BESInfoList::TheList()->add_info_builder( name, function ) ;
1. Function instantiates the derived BESInfo object.
2. Created in BESResponseHandler instances by calling:BESInfo *info = BESInfoList::TheList()->build_info() ;
1. Set in BES configuration file:
BES.Info.Type=<name> e.g. txt, html, xml
![Page 34: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/34.jpg)
Response Handlers:
• represent a specific response, such as a DAS response, a DDS response, a help response ...
• know how to create the response object (DAS, DDS, BESInfo)
• do not necessarily fill in the response object, but know how the response should be filled in
• BESDASResponseHandler knows to create a DAS object and that it needs to go to each request handler for each container.
• A version response handler knows to create an informational response object and that it needs to go to all registered request handlers.
• A status response handler knows to create a text response object and fill it in with the status of the server.
• know how to transmit the response object
![Page 35: OPeNDAP Hyrax Back-End Server (BES) Customization](https://reader035.vdocuments.net/reader035/viewer/2022062309/568159d8550346895dc72648/html5/thumbnails/35.jpg)
Implementing a Response Handler
• Inherits from the C++ abstract class BESResponseHandler (e.g. BESDASResponseHandler, BESHelpResponseHandler)
• Implement the methods:
virtual void execute( BESDataHandlerInterface &dhi ) ;
– Creates the response object– Does the work to fill in the response object, or delegate to a
request handler to fill in the response object
virtual void transmit( BESTransmitter *transmitter, BESDataHandlerInterface &dhi ) ;
– Transmits the resulting response object using the appropriate method on the transmitter (more later)