![Page 1: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/1.jpg)
THREDDS Data ServerUnidata’s Common Data Model
Background / Summary
John CaronUnidata/UCAR
Mar 2007
![Page 2: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/2.jpg)
HTTP Tomcat Server
THREDDS Data Server
Datasets
catalog.xml
motherlode.ucar.edu
THREDDS Server Application
NetCDF-Javalibrary
IDD Data
•HTTPServer
•NetcdfSubset
•WCS•OPeNDAP
![Page 3: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/3.jpg)
THREDDS Catalogs• XML over HTTP• Hierarchical listing of online resources (datasets)• Container for arbitrary search metadata
– Standard set maps to DC, GCMD, ADN – Unidata/CDP
• Metadata can be inherited• Design goal: Make it easy for data providers• TDS uses for configuration
– Client view vs. server view• Data Access URLS
– “Crossing the protocol boundary”
![Page 4: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/4.jpg)
catalog.xml
![Page 5: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/5.jpg)
Motherlode catalog example
![Page 6: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/6.jpg)
THREDDS WCS 1.0 Server
• Each (gridded) Dataset is WCS• Each Grid is a Coverage • Return formats
– GeoTIFF: floating point, greyscale– NetCDF / CF-1.0 (same as NetcdfSubset Service)
• No reprojections, resampling• GALEON 2
– upgrade to WCS 1.1– Try returning point datasets
![Page 7: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/7.jpg)
![Page 8: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/8.jpg)
THREDDS OPeNDAP Server
• Current version 2.0; NASA ESE standard– Working on new 4.0 protocol spec
• Based on Java-OPeNDAP library – shared development by Unidata/opendap.org
• Any CDM dataset can be served• Server4 (Hyrax):
– latest version of opendap.org C++ library – uses THREDDS catalog generation code– THREDDS Catalogs replace dods_dir
![Page 9: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/9.jpg)
HTTP Tomcat Server
Common Data Model
catalog.xml
hostname.edu
THREDDS Server Application
NetCDF-Javalibrary
IDD Data
•HTTPServer
•NetcdfSubset
•WCS•OPeNDAP
Then a miracle
happens
Datasets
![Page 10: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/10.jpg)
NetcdfDataset
ApplicationScientific Datatypes
NetCDF-Java version 2.2 architecture
OPeNDAP
THREDDS
Catalog.xml NetCDF-3
HDF5
I/O service provider
GRIB
GINI
NIDS
NetcdfFile
NetCDF-4
…Nexrad
DMSP
CoordSystem Builder
Datatype Adapter
ADDE
NcML
![Page 11: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/11.jpg)
I/O Service Provider Implementations
• General: NetCDF, HDF5, OPeNDAP• Gridded: GRIB-1, GRIB-2 • Radar: NEXRAD level 2 and 3, DORADE,
Chinese NEXRAD• Point: BUFR, ASCII• Satellite: DMSP, GINI, McIDAS AREA• In development / tentative
– NOAA CLASS legacy files– Barrowdale DataBlade
![Page 12: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/12.jpg)
Coordinate Systems
Common Data Model Layers
Data Access
Scientific Datatypes
Grid
Point
Radial
Trajectory
Swath
Station Profile
![Page 13: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/13.jpg)
NetCDF-4 andCommon Data Model(Data Access Layer)
![Page 14: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/14.jpg)
NetCDF-4 C library
• 4.0 Beta implements CDM access layer– complete, but waiting for HDF5 release 1.8 to
finalize file format (Maybe this month, 1.5 years late!)
– Persistence format for complete CDM• 4.1: adding Coordinate Systems
– Optional layer, focus on CF-1 (libcf)• 4.?: merge OPeNDAP access (pending
funding)
![Page 15: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/15.jpg)
Coordinate Systems UML
![Page 16: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/16.jpg)
NcML: NetCDF Markup Language
XML representation of netCDF metadata• Core: netCDF data access model• Coordinate System: general and
georeferencing coordinate system• Dataset: redefine, aggregate, subset
Luca Cinquini (NCAR/SCD/ESG), John Caron, Ethan Davis, Bob Drach (LLNL), Stefano Nativi (Florence), Russ Rew
![Page 17: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/17.jpg)
NcML
• NcML Coordinate Systems further developed into NcML-G by Stefano et al.
• NcML Core and Dataset combined into single schema to allow dataset modification
• Aggregation:– Union– Syntactic join on (existing or new) outer dimension– Semantic aggregation of (runtime, forecast time) =
Forecast Model Run Collection
![Page 18: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/18.jpg)
<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/schemas/netcdf/ncml-2.2" location=“/data/nids/N0R_20041119_2147">
<attribute name=“cdm_datatype" value=“Radial" /> <remove type=“attribute” name=“password" /> <variable name="Reflectivity" orgName=“R34768”> <attribute name="units" value=“dBZ" /> </variable>
</netcdf>
NcML example
![Page 19: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/19.jpg)
TDS / NcML example<datasetScan name="Ocean Satellite Data" path="ocean/sat"
dirLocation="R:/tds/netcdf/">
<netcdf> <attribute name="Conventions" value="CF-1.0"/> </netcdf>
</datasetScan>
![Page 20: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/20.jpg)
TDS / NcML aggregation<dataset name="WEST-CONUS_4km Aggregation"
urlPath="satellite/3.9/WEST-CONUS_4km">
<netcdf > <aggregation dimName="time" type="joinNew"> <scan location="/data/ldm/pub/satellite/3.9/WEST-CONUS_4km/"
suffix=".gini" /> </aggregation> </netcdf>
</dataset>
![Page 21: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/21.jpg)
Datasets vs. Files
• Must hide actual location of data files on your server
• Would like to hide actual file format• Must encapsulate collections of files into
logical datasets– Homogenous metadata – Hide arbitrary storage decisions– Minimize number of datasets
![Page 22: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/22.jpg)
Forecast Model Run Collection (FMRC)
![Page 23: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/23.jpg)
Data Model: Sampled Functions
Our phenomena are continuous functions: F: Domain → Range
where Domain = subset of space-time (3 spatial, time) (Ε4) Range = Rn (product set of real numbers)
Our measurements are sampled functions Domain is a point subset = {p, p є Ε4}
M: E4 → Rn
![Page 24: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/24.jpg)
Variables
Variable is a container for an Array of valuesdimensions lat = 64; lon = 128;variables: float temperature( lat, lon);
Domain is a set of points in Index space:Temperature : {[0..63] x [0..127]} → RTemperature : I2 → RVariable : Im → Rn
![Page 25: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/25.jpg)
Coordinate Systems
Coordinate Axis : Im → R{Axis} = Coordinate System : Im → E4
V: Im → Rn
CS: Im → E4 V ° CS-1 : E4 → Rn
![Page 26: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/26.jpg)
Scientific Data Types
• Trying to go beyond index-space subsetting• Trying to satisfy V ° CS-1 : E4 → Rn
– I.e. support subsetting using Space, Time “queries”• Based on datasets Unidata is familiar with
– APIs are evolving• Intended to scale to large, multifile collections• Corresponding “standard” NetCDF file format
conventions
![Page 27: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/27.jpg)
Implementations
Datatype• Grid• PointObs• RadialSweep• Swath
Dataset• GridDataset• FMRCDataset• CollectionOfPointObs• StationCollectionOfPointObs• StationCollectionOfRadialSweep
![Page 28: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/28.jpg)
Conclusions
• CDM is our implementation data model• Map to data access models such as OGC• Current work is to serve collections
instead of individual files.• Dataset is desired level of granularity• Scientific data types are implementations
with specialized access
![Page 29: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/29.jpg)
Datatype Collection
• GridDataset collection of GridDatatype
![Page 30: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/30.jpg)
NetcdfDataset
ApplicationScientific Datatypes
NetCDF-Java version 2.2 architecture
OPeNDAP
THREDDS
Catalog.xml NetCDF-3
HDF5
I/O service provider
GRIB
GINI
NIDS
NetcdfFile
NetCDF-4
…Nexrad
DMSP
CoordSystem Builder
Datatype Adapter
ADDE
NcML
![Page 31: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/31.jpg)
Gridded Datatype
float gridData(t,z,y,x); float time(t); float y(y); float x(x); float lat(y,x); float lon(y,x); float z(z); float height(t,z,y,x);
• Cartesian coordinates• All dimensions are connected• horizontal: lat,lon or projection x,y • time(time) orthogonal 1D• seperable: (x, y) X time X z
![Page 32: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/32.jpg)
GridDatatype methodsCoordinateAxis getTaxis();CoordinateAxis getXaxis();CoordinateAxis getYaxis();CoordinateAxis getZaxis();Projection getProjection();
int[] findXYindexFromCoord( double x_coord, double y_coord);
LatLonRect getLatLonBoundingBox();
Array getDataSlice (Range[] …) GridDatatype makeSubset (Range[] …)
![Page 33: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/33.jpg)
Radial Data
radialData(radial, gate) : distance(gate) azimuth(radial) elevation(radial) time(radial)
• Polar coordinates• All dimensions are connected• Not separate time dimension
![Page 34: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/34.jpg)
Swath
swathData(line,cell) lat(line,cell) lon(line,cell) time(line) z(line,cell) ??
• lat/lon coordinates• not separate time dimension• all dimensions are connected
![Page 35: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/35.jpg)
Unstructured Grid
float unstructGrid(t,z,pt); float lat(pt); float lon(pt); float time(t); float height(z);
• Pt dimension not connected• Looks the same as point data• Need to specify the connectivity explicitly
![Page 36: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/36.jpg)
Point Observation Data
Structure { lat, lon, z, time; v1, v2, ... } obs( pt);
• Set of measurements at the same point in space and time• Point dimension not connected
float obs1(pt);float obs2(pt); float lat(pt); float lon(pt); float z(pt); float time(pt);
![Page 37: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/37.jpg)
PointObsDataset Methods
// Iterator<StructureData>Iterator getData( LatLonRect boundingBox, Date start, Date end);
![Page 38: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/38.jpg)
Time series Station Data
Structure { name; lat, lon, z; Structure{ time; v1, v2, ... } obs(*); // connected } stn(stn); // not connected
![Page 39: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/39.jpg)
StationObs Methods
// List<Station>List getStations( LatLonRect boundingBox);
// Iterator<StructureData>Iterator getData( Station s, Date start, Date end);
![Page 40: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/40.jpg)
Structure { name; Structure { lat, lon, z, time; v1, v2, ... } obs(*); // connected } traj(traj) // not connected
Trajectory Data
Structure { lat, lon, z, time; v1, v2, ... } obs(pt); // connected
• pt dimension is connected• Collection dimension not connected
![Page 41: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/41.jpg)
Profiler/Sounding Station Data Structure { name; lat, lon, time; Structure { z; v1, v2, ... } obs(*); // connected } loc(nloc); // not connected
Structure { name; lat, lon; Structure { time, Structure { z; v1, v2, ... } obs(*); // connected } time(*); // connected } stn(stn); // not connected
![Page 42: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/42.jpg)
Data Types Summary
• Data access through a standard API• Convenient georeferencing• Specialized subsetting methods
– Efficiency for large datasets
![Page 43: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/43.jpg)
File Format#N
File Format#2
File Format#1
CDMVisualization
&Analysis
PayoffN + M instead of N * M things on your TODO List!
NetCDF file
OpenDAP Server
WCS Service
Web Service
![Page 44: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/44.jpg)
Next: DataType Aggregation
• Work at the CDM DataType level, know (some) data semantics
• Forecast Model Collection– Combine multiple model forecasts into single
dataset with two time dimensions– With NOAA/IOOS (Steve Hankin)
• Point/Station/Trajectory/Profile Data – Allow space/time queries, return nested sequences– Start from / standardize “Dapper conventions”
![Page 45: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/45.jpg)
Forecast
Model
Collections
![Page 46: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/46.jpg)
Coordinate Systems: implicit/explicit
• NetCDF, OPeNDAP, HDF data models do not have explicit coordinate systems– so georeferencing not part of API– Need conventions to specify (eg CF-1,
COARDS, etc) • GRIB, HDF-EOS (eg) are explicit
– But no uniform API
![Page 47: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/47.jpg)
47
NetCDF-4
C
Library
HDF5 Library
netCDF-4 Library
netCDF-3Interface
NetCDF-4 C Library
![Page 48: THREDDS Data Server Unidata’s Common Data Model Background / Summary](https://reader036.vdocuments.net/reader036/viewer/2022062810/56815b85550346895dc9885e/html5/thumbnails/48.jpg)
Conclusion
• Standardized Data Access in good shape– HDF5, NetCDF, OPeNDAP– Write an IOSP for proprietary formats (Java)
• But that’s not good enough!• To do:
– Standard representations of coordinate systems
– Classifications of data types, standard services for them