Session 2: Using OPeNDAP-enabled Applications to Access Australian Data Services and RepositorieseResearch Australasia 2011, ½ Day Morning Workshop, Thursday 10th November 2011

GENERAL INFORMATION

• This is a half-day workshop (9am to 12:30pm)

• 9:00am Introductions and Participants Goals

• 9:15am Session 1: Discovering OPeNDAP data access services

• 10:00am Session 2: Applicable use cases of OPeNDAP data services

− 10:30am Tea Break for 15 minutes

• 11:00am Session 3: OPeNDAP service protocols and features

• 11:45am Session 4: Accessing complementary features and services

• 12:30pm End of Workshop

10:00am Session 2: Applicable use cases of OPeNDAP data services10:00am Session 2: Applicable use cases of OPeNDAP data services

Session 2

• Applicable use cases of OPeNDAP data services for data cataloging and data access using a variety of applications and tools.

• A short tutorial exploring data access using an OPeNDAP-enabled tool within a scripting language such as python.

• 45 minutes in length + 15 minute Tea Break at 10:30am

Spectrum of Use Cases

Application Data Representation

OGC data modeldomain specificgeospatial, 1-D, 2-D

DAP2 data modeldomain neutraln-D, time series

**DAP4 data modeldomain neutralnew data types and data structuresstreaming, compressed, chunked

Common Data Model (CDM)domain specific

Future data modeldomain neutral??

Application Types

Programmatic / Langauge APIFORTRAN, C/C++, JAVA, Python, NetCDF, Java NetCDF

Programmatic / ToolsNetCDF, NCO, PyDAPCustom Tools: OPeNDAP crawler, ocean_prep

Interactive Data ViewerIDV, Panolopy, IDL, MATLAB, iPython (matplotlib), NCL, web browser (metadata)

Interactive AnalysisMATLAB, IDL, iPython, NCLCustom Application: Inudation Modeller

Web ApplicationLive Access ServerIMOS Data Portal (WMS)Custom Java Servlet

ProgrammingDAP2 Legacy Codeexisting tools

DAP2 New CodeNew tools

**DAP4 programminglegacy code support

**DAP4 programmingnew data model and protocolsstreaming support

**DAP4 programmingAsynchronous access modes, server-side processing

Data Access Protocol

Metadata Requestdas, dds, ddx

ASCII/Binary Data RequestSimple data representation

DAP Binary Object Request NcML Data Requestaggregation, virtual data sets

**DAP4server-side operations, async access mode, new data model, posting

Syntax

Return data set infofile.nc.dds - readablefile.nc.ddx - XMLfile.nc.asc - ASCII data return

Select variablesfile.nc.dods?var1,var2,var3

subset arraysfile.dods?var1(0:1:10)

Return file translationsfile.nc.netcdf - NetCDF file

Server-side operationsfile.nc?GEOLOC()Async access mode??

Clients

Programmatic AccessTsunami inudation modeller, NetCDF,NCO, PyDAP, PyNetCDF, MATLAB, IDL, …

Interactive AccessWeb browser - CatalogMATLAB, IDL, Python, Panolopy,…

Data Library & Catalog Servicemetadata harvestingdirectory listingsremote THREDDS services

Web ServiceJava servlet, Java appletGeospatial Information ServiceOPeNDAP data service

Analysis ServiceLive Access Server

Service CapabilitiesDAP2 response metadata, dods, ASCII / Binary

**DAP4 Responseasync access mode, server-side, streaming,

NcMLAggregation serviceVirtual Data Set ServiceRemote Data Access

Metadata Conversion and RDFmetadata definitions, translations (-> ISO) sematics, ontalogyCF->ISO, CF->WMS, CF->WCS

Layered ServicesCatalogue serviceWMS, WCS servicesAuthenticationConformance checksCF metadata checkISO metadata check

**DAP4 features listed is my estimation and not the official specification

Workshop Use-Cases

Application Data Representation

 

DAP2 data modeldomain neutraln-D, time series

     

Application Types

Programmatic / Langauge APIFORTRAN, C/C++, JAVA, Python, NetCDF, Java Netcdf, PyDAP

Programmatic / ToolsNetCDF, NCO, PyDAPCustom Tools: OPeNDAP crawler

Interactive Data ViewerPanolopy, MATLAB, NCL, web browser

   

ProgrammingDAP2 Legacy Codeexisting tools:

DAP2 New CodeNew tools

     

Data Access Protocol

Metadata Requestdas, dds, ddx

ASCII/Binary Data RequestSimple data representation

DAP Binary Object Request NcML Data Requestaggregation

 

Syntax

Return metadata infofile.nc.das - readable file.nc.dds - readable file.nc.ddx - XML metadatafile.nc.help - help info

Select vars and return datafile.nc.asc?var1,var2,var3file.nc.dods?var1,var2,var3

subset arrays, return datafile.asc?var1(0:1:10)file.dods?var1(0:1:10)

Return file translationsfile.nc.netcdf - NetCDF file

Server-side operationsfile.nc?GEOLOC()

ClientsProgrammatic AccessNetCDF, NCO, PyDAP, PyNetCDF

Interactive AccessWeb browser - CatalogPython, MATLAB, Panolopy

     

Service Capabilities

DAP2 response THREDDS data serviceHyrax data service

 

NcMLAggregation service

 Layered ServicesCatalog serviceWMS

Use Case limitations

• Time to access data is dependent on the following factors:

• Hardware and network performance

• Selection of variables and dimensions

• Number of data requests to be issued

− Latency inherent in the data request

• Number of concurrent accesses to the server

Performance limitations to data delivery

Network connection

Network Bandwidth

Data Transfer(MB per second)

*Elapse Time(seconds)

WiFi 2 – 56 Mbps 0.2 – 5 MBps 500+

ADSL modem 2 – 14 Mbps 0.2 – 1.4 MBps 357+

Home LAN 100 Mbps 10 MBps 50

SATA Disk 20 – 40 MBps 12.5+

Office LAN 1000 Mbps 100 MBps ~5.00

Disk Array 120 – 240 MBps ~3.00

Backbone Ethernet 10 Gbps 1,000 MBps ~0.50

QDR Infiniband 40 Gbps 4,000 MBps ~0.12

Lustre Parallel FS 10,000 MBps 0.05+

*Time to transfer a 500 MB data object

Performance limitations to data delivery

Data Request Data Size *Elapse Time(seconds)

Improved Access

Complete File 3 x fields(3D) doubles

250 MB 178 1.0x

One 3D field5 vertical levels

50 MB 35.7 5.0x

3 x 2D fields 30 MB 21.4 8.3x

One 2D field(1250x1000)

10 MB 7.14 24.9x

Subset 3D field(500 x 500 x 5 )

10 MB 7.14 24.9x

Subset 2D field(500 x 500 )

2 MB 1.43 124x

Vertical Column(100 x 100 x 5 )

0.4 MB 0.28 635x

*Time to transfer a data object on an ADSL2 modem = 14 Mbps

DAP-enabled client tools/applications

OPeNDAP Clients (partial list)

http://opendap.org/whatClients

To be demo’ed today.

1. Web browser returning ASCII data

2. Pydap - is a pure Python library implementation of the DAP2

3. NetCDF - is a set of software libraries and self-describing, machine-independent data formats with interfaces to Python, FORTRAN, C/C++, and Java languages

4. NCO – comprises a dozen standalone, command-line programs that take netCDF files as input

5. MATLAB – session 3

6. Panoply – session 4

Web Browser demo

• “.ascii” tells the OPeNDAP service to return the data in ASCII format.− http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/201111

06-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon

• Try accessing multiple variables such as latitude− http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/201111

06-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon,lat

• What other variables are available in the file? − Try accessing “sst” and download to ascii

Tutorial - Subsetting

Modify the variable indice (C array syntax 0..n-1 ), and request ascii data in the web browser

• http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[10:1:50]

What happens if the middle index number is changed from “1” to “2”?• http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106

-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[10:2:50]

Tutorial – subsetting continued

http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[0:1:1439]

Add a new variable to the above URL, separated by a comma, and make a request for ascii data in the web browser:

• http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/20111106-ABOM-L4LRfnd-GLOB-v01-fv01.nc.ascii?lon[0:1:50],lat[0:1:30]

Now do the same thing in the form and modify the indice range• watch out for large indice ranges returning large amounts of data

Tutorial: .dods response

Try the binary response “.dods”

• “.dods” tells the OPeNDAP service to return the data in binary format− http://opendap.bom.gov.au:8080/thredds/dodsC/gamssa_4deg/2011/201111

06-ABOM-L4LRfnd-GLOB-v01-fv01.nc.dods?lon

This is two part binary DAP data object which contains 1) meta data, and 2) binary data structure.

This is the typical response for OPeNDAP enabled client applications.

Pydap

Pydap is a pure Python library implementing the Data Access Protocol, also known as DODS or OPeNDAP. You can use Pydap as a client or server.• http://pydap.org/

To install Pydap on Windows… see the next slide for Windows

To install Pydap on Mac OS X… see the slide for Mac OS X

To install Pydap on Linux… see the slide for Mac OS X

Pydap installation for Windows

To install Pydap on Windows …

1. Install python onto Windows

2. Install easy_install: ez_setup.py

3. Install Pydap: easy_install Pydap

Pydap installation for Mac OS X

To install Pydap on Mac OS X…

1. Python is install on Mac OS 10.5 and 10.6 by default

2. Install easy_install: ez_setup.py

3. Install Pydap: easy_install Pydap

Test Pydap client installation

>>> from pydap.client import open_url>>> dataset = open_url('http://test.opendap.org/dap/data/nc/coads_climatology.nc')>>> var = dataset['SST']>>> var.shape(12, 90, 180)>>> var.type<class 'pydap.model.Float32'>>>> print var[0,10:14,10:14] # this will download data from the server<class 'pydap.model.GridType'> with data[[ -1.26285708e+00 -9.99999979e+33 -9.99999979e+33 -9.99999979e+33] [ -7.69166648e-01 -7.79999971e-01 -6.75454497e-01 -5.95714271e-01] [ 1.28333330e-01 -5.00000156e-02 -6.36363626e-02 -1.41666666e-01] [ 6.38000011e-01 8.95384610e-01 7.21666634e-01 8.10000002e-01]] and axes366.0[-69. -67. -65. -63.][ 41. 43. 45. 47.]

More Pydap client features

See Pydap client: http://pydap.org/client.html

NetCDF API and Tools

NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.• http://www.unidata.ucar.edu/software/netcdf/

To install, go to • http://www.unidata.ucar.edu/downloads/netcdf/index.jsp

To use with python, build netCDF4 and its python module, or …• easy_install netCDF4

NetCDF demo

>>> import netCDF4>>> url = 'http://test.opendap.org/dap/data/nc/coads_climatology.nc’>>> dataset = netCDF4.Dataset(url)>>> var = dataset.variables['SST']>>> var.shape(12, 90, 180)>>> print var[0,10:14,10:14] # this will download data from the server<class 'pydap.model.GridType'> with data[[-1.26285707951 -- -- --] [-0.769166648388 -0.77999997139 -0.675454497337 -0.595714271069] [0.128333330154 -0.0500000156462 -0.0636363625526 -0.141666665673] [0.638000011444 0.895384609699 0.721666634083 0.810000002384]]

>>> print var

<type 'netCDF4.Variable'>

float32 SST('TIME', 'COADSY', 'COADSX')

…

NetCDF demo

Get metadata information about the following data set:• ncdump -h

http://opendap.bom.gov.au:8080/thredds/dodsC/nmoc/oceanmaps2_ofam_fc/latest/ocean_fc_20111108_000_surface.nc

NCO Tools

The netCDF Operators (NCO) comprise a dozen standalone, command-line programs that take netCDF files as input, then operate (e.g., derive new data, average, print, hyperslab, manipulate metadata) and output the results to screen or files in text, binary, or netCDF formats. NCO aids manipulation and analysis of gridded scientific data.• http://nco.sourceforge.net/

To install NCO tools, go to• http://nco.sourceforge.net/#Binaries

NCO tool demo

Download the initial conditions for regional ocean model• ncks -O -F -d xt_ocean,648,979 -d yt_ocean,467,798

http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_analysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp_2010_03_12.nc

View the original file metadata:• ncdump –h

http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_analysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp_2010_03_12.nc

View the subsetted file’s metadata• ncdump –h ocean_temp_2010_03_12.nc

NCO demo

Download the initial conditions for regional ocean model using longitude and latitudes ranges for the dimensions• ncks -O -F -d xt_ocean,143.55,176.66 -d yt_ocean,-28.35,4.75

http://opendap.bom.gov.au:8080/thredds/dodsC/oceanmaps_access_analysis_ogcm/temp/2010/ocean_an_20100312_temp.nc -o ocean_temp2_2010_03_12.nc

Are the files the same (dimensions and lon/lat range)?ncdump –v xt_ocean ocean_temp_2010_03_12.ncncdump –v xt_ocean ocean_temp2_2010_03_12.nc

Tutorial: Pick a demo to try

Please select from pydap, NetCDF, and NCO demos1. Install the software on your machine2. Run a test case and see if the software is installed correctly3. Access a different file from a TDS or Hyrax data service4. Get the metadata information5. Get the coordinate axes data6. Get a subset of data from an array

Thank you

Authors:

Tim F. Pugh1, James Gallagher2, Dave Fulker3

1Australian Bureau of Meteorology, Melbourne, Australia, t.pugh@bom.gov.au2 OPeNDAP, Butte, Montana, USA, jgallagher@opendap.org3 OPeNDAP, Boulder, Colorado, USA, dfulker@opendap.org