accessing raster data in rasdaman data cubes via r two...

18
Accessing Raster Data in Rasdaman Data Cubes via R using a Shiny App and the CubeR Package Mattia Rossi 1,2 , Daniel Frisinghelli 1 , and Alexander Jacob 1 1 Eurac Research, Institute for Earth Observation, Bolzano, Italy ([email protected]) 2 Free University of Bolzano, Faculty of Science and Technology, Italy R- Package CubeR Click below to explore! Shiny Web Interface Background Integration of in- situ data Outlook Two Minute Madness PICO Presentation at European Geosciences Union General Assembly 2018, Vienna 08-13 April 2018 https://gitlab.inf.unibz.it/earth_observation_public/CubeR

Upload: nguyennguyet

Post on 19-Aug-2019

216 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using a Shiny App and the CubeR Package

Mattia Rossi1,2, Daniel Frisinghelli1, and Alexander Jacob1

1 Eurac Research, Institute for Earth Observation, Bolzano, Italy ([email protected])2 Free University of Bolzano, Faculty of Science and Technology, Italy

R- Package CubeR

Click below to explore!

Shiny Web Interface

Background

Integration of in-situ data

Outlook

Two Minute Madness

PICO Presentation at European Geosciences Union General Assembly 2018, Vienna 08-13 April 2018

https://gitlab.inf.unibz.it/earth_observation_public/CubeR

Page 2: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Oliver, Woodcock (2015). „Australian Geoscience Data Cube - CEOS WGISS 40“

Two Minute Madness IBack to Main Slide

Index

Baumann 2018: “Rasdaman – raster data manager (Version 9.5.0)”; rasdaman.org

During a research with extensive spatial data diverse problems have to be faced:

• Where to Store• Enough Storage• Data management and

standardization• Intermediate Results• Processing Power• Time Consumption

?

Multi-dimensional Arrays (often referred to as Data Cubes)

Principles:• Store Once• Process Once• Use many times• You only get what you need• Direct Processing

Standards:• Coverage Implementation Schema• Web Coverage Service• Web Coverage Processing Service

+ Efficient Data Management+ Scalable+ Processing Power

Page 3: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Background

Two Minute Madness II

1. CubeR Package 2. Shiny Web App

• Create WC(P)S queries • Transform Server Response• Use Case Rasdaman Data Cubes

Back to Main Slide

Main Functionalities• Perform Access • Request Metainformation• Get Products • Perform Calculation

http://saocompute.eurac.edu/rasdaman/owshttps://gitlab.inf.unibz.it/earth_observation_public/CubeR

Combination with an in-situ Database PICO Screen 4.3

Page 4: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Back to Main Slide

Background

Data Cubes?Advantages ofData Cubes ?

Interaction withData Cubes ?

EURACs implementation?

Click the red boxes to explore!

2 Minute Madness

Page 5: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Spatial Data Cubes

Basic Facts about Data Cubes

• A multi-dimensional array significantly bigger than the main memory

• Used to store huge amounts of datasets from one or multiple sensors based on the extent and acquisition time

• Data values of same type sit at grid points

• Defined by d axes of the d-dimensional data cube

• Each grid point has two neighbors in each dimension

Back to Background

Oliver, Woodcock (2015). „Australian Geoscience Data Cube - CEOS WGISS 40“

• Spatio-temporal Information (e.g. Time Series of Raster Images) from one sensor stored in one cube.

• On an easiest scale the cube is three-dimensional (x, y, Time)

• Can increase dimensions (e.g. Bands)

From the Acquisition to a multi-dimensional array (Data Cube):

During a research project intermediate spatial results or final products could be stored together based on time and extent in one array of images

Giuliani et al. (2017) “Building an Earth Observations Data Cube: lessons learned from the Swiss Data Cube (SDC) on generating Analysis Ready Data (ARD)”

Page 6: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Advantages of using Spatial Data Cubes

Lewis et al., 2017: “The Australian Data Cube – Foundations and Lessons Learned”

Schematic Hyperspectral Data Cube

Started mostly for the huge amount of data in hyperspectral images. The storage of several hundred spectral bands and multiple acquisitions over time in data cubes saves time and resources

Behman et al. 2014: “Detection of early plant stress responses in hyperspectral images”

Zhang et al. 2015: “Compression of hyperspectral remote sensing images by tensor approach “

Also the data acquired by multispectral satellites and sensors is augmenting and will continue to rise in the next years.

Storing time series of multispectral data in multi-dimensional arrays provides a powerful opportunity to centralize the storage, distribution and processing of data and to save resources and processing time as well as to increase the reproducibility of a research.Recently many solutions and standards emerged focusing on:

• Efficient Data Management• Scalability• Processing power

Back to Background

Page 7: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Interaction with Data CubesData Cubes are accessed with predefinedstandards as the ones defined by theOGC:

Web Coverage Service (WCS)This service provides standardized access to gridded data represented in the coverage standard (CIS).

Web Coverage Processing Service (WCPS)This is an extension of the WCS standard that adds on the fly processing functionality to coverage retrieval request.

Coverage Implementation Schema (CIS)This standard describes a common schema of representation of collections of values distributed over space and time.

Many Data Cubes offer the possibility to directly interactwith and process data on a Data Cube. An example is theRasdaman DataCube exposing data via WC(P)S:

Get an Image at a given Time

Get a subset of an image

Get a SubsettedImage in a time

interval

Get value of a Pixel over Time

PerformCalculations

Back to Background

Trim Slice

Formula, e.g.

Baumann 2018: “Rasdaman – raster data manager (Version 9.5.0)”; rasdaman.org

Page 8: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

EURAC Implementation

R - Package

Back to Background

CEPH_FS CEPH_RBD

Hardware

File Systems

Cluster Orchestration &

Virtual Environments

Applications

WCS

WCPS

SOS

Data Models &

Data Bases

1.4 PB

2 x 40 Gb/s336 Cores 3 TB RAM

~50 Cores ~200 MB RAM

e.g. pre -processingof S1 / S2

The following chart shows the implementation of the Rasdaman Data Cubes within the computing environment at Eurac Research.

The grey box suggests where the CubeR Package is usable.

Page 9: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Back to Index

R-Package CubeR

The Accessibility of the Rasdaman implementation at EURAC, as provided by using WCPS queries, can be done throughout all programming languages. We focused on developing an R-Package to facilitate the access and to enhance the understanding of EO based Data Cubes for our research center. We named this Package “CubeR”

CubeR consists of 4 main pillars:

Discover Describe

Request/Return

Imports: httr

lubridate

magrittr

ncdf4

raster

sp

stringr

urltools

xml2

Process

Click the red boxes to explore!

Background

https://gitlab.inf.unibz.it/earth_observation_public/CubeR

Page 10: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

DiscoverBack to R-Package

The access to the Rasdaman Database can contain multiple Data Cubes, referred to as „coverages“. The package automatically adds the necessary suffixes to the main Rasdaman URL (e.g. http://saocompute.eurac.edu/rasdaman/ows)

GetCapabilitiesReturns an XML File

getCapabilites()

Coverage1Coverage2...Coverage n

XML Parsing Get the Meta-

information

Return the Data

Capability of coverages stored at the moment*Sensor*_*Level*_*Tile*_*Resolution*

Depending on the type of coverage

Perform mathematical operations

Next Slides

Hosts various Coverages (often referred to as Data Cubes)

Page 11: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Describe

After the access we added core functionalities to receive the metainformation of the coverages

The “Describe Coverage” command requires an XML File for each coverage of interest. We wrapped functions around this XML to return the relevant coverage information such as:

The Coverage is the input for eachMetainformation query(Here a Level-2A Sentinel 2 MSI Coverage)

All Function for findingmetainformation havethe prefix „coverage_get“

Back to R-Package

Page 12: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Request/Return

Central Functions of the Package

• Create WCS queries

• Handle the Server Response

• Return the Tables, Images or Raster

Processing time of the three functions for 170 images (all Sentinel 2 Images of 1 Tile) takes roughly 30s (Depending on the connection to the Server)

Return Image Return Pixel over Time Return Area over Time

Back to R-Package

Baumann 2018: “Rasdaman – raster data manager (Version 9.5.0); rasdaman.org

Page 13: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Process

Shiny App

Direct processing in the Rasdaman Data Cube environment.

For our own scientific goals we implemented a function allowing the computation of Normalized Difference between two bands so far.

NDVI one Image NDVI one Pixel

Products resulting could be gathered into a new Data Cube depending on processing complexity and - time

Back to R-Package

Baumann 2018: “Rasdaman – raster data manager (Version 9.5.0)”; rasdaman.org

Page 14: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Back to Index

Shiny Web Application

Analogous to the Development of the Package we set up a web application based on the Shiny and Leaflet Package for near-real time visualization of requests. We used the functions described in the R-Package beforehand and added responsive panels for the single operations (Discovery, Description, Request, Response and Process) on the Data Cubes as well as a visualization with Leaflet in the Background.

The Shiny App has two main Windows:

Main Interface Pixel Plots

R Package

Click the red boxes to explore!

Page 15: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

The Shiny Main Interface

Back to Shiny Web Application

The Main interface is divided in three main panels:

• Visualization PanelSelecting the Band combination based on the Coverage and a rescale factor (faster computation)

• Calculation PanelBasic Calculation on the Coverage(e.g. NDVI)

• Data Set SelectorSelect Dataset, Spatial Extent and Timestamp of interest

• Based on the Package Functions

Page 16: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

The Shiny Pixel Plots

In situ Integration

Back to Shiny Web Application

The Pixel History tab has also three different Functionalities:

Pixel CoordinatesPanel to insert the exact location of one Pixel

BandsSelect the Bands of interest in base of the coverage selected in the previous window

Normalized DifferenceOn the Cube calculation of the Normalized Difference Indexes

For now the plots are with the base utility since it seems to be the fastest among packages

Page 17: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Outlook

Back to Index

Integration of in-situ data

Shiny App

At Eurac Research we also implemented an SOS based Database for Point observations at fixed stations within the MONALISA Project.

For handling the Data from this Database we created a smaller second package MonalisR aimed at accessing the SOS API directly.

With both these packages it is possible to combine Data Cubes and Station Data information

https://gitlab.inf.unibz.it/earth_observation_public/MonalisR

In Situ NDVI + Sentinel 2 Pixel NDVI Values (MONALISA Station)

Processing time for all images from 2015 – 2017 and 2016 In-situ NDVI in 15min resolution, combination and plot ~5sec (Depending on the connection)

Page 18: Accessing Raster Data in Rasdaman Data Cubes via R Two ...sao.eurac.edu/wp-content/uploads/2018/07/RossiEtAl_EGU2018_PICO_Data... · Accessing Raster Data in Rasdaman Data Cubes via

Accessing Raster Data in Rasdaman Data Cubes via R using CubeR (Rossi, Frisinghelli, Jacob) – [email protected]

Back to Index

Outlook

In situ integration

Future Developments of the Package• The package development is in first stable version (v.1.0)• Further WCPS queries based on the needs of the community/researchers. The base computation of a few spectral indices should be

extended to more difficult mathematical operations• Further use cases within the Sentinel Alpine Observatory research (http://sao.eurac.edu/)

• Vegetation Stress (Drought) detection• Mosaicking

• Highly dependent on the use cases and your interest!

Future Developments in Data Cube Technology at Eurac Research• Organize also Climate Data to easily combine with EO-Data• Closer integration of data and metadata• Discovery looking into “OpenSearch”• Federate and integrate with other Platform

• OpenEO (http://openeo.org/)• Common interface to various other Backends

Thank you for your interest in this PICO Presentation at EGU2018!