evaluation of different cloud computing resources … · cloud-computing platform, provided access...

E VA L U AT I O N O F D I F F E R E N T C L O U D C O M P U T I N G R E S O U R C E S F O R L A R G E S C A L E S AT E L L I T E D ATA P R O C E S S I N G

P O T E N T I A L A N D L I M I TAT I O N

D R . T U O N G - T H U Y V U

B I G D ATA

E A R T H O B S E R VAT I O N

S AT E L L I T E I M A G E D ATA

C L O U D C O M P U T I N G

B I G D ATA

Volume Variety Velocity

analysis, capture, data curation, search, sharing, storage, transfer, visualisation, querying,

updating and information privacy.

CHALLENGE

E A R T H O B S E R VAT I O N S AT E L L I T E D ATA

L A N D S AT 5

L A N D S AT 7

L A N D S AT 8

160 MB (compressed) 366 MB (uncompressed)

900 MB (compressed) 1.6 GB (uncompressed)

1 GB (compressed) 2 GB (uncompressed)

1 scene


RapidEye

PlanetLab

Landsat

Sentinel-2

Multi-sensors

GeoEye


RapidEye

PlanetLab

Landsat

Sentinel-2

Multi-sensors

Ground-truth

GeoEye


RapidEye

PlanetLab

Landsat

Sentinel-2

Multi-sensors

Ground-truth

Modelling

GeoEye


RapidEye

PlanetLab

Landsat

Sentinel-2

Multi-sensors

Ground-truth

Modelling

Spatial analysis

….

GeoEye

Building Petabyte Data Production Systems with the NASA Earth Exchange

(http://www.nas.nasa.gov/SC15/demos/demo22.html)

• Production pipeline for Landsat and NASA Global Imagery and Browse Service – 37 steps, 7 petabytes of data, and 670,000 processor hours.

• NEX Downscaled Climate Projections (NEX-DCP30) and NEX Global Daily Downscaled Projections (NEX-GDDP) pipeline – 40 terabytes (TB) of data and 320,000 processor-hours.

• Estimating biomass by counting trees across the United States by using machine learning and computer vision – 40 TB of data/100,000 compute hours.

http://www.nas.nasa.gov/SC15/demos/demo22.html

C L O U D C O M P U T I N G

… a service over the Internet

I N F R A S T R U C T U R E ( I A A S )

P L AT F O R M ( PA A S )

S O F T WA R E ( S A A S )

R E M O T E S E N S I N G D A T A S T O R E D O N T H E C L O U D - C O M P U T I N G P L A T F O R M , P R O V I D E D A C C E S S V I A W E B P O R TA L , D O W N L O A D A B L E

C U R R E N T T R E N D

S O F T W A R E C O M PA N I E S P R O V I D E D P R O C E S S I N G S E R V I C E S O N T H E C L O U D

A G E N C I E S P R O V I D E C L O U D - B A S E D P R O C E S S I N G T O O L S

L A N D S AT O N AW S

D I G I TA L G L O B E

E S R I C L O U D

E N V I C L O U D

E S A C L O U D T O O L B O X

N A S A E A R T H E X C H A N G E

R E M O T E S E N S I N G D A T A S T O R E D O N T H E C L O U D - C O M P U T I N G P L A T F O R M , P R O V I D E D A C C E S S V I A W E B P O R TA L , D O W N L O A D A B L E

C U R R E N T T R E N D

S O F T W A R E C O M PA N I E S P R O V I D E D P R O C E S S I N G S E R V I C E S O N T H E C L O U D

A G E N C I E S P R O V I D E C L O U D - B A S E D P R O C E S S I N G T O O L S

I N C R E A S I N G R E S E A R C H O N S PAT I A L D ATA P R O C E S S I N G S E R V I C E S O N T H E C L O U D

L A N D S AT O N AW S

D I G I TA L G L O B E

E S R I C L O U D

E N V I C L O U D

E S A C L O U D T O O L B O X


The most open platform so far …


• A collaborative compute platform • Data • Tools & Libraries • Models • Disk storage

• Pleiades 1.3 PB total • Sandboxes: lnxsrv105: 163 TB

total and externally accessible sandbox 305TB

D ATA S E T

Landcover Climate Datasets

MODIS Land MODIS Atmosphere

Landsat 5 Landsat 7 Landsat 8

Landsat GLS Web-Enabled Landsat

ASTER TRMM AVHRR

Ecosystem Modeling Datasets DEM Soils Aerial

M O D E L S

Ecosystem Models • Terrestrial Observation and Prediction System (TOPS) North America • Biome-BGC USA • Lund-Potsdam-Jena Dynamic Global Model (LPJ) Global

Climate Models • The Weather Research and Forecasting (WRF) Model Regional

T O O L S

Analytical Tools MATLAB, IDL/ENVI,

R for statistical computing and graphics Numpy and SciPy for Python

Matplotlib for plotting and graphics GNU Octave

Database Management Systems MySQL, SQLite, MongoDB

Data Manipulation Tools and Libraries MODIS Reprojection Tool (MRT)

MODIS Reprojection Tool for Swath (MRTSwath) HDF-EOS To GeoTIFF Conversion Tool (HEG)

The Land Data Operational Products Evaluation (LDOPE) Tools PROJ.4 Cartographic Projections Library

GDAL - Geospatial Data Abstraction Library NetCDF (Network Common Data Form) tools and libraries

….

• a computing platform

• for petabyte-scale scientific analysis and visualisation of geospatial datasets

C O D E E D I T O R E X P L O R E R C L I E N T L I B R A R I E S

web-based IDE for writing and running scripts

a lightweight web app for

exploring our data catalog and running simple

analyses

Python and JavaScript

wrappers around our web API

D ATA S E T

Landsat

Sentinel

MODIS

1m resolution aerial image data US National Agriculture Imagery Program

(NAIP)

Other sub-meter resolution sample images

SRTM 30-m

Landcover MODIS, MERIS

Surface temperature

Climate & Weather

World population

T O O L S

Most of tools you need are here

Image Math operation, convolutions, edge detection, spectral transformation, object-based methods, compositing & mosaicking, etc.

Reducer Image reductions, statistics, raster-to-vector and vector-to-raster conversion, linear regression

Supervised classification Landsat algorithms Sentinel-1 algorithms

Chart Array …..

A C C E S S I B I L I T Y & L I C E N S E

NASA/NDC or OpenID account, need to signup and be approved

NEX sandbox AWS, GEE, Azure, etc.

A C C E S S I B I L I T Y & L I C E N S E

https://earthengine.google.com/signup. Google evaluates the request

Free for research, education, and nonprofit use.

• Can import your own data, download data & images to work with your own tools

• You own your written algorithms and results on Google EE

NASA/NDC or OpenID account, need to signup and be approved

NEX sandbox AWS, GEE, Azure, etc.

• Cloud-computing platforms/resources are currently available

• Google Earth Engine is growing very fast, both basic and advanced remote sensing processing tools are available, free access

• NEX is more than just a computing platform, is a community and knowledge based system

• Cloud-computing platforms/resources are currently available

• Google Earth Engine is growing very fast, both basic and advanced remote sensing processing tools are available, free access

• NEX is more than just a computing platform, is a community and knowledge based system

Ground truth?

Regional, local context?

Users (ready?)

Crowd-2-Cloud

CROWD-SOURCINGREMOTE SENSING

PROCESSING SERVICES ON CLOUD COMPUTING

Training

Validation

Applications

URBAN

AGR ICULTURE

E C O L O G Y

PUBL IC HEALTH

D ISASTER

• Remote sensing processing services on the cloud

• Disaster recovery monitoring • Damage mapping

• Urban green space

• Urban construction monitoring

• Megafauna index

BIG DATA ANALYTICS

• Crowd-sourcing data quality assessment

• Oil palm plantation management

• Health care delivery • Disaster Health

on-demand, scalable processing capabilities efficiently co-located with large image databases rapid deployment of new algorithms sharing, collaborative platform

CLOUD COMPUTING PLATFORM



Promote open resources (data, tools, etc.) NEX and GEE



Potential or limitation depends on user’sOBJECTIVES

TECHNICAL LEVELS





TECHNICAL LEVELS


PaaS and SaaS

potential to deploy to various SaaS on demand




TECHNICAL LEVELS


PaaS and SaaS

potential to deploy to various SaaS on demand

Many other infrastructures, platforms were not discussed here including ESA, AIST (Japan), other national programmes

T H A N K Y O U

evaluation of different cloud computing resources … · cloud-computing platform, provided access...

Documents