the big data lab for interdisciplinary spatially- enabled ... big data lab for interdisciplinary...
TRANSCRIPT
![Page 1: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/1.jpg)
The Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS)
![Page 2: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/2.jpg)
Goal
Setting up a large database of time series of changes in land use for most of the agricultural
areas of the planet
![Page 3: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/3.jpg)
Agenda
●The problem●The challenge●The Solution
![Page 4: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/4.jpg)
The problem
![Page 5: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/5.jpg)
Mato Grosso, Brasil, May 8 – Jun 9, 1984
![Page 6: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/6.jpg)
Mato Grosso, Brasil, Jun 10 – Jul 12, 1985
![Page 7: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/7.jpg)
Mato Grosso, Brasil, Jul 12 – Aug 13, 1986
![Page 8: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/8.jpg)
Mato Grosso, Brasil, May 8 – Jun 9, 1988
![Page 9: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/9.jpg)
Mato Grosso, Brasil, Aug 13 – Sep 14, 1989
![Page 10: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/10.jpg)
Mato Grosso, Brasil, Jul 12 – Aug 13, 1990
![Page 11: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/11.jpg)
Mato Grosso, Brasil, Jul 12 – Aug13, 1991
![Page 12: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/12.jpg)
Mato Grosso, Brasil, Aug 12 – Sep 13, 1992
![Page 13: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/13.jpg)
Mato Grosso, Brasil, Jun 10 – Jul 12, 1993
![Page 14: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/14.jpg)
Mato Grosso, Brasil, Jul 12 – Aug 13, 1994
![Page 15: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/15.jpg)
Mato Grosso, Brasil, Jul 12 – Aug 13, 1995
![Page 16: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/16.jpg)
Mato Grosso, Brasil, Jun 9 – Jul 11, 1996
![Page 17: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/17.jpg)
Mato Grosso, Brasil, Jun 10 – Jul 12, 1997
![Page 18: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/18.jpg)
Mato Grosso, Brasil, Jun 10 – Jul 12, 1998
![Page 19: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/19.jpg)
Mato Grosso, Brasil, Jun 10 – Jul 12, 1999
![Page 20: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/20.jpg)
Mato Grosso, Brasil, Jun 9 – Jul 11, 2000
![Page 21: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/21.jpg)
Mato Grosso, Brasil, Jul 12 – Aug 13, 2001
![Page 22: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/22.jpg)
Mato Grosso, Brasil, Jul 12 – Aug 13, 2003
![Page 23: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/23.jpg)
Mato Grosso, Brasil, Jun 9 – Jul 11, 2004
![Page 24: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/24.jpg)
Mato Grosso, Brasil, Jul 12 – Aug 13, 2005
![Page 25: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/25.jpg)
Mato Grosso, Brasil, May 9 – Jun 10, 2006
![Page 26: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/26.jpg)
Mato Grosso, Brasil, Jun 10 – Jul 12, 2007
![Page 27: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/27.jpg)
Mato Grosso, Brasil, Jun 9 – Jul 11, 2008
![Page 28: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/28.jpg)
Mato Grosso, Brasil, Jul 12 – Aug 13, 2009
![Page 29: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/29.jpg)
Mato Grosso, Brasil, Jun 10 – Jul 12, 2010
![Page 30: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/30.jpg)
“Remote sensing images describe landscape dynamics”
What's in an image?
![Page 31: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/31.jpg)
2010 2011
Deforestation event detection: images and time series
![Page 32: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/32.jpg)
Vegetation index time series
Área 1
Área 2
Área 3
source: Victor Maus (INPE)
Time series analysis of land change
Forest
PastureForest
Forest Agriculture
![Page 33: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/33.jpg)
The data
![Page 34: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/34.jpg)
Earth observation satellites and geosensor webs provide key information about global change…
…but that information needs to be modelled and extracted
![Page 35: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/35.jpg)
EO data is now free…and bigImage source: NASA
Sentinels: 3 Tb/day
![Page 36: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/36.jpg)
Is free data download our answer?
Currently, users download one snapshot at a time
![Page 37: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/37.jpg)
Data Access Hitting a Wall
How do you download a petabyte?You don’t! Move the software to the archive
![Page 38: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/38.jpg)
Landsat/TM (August 2007)
MODIS (November 2007)
How hard is to use MODIS?
Detection of deforestation and degradation in MODIS requires much expertise (low-resolution artifacts)
![Page 39: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/39.jpg)
The challenge
![Page 40: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/40.jpg)
Daily warnings of newly deforested large areas
Real-time Deforestation Monitoring
![Page 41: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/41.jpg)
Evaluation of automated methods in one image only!
Real-time Deforestation Monitoring: how to make progress?
![Page 42: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/42.jpg)
The practices of the research community do not match the needs of the end-users!
Real-time Deforestation Monitoring: how to make progress?
![Page 43: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/43.jpg)
Where we want to get to
Remote visualization and method development
Big data EO management and analysis
40 years of Earth Observation data of land change accessible for analysis and modelling.
![Page 44: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/44.jpg)
30 years of EO experience Powerful analysis engine (R)EO database tech (Terralib)Time series EO analysis
SciDB: innovative DMAS for big
arrays
INPE + IFGI
What we know
![Page 45: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/45.jpg)
What we know we don’t know 1: Data
How to put all EO data together? How to work with different ST resolutions?Different satellites have different calibrationsGeometric and radiometric problems
![Page 46: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/46.jpg)
How to organize scientific data in array databases?How to match data semantics to arrays?What’s the equivalent of transaction? What about concurrency control? How to support worldwide users?
What we know we don’t know 2: databases
![Page 47: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/47.jpg)
What are good tools for space-time modelling of EO data?How to combine time series with spatial statistics?How to do space-time object and event detection? How to develop a library of methods for SciDB-R env?
What we know we don’t know 3: methods
![Page 48: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/48.jpg)
What we know we don’t know 4: applications
How best to use ST EO data for global forest studies?How best to use ST EO data for global food studies?
![Page 49: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/49.jpg)
The technology
![Page 50: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/50.jpg)
Nature
“A few satellites can cover the entire globe, but there needs to be a system in place to ensure their images are readily available to everyone who needs them. Brazil has set an important precedent by making its Earth-observation data available, and the rest of the world should follow suit.”
![Page 51: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/51.jpg)
The technology
![Page 52: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/52.jpg)
R: The lingua franca for data analysis
Database
![Page 53: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/53.jpg)
Array databases: all data from a sensor put together in a single array
Xy
t
result = analysis_function (points in space-time )
y
![Page 54: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/54.jpg)
SciDB Architecture: “shared nothing”
Large data is broken into chunks Distributed server process data in parallel
![Page 55: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/55.jpg)
Chunks
1 1 2
5 8 13
34 55 89
233 377 610
0
3
21
144
1
5
0
3
55 89
377 610
1 2
8 13
34
233
21
144
![Page 56: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/56.jpg)
The Proposed Solution
![Page 57: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/57.jpg)
Software goes where the data is!
SciDB: array database for big scientific data
Free satelliteimages
R: Powerful data analysis methods
Global Land Observatory: describing change in a connected
world
![Page 58: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/58.jpg)
Unique repository of knowledge and data about global land change
40 years of LANDSAT + 12 years of MODIS + SENTINELs + CBERS
Free satelliteimages
Global Land Observatory: describing change in a connected
world
Methods for land change for forestry and
agriculture uses
![Page 59: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/59.jpg)
59
ST arrays allows new questions:Are biofuels replacing food production in Brazil?
source: B. Rudorff, INPE
![Page 60: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/60.jpg)
RFields
![Page 61: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/61.jpg)
Current GIS architecture
Single data source, single data schema, layer-oriented view
![Page 62: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/62.jpg)
Distributed architecture for GIS
![Page 63: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/63.jpg)
Consumer
Broker
Provider(s)
- Catalog of available data sets - Location and access information - Data sets meta-data
I need:Rainfall of the Brazilian Amazon from 1999 till 2005
Data set 1 WGS84 (SciDB) Data set 2 SAD69 (GeoTIFF)
Get Data Set
Data Set
SOA
![Page 64: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/64.jpg)
Huge diversity of Geospatial Data...
What is the data about?What is the data format?Where to find the data?
Data Discovery
![Page 65: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/65.jpg)
ConsumerR Package (RGIS?)
- Creates an abstraction layer between users and data sources
- Provides direct access to geospatial data types (Coverage, Time Series Trajectories)
![Page 66: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/66.jpg)
Broker(RDF Triple Store)
- Manages available data sources and their data sets- Stores data sets meta-data- Provides thematic, temporal and geospatial filters- Links data sets to other repositories (meta data enhancement)- Provides credentials for accessing data sources.
Main Challenge:- Generic vocabulary for describing data sets / data sources
![Page 67: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/67.jpg)
Generic Fields data type:new, add obs,
domain, extent, value, combine, neigh,
apply, select, filter, reducereference systems
Generic Fields data type for Big Spatial Data
● GI representation as arrays
● Use of ADBMS routines to GI (server-side processing)
● Keep existing interfaces to data (R & Terralib, Terraview)
![Page 68: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/68.jpg)
MOD09Q1
250 mts spatial resolution
8 days temporal resolution
4800 x 4800 pixels 3 bands (red, nir, qc) 13 years of data (since
2000) HDF format tiles
![Page 69: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/69.jpg)
MODIS tiles
![Page 70: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/70.jpg)
HDFs
REGION HDFs HDF Size (TB)
Binary size (TB)
Binary size ijt (TB)
Amazon(8 tiles x 46 weeks x 13years)
4784 0.30 2.41 4.81
South America(24 x 46 x 13) 14352 0.91 7.22 14.44World land area(225 x 46 x 13) 134550 8.53 67.67 135.33
![Page 71: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/71.jpg)
Data loading
Export
MODISHDF
SciDB bin Load 1D
Array
InsertRedimApply
3DArray
1.3 min / HDF
0.8 min / HDF● Ubuntu server 12 LTS● Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz● 24 cores● RAM 125 GB
![Page 72: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/72.jpg)
Estimated loading time
REGION HDFs Time
Amazon(8 tiles x 46 weeks x 13years)
4784 2.6 days
South America(24 x 46 x 13)
14352 7.9 daysWorld land area(225 x 46 x 13)
134550 74.2 days
![Page 73: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/73.jpg)
SciDB performance
![Page 74: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/74.jpg)
References
![Page 75: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/75.jpg)
Damien Arvor et al. “Analyzing the agricultural transition in Mato Grosso, Brazil, using satellite-derived indices.” In: Applied Geography 32.2 (2012), pp. 702 –713. ISSN: 0143-6228. DOI : http://dx.doi.org/10.1016/j.apgeog.2011.08.007. URL: http://www.sciencedirect.com/science/article/pii/S0143622811001603.
Robert Battle and Dave Kolas. “Enabling the geospatial Semantic Web with Parliament and GeoSPARQL.” In: Semantic Web 3.4 (2012), pp. 355–370. URL: http://dblp.uni-trier.de/db/journals/semweb/semweb3.html#BattleK12.
J. Beddington. “Food, energy, water and the climate: A perfect storm of global events.” In: Sustainable development UK 9 (2009).
Mark Broich et al. “Time-series analysis of multi-resolution optical imagery for quantifying forest cover loss in Sumatra and Kalimantan, Indonesia.” In: International Journal of Applied Earth Observation and Geoinformation 13.2 (2011), pp. 277 –291. ISSN: 0303-2434. DOI: http://dx.doi.org/10.1016/j.jag.2010.11.004. URL: http://www.sciencedirect.com/science/article/pii/S0303243410001340.
Gilberto Camara et al. “Fields as a Generic Data Type for Big Spatial Data.” In: Steffen Fritz et al. “Highlighting continued uncertainty in global land cover maps for the user community.” In: Environmental Research Letters 6.4 (2011), p. 044005. URL: http://stacks.iop.org/1748-9326/6/i=4/a=044005.
![Page 76: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/76.jpg)
J. Gray et al. “Scientific data management in the coming decade.” In: ACM SIGMOD Record 34.4 (2005), pp. 34–41.
P. Griffiths et al. “A Pixel-Based Landsat Compositing Algorithm for Large Area Land Cover Mapping.” In: Selected Topics in Applied Earth Observations and Remote Sensing, IEEE Journal of 6.5 (2013), pp. 2088–2101. ISSN: 1939-1404. DOI: 10.1109/JSTARS.2012.2228167.
Patrick Griffiths et al. “Using annual time-series of Landsat images to assess the effects of forest restitution in post-socialist Romania.” In: Remote Sensing of Environment 118.0 (2012), pp. 199214. ISSN : 0034-4257. DOI: http : / / dx . doi . org / 10 . 1016 / j . rse . 2011 . 11 . 006. URL: http://www.sciencedirect.com/science/article/pii/S0034425711004019.
M. C. Hansen et al. “High-Resolution Global Maps of 21st-Century Forest Cover Change.” In: Science 342.6160 (2013), pp. 850–853.
Manolis Koubarakis et al. “Building Virtual Earth Observatories Using Ontologies and Linked Geospatial Data.” In: Proceedings of the 6th International Conference on Web Reasoning and Rule Systems. RR’12. Vienna, Austria: Springer-Verlag, 2012, pp. 229–233. ISBN : 978-3-642-33202-9. DOI: 10.1007/978-3-642-33203-6_21. URL: http://dx.doi.org/10.1007/978-3-642-33203-6_21.
![Page 77: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/77.jpg)
J.G. Masek et al. “A Landsat surface reflectance dataset for North America, 1990-2000.” In: Geoscience and Remote Sensing Letters, IEEE 3.1 (2006), pp. 68–72. ISSN: 1545-598X. DOI: 10.1109/LGRS.2005.857030.
Ian McCallum et al. “A spatial comparison of four satellite derived 1km global land cover datasets.” In: International Journal of Applied Earth Observation and Geoinformation 8.4 (2006), pp. 246 –255. ISSN: 0303-2434. DOI: http : / / dx . doi . org / 10 . 1016 / j . jag . 2005 . 12 . 002. URL : http//www.sciencedirect.com/science/article/pii/S0303243405001212.
Edzer Pebesma. “spacetime: Spatio-Temporal Data in R.” In: Journal of Statistical Software 51.7 (Nov. 2012), ISSN : 1548-7660. URL: http://www.jstatsoft.org/v51/i07.
Stephen G. Perz. “Grand Theory and Context-Specificity in the Study of Forest Dynamics: Forest Transition Theory and Other Directions.” In: The Professional Geographer 59.1 (2007), pp. 105–114. ISSN: 1467-9272. DOI : 10.1111/j.1467-9272.2007.00594.x. URL: http://dx.doi.org/10.1111/j.1467-9272.2007.00594.x.
Toshihiro Sakamoto et al. “A crop phenology detection method using time-series {MODIS} data.” In: Remote Sensing of Environment 96.3–4 (2005), pp. 366 –374. ISSN: 0034-4257. DOI: http//dx.doi.org/10.1016/j.rse.2005.03.008. URL: http://www.sciencedirect.com/science/article/pii/S0034425705001057.
![Page 78: The Big Data Lab for Interdisciplinary Spatially- Enabled ... Big Data Lab for Interdisciplinary Spatially-Enabled Science (BLISS) ... Use of ADBMS ... Gilberto Camara et al. “Fields](https://reader036.vdocuments.net/reader036/viewer/2022070609/5ad9973d7f8b9a86378c2f34/html5/thumbnails/78.jpg)
Michael Stonebraker et al. “The architecture of SciDB.” In: 23rd International Conference on Scientific and Statistical Database Management (SSDBM 2011). Ed. by Judith Bayard Cushing, James French, and Shawn Bowers. Vol. 6809. Lecture Notes in Computer Science. Springer, 2011, pp. 1–16.
Armel Thibaut Kaptue Tchuente, Jean-Louis Roujean, and Steven M. De Jong. “Comparison and relative quality assessment of the GLC2000, GLOBCOVER, {MODIS} and {ECOCLIMAP} land cover data sets at the African continental scale.” In: International Journal of Applied Earth Observation and Geoinformation 13.2 (2011), pp. 207 –219. ISSN: 0303-2434. DOI: http://dx.doi.org/10.1016/j . jag . 2010 . 11 . 005. URL: http://www.sciencedirect.com/science/article/piiS0303243410001352.
P. Vitousek et al. “Human domination of Earth’s ecosystems.” In: Science 277 (2007), pp. 494–500.
Xiaoyang Zhang et al. “Monitoring vegetation phenology using {MODIS}.” In: Remote SensingEnvironment 84.3 (2003), pp. 471 –475. ISSN: 0034-4257. DOI: http://dx.doi.org/10.1016/S0034-4257(02)00135-9. URL: http://www.sciencedirect.com/science/article/pii/S0034425702001359.
Zhe Zhu, Curtis E. Woodcock, and Pontus Olofsson. “Continuous monitoring of forest disturbance using all available Landsat imagery.” In: Remote Sensing of Environment 122.0 (2012). Landsat Legacy Special Issue, pp. 75 –91. ISSN : 0034-4257. DOI: http : / / dx . doi . org / 10 . 1016 / jrse . 2011 . 10 . 030.