grass cloud

34

Click here to load reader

Upload: luca-delucchi

Post on 11-May-2015

1.658 views

Category:

Technology


3 download

DESCRIPTION

The presentation of g.cloud module for GRASS GIS at FOSS4G 2011

TRANSCRIPT

Page 1: Grass cloud

IntroductionIdea

ImplementationConclusions

GRASS GIS in the Cloud

Luca Delucchi, Markus Neteler

Fondazione Edmund Mach – GIS & Remote Sensing Platform

http://gis.cri.fmach.it

FOSS4G 201114th September 2011, Denver (USA)

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 2: Grass cloud

IntroductionIdea

ImplementationConclusions

GRASS GISOur cluster

GRASS GIS

The first Open Source GIS

More than 400modules for everykind of analysis

Interoperability withother software

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 3: Grass cloud

IntroductionIdea

ImplementationConclusions

GRASS GISOur cluster

GRASS GIS

The first Open Source GIS

More than 400modules for everykind of analysis

Interoperability withother software

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 4: Grass cloud

IntroductionIdea

ImplementationConclusions

GRASS GISOur cluster

GRASS GIS

The first Open Source GIS

More than 400modules for everykind of analysis

Interoperability withother software

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 5: Grass cloud

IntroductionIdea

ImplementationConclusions

GRASS GISOur cluster

Our cluster: infrastructure

FEM GIS cluster consists of

300 nodes

610 GB RAM

10 TB disk storage + 15 TB tapebackup

10 Gb/s internal bus

Scientific Linux 5.6

Grid Engine

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 6: Grass cloud

IntroductionIdea

ImplementationConclusions

GRASS GISOur cluster

Our cluster: massive spatial data analysis

MODIS Land Surface Temperature(LST) map gap-filling (13,000 maps)

MODIS data analysis (LST, NDVI,Snow products)

LiDAR data analysis

Solar energy calculations

Neteler et al., 2011: Tiger mosquito, IJHG, doi:10.1186/1476-072X-10-49

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 7: Grass cloud

IntroductionIdea

ImplementationConclusions

Idea to facilitate GRASS GIS usage on a cluster

Simplify the work

Offer the possibility to use Grid Engine by all our colleagues

Launch jobs remotely without connecting every time to thecluster

Offer the possibility to external people to use the clusterwithout learning Grid Engine

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 8: Grass cloud

IntroductionIdea

ImplementationConclusions

Idea to facilitate GRASS GIS usage on a cluster

Simplify the work

Offer the possibility to use Grid Engine by all our colleagues

Launch jobs remotely without connecting every time to thecluster

Offer the possibility to external people to use the clusterwithout learning Grid Engine

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 9: Grass cloud

IntroductionIdea

ImplementationConclusions

Idea to facilitate GRASS GIS usage on a cluster

Simplify the work

Offer the possibility to use Grid Engine by all our colleagues

Launch jobs remotely without connecting every time to thecluster

Offer the possibility to external people to use the clusterwithout learning Grid Engine

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 10: Grass cloud

IntroductionIdea

ImplementationConclusions

Idea to facilitate GRASS GIS usage on a cluster

Simplify the work

Offer the possibility to use Grid Engine by all our colleagues

Launch jobs remotely without connecting every time to thecluster

Offer the possibility to external people to use the clusterwithout learning Grid Engine

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 11: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: requirements

GRASS GIS 7 (current development version)

Grid Engine

Python > 2.4

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 12: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: requirements

GRASS GIS 7 (current development version)

Grid Engine

Python > 2.4

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 13: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: requirements

GRASS GIS 7 (current development version)

Grid Engine

Python > 2.4

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 14: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Required parameters

conf = file with username and password or stdin to connectto the cluster

server = hostname of the cluster

qsub script = file containing qsub script (template)

grass script = own GRASS GIS script

g.cloud conf=/tmp/passwd server=giscluster

qsub script=test launch SGE grassjob.sh grass script=test novariables.sh

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 15: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Required parameters

conf = file with username and password or stdin to connectto the cluster

server = hostname of the cluster

qsub script = file containing qsub script (template)

grass script = own GRASS GIS script

g.cloud conf=/tmp/passwd server=giscluster

qsub script=test launch SGE grassjob.sh grass script=test novariables.sh

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 16: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Required parameters

conf = file with username and password or stdin to connectto the cluster

server = hostname of the cluster

qsub script = file containing qsub script (template)

grass script = own GRASS GIS script

g.cloud conf=/tmp/passwd server=giscluster

qsub script=test launch SGE grassjob.sh grass script=test novariables.sh

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 17: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Required parameters

conf = file with username and password or stdin to connectto the cluster

server = hostname of the cluster

qsub script = file containing qsub script (template)

grass script = own GRASS GIS script

g.cloud conf=/tmp/passwd server=giscluster

qsub script=test launch SGE grassjob.sh grass script=test novariables.sh

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 18: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Required parameters

conf = file with username and password or stdin to connectto the cluster

server = hostname of the cluster

qsub script = file containing qsub script (template)

grass script = own GRASS GIS script

g.cloud conf=/tmp/passwd server=giscluster

qsub script=test launch SGE grassjob.sh grass script=test novariables.sh

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 19: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Optional parameters

raster = name of raster(s)

vector = name of vector(s)

variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”

email = user’s email address

path = path to the folder which must be accessible fromcluster frontend and blades

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 20: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Optional parameters

raster = name of raster(s)

vector = name of vector(s)

variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”

email = user’s email address

path = path to the folder which must be accessible fromcluster frontend and blades

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 21: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Optional parameters

raster = name of raster(s)

vector = name of vector(s)

variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”

email = user’s email address

path = path to the folder which must be accessible fromcluster frontend and blades

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 22: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Optional parameters

raster = name of raster(s)

vector = name of vector(s)

variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”

email = user’s email address

path = path to the folder which must be accessible fromcluster frontend and blades

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 23: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: how does it work?

Optional parameters

raster = name of raster(s)

vector = name of vector(s)

variables = string with name and values of variables statedas Python dictionary “’key’:[’value0’,’value1’]”

email = user’s email address

path = path to the folder which must be accessible fromcluster frontend and blades

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 24: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: more examples

North Carolina sample data

r.texture exampleg.cloud conf=/tmp/passwd server=gisclusterqsub script=test launch SGE grassjob.shgrass script=test onevariables raster.sh variables=“’TEXT’ :[’asm’,’contrast’,’corr,var’,’idm’,’sa’,’se’,’sv’,’entr’,’dv’,’de’,’moc1’,’moc2’]”raster=lsat7 2002 40

r.sun exampleg.cloud conf=/tmp/passwd server=giscluster

qsub script=test launch SGE grassjob.sh grass script=test onevariables sun.sh

variables=“’DOY’ : [1,2,3,4,5,6,7,8,9,10]” raster=elevation

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 25: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: more examples

North Carolina sample data

r.texture exampleg.cloud conf=/tmp/passwd server=gisclusterqsub script=test launch SGE grassjob.shgrass script=test onevariables raster.sh variables=“’TEXT’ :[’asm’,’contrast’,’corr,var’,’idm’,’sa’,’se’,’sv’,’entr’,’dv’,’de’,’moc1’,’moc2’]”raster=lsat7 2002 40

r.sun exampleg.cloud conf=/tmp/passwd server=giscluster

qsub script=test launch SGE grassjob.sh grass script=test onevariables sun.sh

variables=“’DOY’ : [1,2,3,4,5,6,7,8,9,10]” raster=elevation

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 26: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: future

Support for more clustering system (for example Eucalyptus)

Auto installation of GRASS and its dependencies on thecluster if not present (GRASS 7 release as stable version)

Show the status of jobs

Return outputs to the client

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 27: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: future

Support for more clustering system (for example Eucalyptus)

Auto installation of GRASS and its dependencies on thecluster if not present (GRASS 7 release as stable version)

Show the status of jobs

Return outputs to the client

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 28: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: future

Support for more clustering system (for example Eucalyptus)

Auto installation of GRASS and its dependencies on thecluster if not present (GRASS 7 release as stable version)

Show the status of jobs

Return outputs to the client

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 29: Grass cloud

IntroductionIdea

ImplementationConclusions

RequirementsHow does it work?Future

Implementation: future

Support for more clustering system (for example Eucalyptus)

Auto installation of GRASS and its dependencies on thecluster if not present (GRASS 7 release as stable version)

Show the status of jobs

Return outputs to the client

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 30: Grass cloud

IntroductionIdea

ImplementationConclusions

Conclusions

First implementation of a module to run GRASS on a cluster

Experimental version

Please test it and report bugs or improvements

Contact us for more [email protected] - [email protected]

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 31: Grass cloud

IntroductionIdea

ImplementationConclusions

Conclusions

First implementation of a module to run GRASS on a cluster

Experimental version

Please test it and report bugs or improvements

Contact us for more [email protected] - [email protected]

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 32: Grass cloud

IntroductionIdea

ImplementationConclusions

Conclusions

First implementation of a module to run GRASS on a cluster

Experimental version

Please test it and report bugs or improvements

Contact us for more [email protected] - [email protected]

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 33: Grass cloud

IntroductionIdea

ImplementationConclusions

Conclusions

First implementation of a module to run GRASS on a cluster

Experimental version

Please test it and report bugs or improvements

Contact us for more [email protected] - [email protected]

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver

Page 34: Grass cloud

IntroductionIdea

ImplementationConclusions

Conclusions

Thanks for your attention

gis.cri.fmach.it

Luca Delucchi, Markus Neteler FOSS4G 2011, 14th September 2011, Denver