open source gis: explaining the software stack presented by aaron racicot – gis programmer...
TRANSCRIPT
Open Source GIS:
Explaining the Software Stack
Presented by Aaron Racicot – GIS Programmer
[email protected] 17th, 2006
A Citizen of Salmon Nation
Outline
Introduction – Personal and Ecotrust Software Stack – Desktop, Web, DST Raster Tools – GDAL, GRASS/QGIS Vector Tools – OGR, PostGIS,
GRASS/QGIS Where is it all going? Backups – Raster/Vector DST’s,
Installation, and screen shots
Who am I…B.S. Computer
Science
Split Personality
M.S. Environmental Science
Open Source User/Developer
GIS Programmer
Ecotrust - Salmon Nation
Conservation Economy
Why Open Source?
“Free software” is a matter of liberty, not price.
Free Software!
Freedom
Four basic freedoms (The Free Software Foundation)
The freedom to run the program, for any purpose.
The freedom to study how the program works,
and adapt
it to your needs. Access to the source code is a precondition
for this.
The freedom to redistribute copies so you can
help your
neighbor.
The freedom to improve the program, and release
your
improvements to the public, so that the whole
community
benefits. Access to the source code is a precondition for this.
Benefits/Limitations of Open Source
Benefits Software cost = $0 Source code is available and modifiable User and development communities flourish Development cycles are VERY fast
Limitations Total cost is NOT = $0 Many applications are not as polished as the
proprietary counterparts Compatibility with proprietary software can be an
issue
OS GIS – Desktop Applications
GRASS GIS
Software Stacks… Desktop
Generic Stack FOSS4G Stack Grouping
Application QGis, Grass, OSSIM, JUMP, UDig
User Interface
Application Dev. Environment
Eclipse, QT, OpenGL
High Level Utilities GeoTools, PostGIS Data Serving
High Level Scripting Languages
PHP, PERL, Python
Low Level Utilities Shplib, GEOS, OGR/GDAL, PostGIS, R-Statistics, GMT
Data Processing
Low Level Languages
C, C++, Java, Fortran System Software
Operating System
Linux, Darwin, Cygwin
Hardware Drivers
32-64 bit processor drivers
Desktop Software Stack
OS GIS - Server Tools
GRASS GIS
GDAL/OGR
R-Statistics
Software Stacks... Server
Web-based Software Stack
Generic Stack FOSS4G Stack Grouping
Client Side Browser Firefox, Safari, Netscape User Interface
Client Side Scripting
Java Script, Java Applets
~~~~~~~~~~~WEB~~~~~~~~~~~
Server Side Application
Chameleon, Cartoweb3, Custom
Data Serving
High Level Utilities Mapserver
Low Level Utilities Shplib, GEOS, OGR/GDAL, PostGIS, R-
Statistics, GMT
Data Processing
High Level Scripting Languages
PHP, PERL, Python
Low Level Languages
C, C++, Java, Fortran System Software
Operating System/Drivers
Linux, Darwin, Cygwin
Hardware Drivers 32-64 bit processor drivers
Software Stack... DSTs (Ecotrust)
DST Software StackGeneric Stack FOSS4G Stack Grouping
Client Side Browser Firefox, Safari, Netscape User Interface
Client Side Scripting
Chameleon/Custom
~~~~~~~~~~~WEB~~~~~~~~~~~
Server Side Application
Chameleon/Custom Data Serving
High Level Utilities Mapserver, GRASS, PostGIS
Low Level Utilities Shplib, GEOS, R-Statistics, GMT, OGR/GDAL
Data Processing
High Level Scripting Languages
PHP, PERL
Low Level Languages
C, C++, Java, Fortran System Software
Operating System/Drivers
Linux (Fedora)
Hardware Drivers 32 bit processor drivers
Raster Processing
GRASS/GDAL - Data Crunching Very powerful tool set (similar to ArcINFO/ArcGRID) Poor GUI, but getting better Very fast Cross platform
QGIS/JUMP/UDIG – Data Viewing Both Raster and Vector support Nice GUI Cross platform QGIS can integrate with GRASS
GRASS-GDAL
Vector Processing
GRASS/OGR - Data Crunching Very powerful tool set (similar to ArcINFO/ArcGRID) Poor GUI, but getting better Very fast Cross platform
QGIS/JUMP/UDIG – Data Viewing Both Raster and Vector support Nice GUI Cross platform QGIS can integrate with GRASS
GRASS-OGR
Where to go for more infoOSGIS
Maptools - http://www.maptools.org FreeGIS - http://freegis.org/ Open Source GIS - http://opensourcegis.org/
Standards OGC - http://www.opengeospatial.org/
Desktop GRASS - http://grass.itc.it/ QGIS - http://qgis.org/ UDIG - http://udig.refractions.net/confluence/display/UDIG/Home JUMP – http://jump-project.org/ OpenEV - http://openev.sourceforge.net/
Server/Web Mapserver - http://mapserver.gis.umn.edu/ GRASS - http://grass.itc.it/ PostGIS - http://postgis.refractions.net/
Tools Remote Sensing - http://remotesensing.org/tiki-index.php GDAL/OGR - http://gdal.maptools.org/index.html PROJ.4 - http://proj.maptools.org/ R-Statistics - http://www.r-project.org/ GMT - http://gmt.soest.hawaii.edu/
The End
Raster/Vector DST’sInstallation
Tool Screen Shots
Raster DST’s
Raster based DSTSiuslaw Watershed Restoration Initiative
Data Gathering
Data Formatting
Data Processing
PHP Mapscript
GRASS
Raster based DST
$string = “host db_str usr_str”;$connection = pg_connect($string);if (!$connection){ pg_close($connection); echo "Error:Cannot connect to database<br>";} else { echo "Connected to the SCA database <br>\n";}
$res_pg_exec = pg_exec($connection,$exec_str);
PostgreSQL Type Interface for GRASS GIS
Raster based DST – GRASS Class
class Grass_GIS { var $data_dir; // Where the GRASS data
dir is var $rc_dir; // Where the GRASS rc dir is var $bin_dir; // Where the GRASS bin dir
is var $location; var $mapset; function Grass_GIS($data_dir,
$rc_dir, $bin_dir, $location, $mapset){
$this->data_dir = $data_dir; $this->rc_dir = $rc_dir; $this->bin_dir = $bin_dir; $this->location = $location; $this->mapset = $mapset; } function
Grass_GIS_copy($grass_gis){ $this->data_dir = $grass_gis-
>data_dir; $this->rc_dir = $grass_gis->rc_dir; $this->bin_dir = $grass_gis->bin_dir; $this->location = $grass_gis-
>location; $this->mapset = $grass_gis->mapset; } function set_location($location) {$this->location = $location;} function set_mapset($mapset) {$this->mapset = $mapset;} function set_data_dir($data_dir) {$this->data_dir = $data_dir;} function set_rc_dir($rc_dir) {$this->rc_dir = $rc_dir;} function set_bin_dir($bin_dir) {$this->bin_dir = $bin_dir;}
function run_command($cmd){ $std_output = ""; $grass_exec = ""; // Check that all the required info is available.. if ($this->data_dir != NULL && $this->rc_dir != NULL &&
$this->bin_dir != NULL && $this->location != NULL &&$this->mapset != NULL){// Here we will write out the script to run...$fp_data = fopen($this->data_dir."/grass_run.sh","w");// write out text headerfwrite($fp_data,"export GISBASE=".$this->bin_dir." \n");fwrite($fp_data,"export GISDBASE=".$this->data_dir." \n");fwrite($fp_data,"export GISRC=".$this->rc_dir."/.grassrc6 \n");fwrite($fp_data,"export ETC=".$this->bin_dir."/etc \n");fwrite($fp_data,"export PATH=".$this->bin_dir."/bin:".$this->bin_dir."/scripts:".$this->bin_dir. "/garden/bin:\$PATH:/usr/bin:/usr/local/bin/ \n");fwrite($fp_data,"export LOCATION_NAME=".$this->location." \n");fwrite($fp_data,"export MAPSET=".$this->mapset." \n");fwrite($fp_data,"export LOCATION=".$this->data_dir."/".$this->location."/".$mapset." \n");fwrite($fp_data,"\n");fwrite($fp_data,$cmd."\n");fflush($fp_data);fclose($fp_data);$fp_data = fopen($this->rc_dir."/.grassrc6","w");fwrite($fp_data,"GISDBASE: ".$this->data_dir."\n");fwrite($fp_data,"LOCATION_NAME: ".$this->location."\n");fwrite($fp_data,"MAPSET: ".$this->mapset."\n");fwrite($fp_data,"PAINTER: ppm\n");fwrite($fp_data,"MAPLP: stuff.ppm\n");fflush($fp_data);fclose($fp_data);$grass_exec = sprintf("chmod u+x ".$this->data_dir."/grass_run.sh");exec($grass_exec,$std_output,$std_error);
$grass_exec = sprintf($this->data_dir."/grass_run.sh");exec($grass_exec,$std_output,$std_error);
}
Raster based DST – PHP GRASSdefine("GRASSDATA_DIR", "/var/www/html/apps/siuslaw_target/data/GRASSDATA/");$location_name = "OR_siuslaw“;$mapset_name = "land_targets“;$session_grass_dir = $session_tmp_dir."/GRASSDATA";
// Make a Dir to work inecho " Make a directory to work in inside this session<br>";$dir_exec = sprintf("mkdir ".$session_grass_dir);exec($dir_exec,$arr,$err1); $dir_exec = sprintf("mkdir ".$session_grass_dir."/rc");exec($dir_exec,$arr,$err1); $template_path = GRASSDATA_DIR."siuslaw_epa_template";echo " Copying over new GRASS GIS template<br>";$dir_exec = sprintf("cp -R ".$template_path." ".$session_grass_dir."/".$location_name);exec($dir_exec,$arr,$err1); // Make a MAPSETecho " Setting up the siuslaw specific area<br>";$dir_exec = sprintf("mkdir ".$ session_grass_dir."/".$location_name."/".$mapset_name);exec($dir_exec,$arr,$err1); // Copy over the projection info from the PERMANENT areaecho " Setting projections<br>";$dir_exec = sprintf("cp ".$template_path."/PERMANENT/PROJ* ".$ session_grass_dir."/".
$location_name."/".$mapset_name."/");exec($dir_exec,$arr,$err1); $grass_session = new Grass_GIS($session_grass_dir, $session_grass_dir."/rc", "/usr/local/grass-6.0.0", $location_name,
$mapset_name);$grass_session->run_command("g.list type=rast");
Raster based DST – AML<->GRASS
$arr = $grass_session->run_command("g.region rast=owng_grass@PERMANENT"); echo "<b>Starting to process the Siuslaw data</b><br>\n";//AML step//AH2 = (ah_idx + 1) * 10 echo " Creating ah2_idx_grass... anchor habitat priorities <br>\n"; $arr = $grass_session->run_command("r.mapcalc ah2_idx_grass = \(\(ah_idx_grass@PERMANENT+1\)*10\)");//AML step//own_idx = con(owng eq 31, 40,con(owng eq 83,30,con(owng eq 95,10,20))) echo " Creating own_idx_grass... ownership priorities<br>\n"; $arr = $grass_session->run_command("r.mapcalc own_idx_grass = if\(owng_grass@PERMANENT==31,40,". "if\(owng_grass@PERMANENT==83,30,if\(owng_grass@PERMANENT==95,10,20\)\)\)");//AML step//reclass tpa (use slice) and setnull trees younger than 15 years//veg_idx = setnull(age_g lt 15, slice(tpa_g,eqarea,40)) echo " Reclasification of tpa_g... to 0-40<br>\n"; $arr = $grass_session->run_command("r.rescale.eq input=tpa_g_grass@PERMANENT ". "output=tpa_rescale to=0,40"); echo " Creating the veg_idx... trees over ".$tpa_age." years of age<br>\n"; $arr = $grass_session->run_command("r.mapcalc veg_idx_grass = ". "if\(age_g_grass@PERMANENT\>=".$tpa_age.",tpa_rescale,null\(\)\)");//AML step//FING = setnull(rip_stab eq 0, (ah2 + veg_idx + own_idx + ..\infra\rd_d_inv)) echo " Creating FING... the sum of ah2_idx_grass+own_idx_grass+veg_idx_grass+rd_d_inv_grass<br>\n"; $arr = $grass_session->run_command("r.mapcalc fing_grass = ". "if\(rip_stab_grass@PERMANENT==0,null\(\),". "\(ah2_idx_grass+own_idx_grass+veg_idx_grass+rd_d_inv_grass@PERMANENT\)\)")\;
Raster Example – TREESystemMapserver on the front end… GRASS on the back end…
Web-based Query
Raster based DST - Mapfile LAYER NAME "Fishery Rev" STATUS DEFAULT DATA
"./GRASSDATA/CAL_ocean2/FISHERY/cellhd/fishery_rast_rev_zone_8bit" TYPE RASTER PROJECTION "proj=aea" "datum=NAD27" "ellps=clrk66" "lat_1=34" "lat_2=40.5" "lat_0=0" "lon_0=-120" "x_0=0" "y_0=-4000000" "units=m" END END
Vector DST’s
Vector based DST
Mapserver
PHP -> PHPMapscript -> PostGIS
PostGIS
Vector based DST - Database// Now connect to the db$host_str = "host=localhost";$db_str = "dbname=shelf_closure_db";$usr_str = "user=web_user";$string = $host_str." ".$db_str." ".$usr_str;$connection = pg_connect($string);if (!$connection){ pg_close($connection); echo "Error: Cannot connect to the database <br>\n";} else { echo "Connected to the SCA database <br>\n";}
// Create a dynamic VIEW in PostGISecho "<b>* Create postgresql VIEW * </b><BR>\n";$exec_str = sprintf("CREATE VIEW ". $previous_session_name."_".$previous_session_id." AS ". "SELECT bk.the_geom, sum(tr.total_lbs*tb.percent) as total_lbs ". "FROM tow%02d_3kblk_intersect as tb, trawl%02d". " as tr, cablk3km as bk, port as pt ". "where tb.blk_gid=bk.gid ".$species_sql." ".$port_sql." ".$gear_sql. " and tb.tow%02d_id=tr.tow_id2 ". "GROUP BY bk.the_geom HAVING count(tr.tow_id2)>2", $start_year_short,$start_year_short,$start_year_short);$result_pg_exec = pg_exec($connection, $exec_str);if (!$result_pg_exec) { printf("<B>%s</B><BR>\n", pg_errormessage());}
Vector based DST – Dump results// Execute the pgsql2shp to create the outputecho "<B>* Execute pgsql2shp create the shapefile in session directory *</B><br>";$dir_exec = sprintf("/usr/local/pgsql/bin/pgsql2shp -u web_user -f ". $session_tmp_dir."data/target_areas/test_output.shp shelf_closure_db ". "\"SELECT * from ".$previous_session_name."_".$previous_session_id."\"");exec($dir_exec,$arr,$err1); for ($ii=0;$ii<count($arr);$ii++){ echo $arr[$ii]."<br>\n";} // Drop the dynamic table echo "<b>* DROP the postgresql VIEW * </b><BR>\n";$result_pg_exec = pg_exec($connection, "DROP VIEW ". $previous_session_name."_".$previous_session_id);if (!$result_pg_exec) { printf("<B>%s</B><BR>\n", pg_errormessage());}
Vector Example - OCEANSystem
Rockfish – SFA - 2001 Rockfish – SFA - 2003
Shelf Closure mid-2002
AJAX - Prototypefunction gotVariable(varName, varVal, showWait){ // URL to send the AJAX request to var url = 'http://boris.ecotrust.org/gulf_project/core/SCA_Model/contents.php'; // Push the variable onto the array of parms var newVars = new Array(varName,varVal); varStack.push(newVars); // Build the params for the URL request var pars = ""; for (var i=0; i<varStack.length;i++) { if (i == 0) {pars = pars + varStack[i][0] + '=' + varStack[i][1];} else {pars = pars + '&' + varStack[i][0] + '=' + varStack[i][1];} } // Debug text var boldText = document.createElement('b'); var actionText = document.createTextNode(varName + ' :: '); var action2Text = document.createTextNode(varVal); boldText.appendChild(actionText); var breakText = document.createElement('br'); document.getElementById('placeholder_current').appendChild(boldText); document.getElementById('placeholder_current').appendChild(action2Text); document.getElementById('placeholder_current').appendChild(breakText); // Make the AJAX request var myAjax; myAjax = new Ajax.Updater('placeholder_select', url, {method: 'get', parameters: pars,
onLoading: showWaitImage, onComplete: removeWaitImage}); currentStep++;}
AJAX - Prototype$action_array = array("0" => "init_action", "Revenue by Species" => "process_rev_by_species", "Revenue by Gear Type" => "process_rev_by_gear", "Revenue Total" => "process_rev_by_total", "Num Vessel Landings" => "process_num_vessel_landings", "Distinct Vesseles" => "process_distinct_vessels", "Average Rev Per Vessel" => "process_average_rev_per_vessel", "Distinct Fishermen" => "process_distinct_fishermen", "Distinct Species" => "process_distinct_species");
if (isset($http_form_vars["Action"])){$function_ret = $action_array[$http_form_vars["Action"]]($connection, $http_form_vars);}else{ // deal with the initial request echo "<div id=\"placeholder_select\">\n"; $function_ret = $action_array[0]($connection, $http_form_vars); echo "</div>\n"; echo "<div id=\"placeholder_current\">\n"; echo "</div>\n";}
AJAX - Prototypefwrite($fp_data, "palette(rainbow(".count($y_string)."))\n");fwrite($fp_data, "bitmap(\"species_revenue_".$rand_number.".png\", type =\"png256\", height =
2, width = 4, res = 300)\n");fwrite($fp_data,"plot(x,seq(min(".$y_s."),max(".$y_s.")*1.4,length=length(x)),type=\"n\",xl
b=\"Year\",ylab=\"Revenue\")\n");for ($i=1; $i<=count($y_string); $i++) { fwrite($fp_data,"lines(x,y_".$i.",col=".$i.")\n"); if ($i==1) { $fill_col = $i; } else { $fill_col = $fill_col.",".$i; } }fwrite($fp_data,"legend(x=min(x),y=max(".$y_s.")*1.4,c(".$species_s."),fill=c(".$fill_col."),
bg=\"white\", ncol=4)\n");fwrite($fp_data,"title(\"Revenue plot for Species Groups at Port Group ".$port."\")\n");fwrite($fp_data,"dev.off()\n");fflush($fp_data);fclose($fp_data);$dir_exec = "/usr/local/bin/R CMD BATCH species_revenue_".$rand_number.".R";exec($dir_exec,$arr,$err1);echo "<img height=\"400\" width=\"800\" src=\"./SCA_Model/pictures/species_revenue_".
$rand_number.".png\"><br>\n";
GDAL/OGR – Shapefile validityfunction check_valid_shapefile($shape_file_name){ //echo "Checking that shapefile ".$shape_file_name." is valid...<br>\n"; $valid = TRUE;
$duplicate = FALSE; $importance_present = FALSE;
$name_temp = array(); // Here we need to check for the validity of the shapefile $dir_exec = sprintf("/usr/local/bin/ogrinfo -al ".$shape_file_name); exec($dir_exec,$arr1,$err1); for ($ii=count($arr1)-1; $ii>=0; $ii--) { if ((substr_count($arr1[$ii],": Integer") > 0) || (substr_count($arr1[$ii],": Real") > 0) || (substr_count($arr1[$ii],": String") > 0)) { // Push them all onto an array $string_to_chop = strstr($arr1[$ii],":"); $key_word = str_replace($string_to_chop,"",$arr1[$ii]); array_push($name_temp,$key_word); //echo $key_word."<br>\n"; } }
GDAL/OGR – Cont.for ($ii=0; $ii < count($name_temp); $ii++) { for ($iii=$ii+1; $iii < count($name_temp); $iii++) { if ($name_temp[$iii] == $name_temp[$ii]) { // we have a duplicate $duplicate = TRUE; $duplicate_name = $name_temp[$ii]; } if ($name_temp[$ii] == "IMPORTANCE") { $importance_present = TRUE; } } }
if ($duplicate == TRUE) { echo "<br> ERROR: We found a shapefile with duplicate columns named ". $duplicate_name."<br>\n"; $valid = FALSE; } if ($importance_present == FALSE) { echo "<br> ERROR: We found a shapefile named ".$shape_file_name. " without an IMPORTANCE column<br>\n"; $valid = FALSE; }
return $valid;}
Installation
10 Step Tool Install1) First install Fedora Core4 with
updates2) Proj.4tar -xzvf proj-4.4.9.tar.gzcd proj-4.4.9cd nad/cp ../../proj-nad27-1.2.tar.gz .tar -xzvf proj-nad27-1.2.tar.gzcd .../configuremakemake install
3) GDALInstalled the ECW SDK (do a config, make
and install)Installed the MrSid SDK (moved to
/usr/local)wget http://gdal.org/dl/gdal-1.3.1.tar.gztar -xzvf gdal-1.3.1.tar.gzcd gdal-1.3.1./configure --with-ecw --with
mrsid=/usr/local/GeoExpressSDK/make ogr-allmake installwhich gdalinfo4) GEOSwget http://geos.refractions.net/geos-
2.2.1.tar.bz2tar -xjvf geos-2.2.1.tar.bz2cd geos-2.2.1./configuremakemake install
10 Step Tool Install5) POSTGRESQL
tar -xzvf postgresql-8.1.2.tar.gz
cd postgresql-8.1.2
NOTE the LDFLAGS here... this is to support GEOS for postGIS
LDFLAGS=-lstdc++ ./configure --with-perl --prefix=/usr/local/pgsql_8_1_2
gmake
gmake install
Add the shared library path to /etc/ld.so.conf :
/usr/local/pgsql_8_1_2/lib
Run /sbin/ldconfig
Finally add the following to the profile file to make paths available:
PATH=$PATH:/usr/local/pgsql/bin
MANPATH=$MANPATH:/usr/local/pgsql/man
export MANPATH
Add the postgis user
useradd postgis
passwd
[postgis@localhost pgsql]$ postmaster -D ./data/ > ./data/logfile.txt 2>&1 &
6) POSTGISJust make sure that the Postgresql is
configured with the LDFLAGS variable set and that the GEOS software is downloaded and installed from Refractions.
./configure --with-proj --with-geos --with-pgsql
makemake installNow we can create the database:createdb -O aaronr test_dbcreatelang -U aaronr plpgsql test_dbpsql -f
/usr/local/pgsql_8_1_2/share/postgresql/
contrib/lwpostgis.sql -d test_db
10 Step Tool Install7) PHP./configure --prefix=/usr/local/php4 --program-
suffix=4 --enable-force-cgi-redirect --with-config-file-path=/etc/httpd/ --with-gd=/usr/local/ --with-jpeg --with-png --with-tiff --with-zlib --with-freetype-dir --without-ttf --with-mysql --with-regex=system --enable-dbase --enable-dbx --enable-versioning --with-pgsql=/usr/local/pgsql_8_1_2/
makemake installstrip sapi/cgi/phpcp sapi/cgi/php /var/www/cgi-bin/php4cp php.ini-dist /etc/httpd/php.iniMade the following mod to /etc/httpd/php.ini:; Directory in which the loadable extensions
reside.;extension_dir = "./"extension_dir = "/etc/httpd/php_mods"mkdir /etc/httpd/php_modsAdded the following to /etc/httpd/conf/httpd.conf## For PHP scripts as CGI-BINAddType application/x-httpd-php-
cgi .php4 .phtmlAction application/x-httpd-php-cgi /cgi-bin/php4Now re-start the server:/etc/rc.d/init.d/httpd restartStopping httpd:
[ OK ]Starting httpd: [ OK ]
8) MAPSERVER./configure --with-jpeg --with-gd --with-
freetype --with-zlib --with-png --with-pdf --without-tiff --with-proj --with-threads --with-ogr --with-gdal --with-postgis --with-wfs --with-wmsclient --with-wfsclient --enable-debug --with-php=/src/php/php-4.4.2
makecp legend mapserv scalebar /var/www/cgi-
bin/cp mapscript/php3/php_mapscript.so
/etc/httpd/php_mods/9) GRASSCFLAGS="-g -Wall" ./configure --with-
gdal=/usr/local/bin/gdal-config --with-postgres-includes=/usr/local/pgsql/include/ --with-postgres-libs=/usr/local/pgsql/lib/
makemake install10) R-Statisticstar -xzvf R-2.1.1.tar.gzcd R-2.1.1./configuremakemake install
Tool Screen Shots
Backup - GRASS
Backup – R-Statistics
Backup – PostGIS
Geometry
WKT Geometry
Backup - QGIS