python for earth
DESCRIPTION
Slides from Python Indonesia meetup http://www.python.or.id/2014/05/kopi-darat-python-indonesia-juni-2014.htmlTRANSCRIPT
© 2014 Mohammad Fadli
Python for Earth
Mohammad Fadli
Days before python (back in college era)
• Using fortran for numerical programming
• Using matlab for matrix operational or data
plotting
• Using excel for data processing
• Using surfer or matlab for plotting
• Using arcmap or or for spatial data processing
• Using delphi for UI
• No database (all textfile matrix)
Create matrix
do j = 1, 3
do i = 1, 3
A(i,j) = real(i)/real(j)
blah.. Blah ..
read text file ..
write text file ..
enddo
enddo
Matlab
• Matrix Operation • a = [ 1 2 3 4 5; 2 3 4 5 6; 3 4 5 6 7; 4 5 6 7 8]
• a(2,5)
Courtesy of Mathworks Inc.
Plot stuff
Courtesy of Golden Software Inc.
Courtesy of Embarcadero Technologies, Inc.
Courtesy of ESRI, Inc.
Hydrodynamics model
Courtesy of TideTech, Inc.
Courtesy of Kementerian Kelautan dan Perikanan
Courtesy of Kementerian Kelautan dan Perikanan
Courtesy of Kementerian Kelautan dan Perikanan
Courtesy of Kementerian Kelautan dan Perikanan
Courtesy van oord nv
• Build model for
pollutant trajectory
Grid Analysis and Display System (GrADS) Courtesy of Weather & Climate Prediction Lab. ITB
Do something different!
• Install freeBSD and runs model on freeBSD
• Install mapserver on freeBSD
• Using grads for plotting
• Using fortran for data processing
• Using php for web (still)
• Parallel computing (not yet)
First time using python ArcGIS ArcPy
Courtesy of ESRI Inc.
Heboh Python (Python Fuss) :p
• A clear and readable syntax.
• Modules can be easily written in C, C++.
• It can be used on all major platforms (Windows,
Linux/Unix, Mac etc.)
• Easy to learn.
• And it is free.
• Python this, python that
Nowadays
• ANUGA for Hydrodynamics model
• EQRM for Earthquakes model
• Python for Climate Data Analysis Tools (CDAT)
• GDAL library for spatial data processing
• Django, Grok, Pylons, web2py (and other tons of
web library) for web
• Psycopg for postgre database connection
• PyMPI for Clusters computing and parallelization
Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker
Science with Python: NumPy, SciPy and Matplotlib Courtesy of Andrew Walker
SciPy: a collection of useful modules
Courtesy of Andrew Walker
• Python enables users to do more and better science.
• Python is a modern, interpreted, object-oriented, open-source
language used in all kinds of software engineering.
• Python is now a robust integration platform for all kinds of atmospheric
sciences work, from data analysis to distributed computing, and
graphical user interfaces to geographical information systems.
• Python has a concise but natural syntax for both arrays and nonarrays,
making programs exceedingly clear and easy to read; as the saying
goes, “Python is executable pseudocode.”
• Include visualization, numerical libraries, interconnection with compiled
and other languages, memory caching, Web services, mobile and
desktop graphical user interface programming, and others.
Another advantages
• Several packages exist that handle file formats used by
atmospheric and oceanic sciences users, including netCDF,
HDF4, HDF-EOS 2, and GRIB 1 and 2 (e.g., via the netCDF4-
Python, UV-CDAT, and PyNIO packages).
• Python can be used as a scripting language for
applications such as Unidata’s Integrated Data Viewer,
the NCAR Command Language (via PyNGL), GrADS (via
OpenGrADS), and Esri’s ArcGIS.
Disadvantages
• To be fair, Python has real disadvantages, including the fact that
pure Python code runs much slower than compiled code, there
are fewer scientific libraries compared to Fortran, and
documentation and support for new science users is relatively
sparse.
• There are tools to overcome the speed penalty, the collection of
scientific libraries is growing, and science support resources are
becoming more robust; nevertheless, these are real issues.
• For most Earth science applications, the strengths of Python
outweigh the weaknesses.
What we are doing with python now?
• Try to runs (and optimize) anuga on 128
core HPC
• Optimize and automate earthquake model
• Create our own spatial analysis framework
using GDAL lib
• Prove what John-Wei said
Flood Research 2013: Domain Pasar Minggu to Manggarai (Segment 1)
• Criteria:
– Major factor only from discharge and
local rainfall
– Homogenous Hydrologic characteristic 27
Manggarai Detail Domain
Katulampa
Courtesy of PT Reasuransi Maipark Indonesia 2013
Model Hidrodinamika 2D AnuGA
ANUGA adalah model hidrodinamika Free and Open Source (FOSS) yang dikembangkan oleh Australian National University dan Geoscience Australian; metode pemecahan persamaan menggunakan finite volume; dapat diunduh dari http://anuga.anu.edu.au/; Salah satu kelebihan ANUGA : dapat meng-input-kan hujan pada domain
Kenapa Anuga: 1. Public Domain 2. Open Source 3. Parallel 4. Finite Volume Shock Capturing
Capability Hydraulic Jump
Anuga for Jakarta Flood 2013 2013 flood simulation
Bukit Duri dan Kampung Melayu Kalibata
Courtesy of PT Reasuransi Maipark Indonesia 2014
Hasil Simulasi Banjir Jakarta
Simulasi Banjir Jakarta 2013 Hadi, dkk., 2013
Input Curah Hujan Spasial, Robert, 2013
Anuga dapat mengakomodasi simulasi banjir dengan input curah hujan spasial, simulasi paralel Dan input debit serta TMA
Courtesy of PT Reasuransi Maipark Indonesia 2014
Courtesy of LAPAN
Courtesy of LAPAN
Rendaman Max. vs Data UAV LAPAN
Courtesy of LAPAN, PT Reasuransi Maipark Indonesia
Courtesy of PT Reasuransi Maipark Indonesia
K.H. Abdullah Syafei Foto: Kantor Berita ANTARA (19/01/2014)
Courtesy of PT Reasuransi Maipark Indonesia
Depo, Bukit Duri (19/01/2014) Foto: Kantor Berita ANTARA
Bukit Duri
Bukit Duri (Depo & SMA 8)
Depo
SMA 8
Depo
Depo
Depo
SMA 8
SMA 8
(16/01/2013) Foto: Kantor Berita ANTARA
(19/01/2014) Foto diambil dari Heli PMI
Courtesy of PT Reasuransi Maipark Indonesia
K.H. Abdullah Syafei, (19/01/2014), Foto: ANTARA
Courtesy of PT Reasuransi Maipark Indonesia
Jembatan Layang Kalibata, 19/1/2014, Foto: ANTARA Courtesy of PT Reasuransi Maipark Indonesia
Jl. Jatinegara barat raya
Depo
Depo
Jatinegara Bukit Duri
Courtesy of PT Reasuransi Maipark Indonesia
Rawajati
Kampung Melayu
Diambil dari helikopter PMI, 16/1/2013
Thunder System
Summary Specification
Nodes 4 nodes
Cores 16 cores (4 cores/node)
Processors (per nodes)
AMD FX (tm) – 8350, 4 Ghz
Memory (per nodes)
8 GB DDR 3, 800 Mhz
Scenario: Spin up scenario for 16.67 hours Domain : Full Jakarta consist of Ciliwung River (Bogor-Pluit)
Even though faster
simulation could be
performed, we have to
notify that low
resolution mesh is used
in this case
Scenario: 15-18 January 2013 (3 days) Domain : Ciliwung River (Pasar Minggu-Manggarai)
“The computation
speed may increase as
the water volume inside
of the domain is
increasing”
Courtesy of WCPL-ITB
National Computational Infrastructure (NCI), Australia
Summary Specification
Nodes 3592 node
Cores 57,472 cores (16 cores/node)
Model Fujitsu Server Primergy Cx 250 S2
Processors 2 x intel Sandy E5-2600 (8-core, 2.6 GHz)
Memory 8 GB DDR 3, 1600 MHz 2395 nodes ->32 GB 1125 nodes -> 64 GB 72 nodes -> 128 GB TOTAL : 158 TB
Harddisk 500 GB 7200 rpm SATA
www.nci.org.au, www.fujitsu.com
Cluster Raijin, NCI
Domain 2 :Jakarta Approximately 10 rivers will be included in our scenario Approximately total mesh element
number: 3.300.940 triangles
Problem in computation time while using
large number of mesh elements could be
tackled by employing large core number
of machine used for simulation
NCI Building Australia National University
Contoh :
Reference
• http://www.johnny-lin.com/pyintro/
• http://jakevdp.github.io/blog/2013/03/23/matplotlib-and-the-future-of-
visualization-in-python/
• http://greenteapress.com/pythonhydro/pythonhydro.html
• http://resources.arcgis.com/en/communities/python/
• http://arcpy.wordpress.com/page/3/
• http://pyaos.johnny-lin.com/?p=1271
• http://www1.gly.bris.ac.uk/~walker/PythonEarthSci/
• http://conferences.ncl.ac.uk/sciprog/Presentations/python_processing_spyder_v1.0.2.
• http://numpy.sourceforge.net/numdoc/HTML/numdoc.htm
• http://arcpy.wordpress.com/
grads
• The Grid Analysis and Display System (GrADS) is an
interactive desktop tool that is used for easy access,
manipulation, and visualization of earth science
data.
• Support binary (stream or sequential), GRIB
(version 1 and 2), NetCDF, HDF (version 4 and 5),
and BUFR (for station data).
• Windows & Unix support