better than free: data explorations with public data and ... · better than free: data explorations...

air • planet • people Explorations with public data & tools

Better than Free: Data Explorations with Public Data

and Software Tools

Introduction to Free Tools

SEA 2016 Tutorial April 6, 2016

Mary Haley

air • planet • people NCAR/CISL/TDD


Quicksurvey(2/2)

Whatso5waretoolsdoyoucurrentlyuse?

- Fortran,C,C++,Java

- ScripBnglanguages:IDL,MATLAB,NCL,Python,R,Ruby,etc.

- GUIs:ncview,panoply

2


Categoriesoftools •  Commandlinetools

•  GUIs

•  ScripBnglanguages

– Pythonpackages

3


Commandlinetools

•  Quickwaytolookatdata•  EasytoperformmulBpleoperaBonsonfile(s)

•  SomewhatlimitedfuncBonality

•  Datafilesmayhavetomeetrequirements

•  Canbeslow

4


Commandlinetoolsforexaminingdata

•  ncdump–NetCDF

•  ncl_filedump–NetCDF,HDF,HDF-EOS,GRIB

•  h5dump–HDF5

•  wgrib/wgrib2–GRIB1/GRIB2

5


NCO–NetCDFOperators(worksonHDF5too)

6


CDO–ClimateDataOperatorsNetCDFandGRIB1/2

7


wgrib/wgrib2

•  wgrib-sliceanddiceGRIB1files

•  wgrib2–“fourdrawersofkitchenutensilsaswellasthemicrowaveandblender”

8


DEMO

ExaminingNetCDFandGRIBfiles

9


GUI-basedtools

10

•  QuickexploraBonoffilecontents•  QuickvisualizaBonofdata

•  CanbeintuiBvetouse

•  MayhavelimitedfuncBonality


ncview–netCDFvisualbrowser

11


Panoply–netcdf,hdf,gribdataviewer(defn:impressivecollec1onofthings)

12


DEMO

panoply-SKIP

13


ScripBnglanguages

•  Powerful–loadedwithfuncBonality•  Havetowritecode;easierthanC/Fortran

•  Canbeslow

•  Canbedifficulttogetsupport

•  Looksgoodonresume!

14


“Pythonispowerful...andfast;playswellwithothers;

runseverywhere;isfriendly&easytolearn;

isOpen.”

15

air • planet • people Explorations with public data & tools 16

www.Bobe.com


hnp://pypl.github.io/PYPL.html

17


OpendatascienceplapormthatempowersthescienBficPythoncommunity

www.conBnuum.io

18


NCAR Command Language

Developed at NCAR, tailored for the analysis and visualization of geoscientific data

1.  Simple, robust file input and output 2.  Hundreds of analysis functions

3.  Publication-quality visualizations

4.  Well-supported, 1000+ examples

5.  Extensive website, training

6.  Now under “conda”


NCLhashundredsofexampleshnp://www.ncl.ucar.edu/ApplicaBons/

20


Pythonpackages

•  There’sLOTSofthem(proandcon)

– Morethanonewaytodosomething

•  WiderangeoffuncBonality

•  EasyinstallaBonvia“pip”and“conda”

•  Lackofunifiedinterfacesandsupport

21


TipIfwriBngyourownpublicPythonpackage:

StyleGuideforPythonCodehnps://www.python.org/dev/peps/pep-0008/

22


PythonpackagesandtoolsforAOS•  cf-python•  ESMPy•  H5py/PyTables•  Iris•  netcdf4-python•  ESMPy•  Pygrib•  PyNIO•  pandas/xarray•  Siphon

•  Cfplot•  matplotlib/basemap•  cartopy•  PyGRaDS•  PyFerret•  PyNGL•  PyAverager/PyReshaper•  seaborn•  RPy

23

hnp://pyaos.johnny-lin.com/


PyNGL-PythoninterfacetoNCL’sgraphicslibrary

PyNIO-PythoninterfacetoNCL’sfileI/Olibrary

PyNIOandPyNGL

PythonmodulesbasedonasubsetofNCL’scapabili1es


PyNIO

•  Providesunifiedinterfacetotheseformats:

– NetCDF-3/NetCDF-4

– HDF4/HDF5

– HDF-EOS2/HDF-EOS5

– GRIB1/GRIB2

– Shapefiles

25


PyNGL

•  Python2Dplosnglibrary(basedonNCL)

•  Highlycustomizable

•  Handlesdataonregularandirregulargrids

•  SpecializedscriptsforMJOCLIVAR,skewT,

windroses,barcharts,overlays

26


•  Python2Dplosnglibrary

•  BasemapprovidesmappinguBlity

•  GeneratesgraphicssimilartoNCL

•  MoremainstreamusagethanNCL

27


xarraybringsthelabeleddatapowerofpandastothephysicalsciences,by

providingN-dimensionalvariantsofthecorepandasdata

28

pandasprovideshigh-performance,easy-to-usedatastructuresanddata

analysis.


PyAOSwebsite

Pythonresourcesforatmosphericandoceanicsciences

hnp://pyaos.johnny-lin.com/

GoodstarBngpoint...


Allthisso5ware!

WhatshouldIuse?!?

30


Depends...•  Toolscanbedomainspecific

•  HowmoBvatedyouare–doyouliketowritecode?

•  Doyouneedquicklookorheavyli5ing?

•  Theoneyouradvisor/bosstoldyouto

31


HelpavailablefromNCARandbeyond

NCLemaillisthnp://mailman.ucar.edu/mailman/lisBnfo/ncl-talk

PyAOSemaillisthnp://lists.johnny-lin.com/lisBnfo.cgi/pyaos-johnny-lin.com


Forthehands-onlab

•  FocuswillbedownloadingdatafromtheRDA

•  UsetoolslikeNCL,panoply,Python(PyNIO,xarray,netcdf4python)toexaminefiles

•  TakeadvantageoftheexperBseintheroom!

33

hnp://www.ncl.ucar.edu/Training/Tutorials/SEA2016/

better than free: data explorations with public data and ... · better than free: data explorations...

Documents