open–source python tools for environmental data processing

Open–Source Python Tools for Environmental Data Processing, Analysis, and Visualization

Open –Source Python Tools for Environmental Data Processing, Analysis, and Visualization

9/17/2019

Michael J. MurphyEnvironmental Data Analyst/ Data Scientist and Senior Staff Geologist

Terraphase Engineering Inc.

OVERVIEW

9/17/20192

• The Python programming language

• Why Python?

• Python data analysis basics

• Case studies

• Review

PYTHON* IS A MODERN PROGRAMMING LANGUAGE THAT IS ESPECIALLY USEFUL FOR QUANTITATIVEDATA ANALYSIS AND SCIENTIFIC PROGRAMMING

9/17/20193

*Python is named after Monty Python, not Pythonidae.

WHY PYTHON INSTEAD OF EXCEL?

9/17/20194

BASIC PYTHON DATA TOOLS

9/17/20195

• DataFrame table structures • Access, process, analyze, export and visualize data

• Vast collection of functions for array operations• Quantitative analysis

• High-level plotting functions• Can produce publication-quality data visualizations• Works seamlessly with Python data tools such as NumPy, pandas

EXAMPLE: PANDAS DATAFRAMES

9/17/20196

Key Values

EXAMPLE: PANDAS DATAFRAMES

9/17/20197

Original wide-format table:

Melted long-format table:

EXAMPLE: NUMPY OPERATIONS

9/17/20198

Vectorize function:

Draw random values:

Define function to calculate ion balance:

EXAMPLE: PLOTTING WITH MATPLOTLIB

9/17/20199

A very basic example of a Matplotlib plot:

9/17/201910

CASE STUDIES

9/17/201911

CASE STUDY: MACHINE LEARNING ANALYSIS OF VOC DISTRIBUTION INFRACTURED AQUIFER

9/17/201912

ML STUDY PT. 1 – PRINCIPLE COMPONENT ANALYSIS

DATA CONSISTS OF DEPTH, DIP AND STRIKE, VOCS, AND DESCRIPTION

9/17/201913

DATA IS CONVERTED TO A NUMERICAL ARRAY OR MATRIX

9/17/201914

Original data table:

Numerical data in array:

Categorical data as target values:

DATA IS SCALED, THEN DECOMPOSED INTO TWO PRINCIPLE COMPONENTS(EIGENVECTORS)

9/17/201915

Scale data:

Transform data with PCA:

COMPONENT 2 IS PLOTTED AGAINST COMPONENT 1, AND TARGET NAMES ARE ASSIGNED

9/17/201916

COMPONENT 2 IS PLOTTED AGAINST COMPONENT 1, AND TARGET NAMES ARE ASSIGNED

9/17/201917

Target categories show clustering:

9/17/201918

ML STUDY PT. 2 – K-MEANS CLUSTERING

DIP AND VOCS ARE SELECTED FROM SAME DATA AS AN ARRAY

9/17/201919

Same dataset as PCA example:

Select dip and VOC columns as array:

DATA IS SCALED AND A NEW VECTOR CREATED WITH FOUR K-MEANS CLUSTERS

9/17/201920

Create K-means pipeline:

Fit and predict four clusters:

LABELS ARE ADDED, AND EACH CLUSTER LABEL COUNTED

9/17/201921

Original data:

Two new columns with cluster labels:

Counts of cluster labels:

CLUSTERS ARE PLOTTED WITH DEPTH

9/17/201922

Clusters plotted with depth:

THE CLUSTER COUNTS INDICATE THAT A HIGHER PERCENTAGE OF LOW-ANGLE FEATURES HAVE HIGHVOCS

9/17/201923

Count = 71

Count = 30Count = 29

Count = 48

9/17/201924

BOTH UNSUPERVISED METHODS INDICATE CLUSTERING BETWEEN VARIABLES

Count = 71

Count = 30

Count = 29

Count = 48

CASE STUDY: WELL TRANSDUCER DATA PROCESSING AND VISUALIZATION

9/17/201925

DATA FROM A SINGLE TRANSDUCER CONTAINS NEARLY 70,000 DATA POINTS

9/17/201926

Data collected every 1m; ~70,000 records:

SEVERAL LARGE OUTLIERS ARE PRESENT WHERE TRANSDUCER WAS MOVED, ETC.

9/17/201927

Large artificial outliers:

OUTLIERS ARE ITERATIVELY REPLACED WITH INTERPOLATED VALUES

9/17/201928

Interpolate and impute outliers:

THE RESULTING HYDROGRAPH IS MUCH MORE READABLE

9/17/201929

Hydrograph with outliers replaced:

THE PROCESSED DATASET IS THEN RESAMPLED TO THE MEAN HOURLY VALUE

9/17/201930

Resample dataset to the hourly mean:

Data is reduced by ~98% to ~1,200 records:

THE PROCESSED AND REDUCED DATASET CONTAINS THE ESSENTIAL INFORMATION

9/17/201931

Hydrograph of reduced dataset:

BY ITERATIVELY REPLACING ARTIFICIAL OUTLIERS AND REDUCING THE SIZE OF THE DATASET, WHATWOULD TAKE SEVERAL HOURS IN EXCEL IS ACCOMPLISHED IN LESS THAN 1 MINUTE

9/17/201932

REVIEW AND FURTHER TOPICS

9/17/201933

• Python presents a comprehensive and free open-source toolkit for data processing, analysis and visualization– Python libraries, such as NumPy, pandas, Matplotlib, Seaborn, and Jupyter Notebook

– Developed by universities, scientists, independent developers

– Libraries can be used together to process, analyze, and visualize groundwater and environmental data

– More accurate, powerful, and repeatable than Excel, etc.

– Range of applications from simple EDA to complex ML studies

• Python can also be used to interact with other software and codes– FloPy- MODFLOW library

– PHREEQPY- PHREEQC library

– ArcPy, Python API- Interact and script functions within ESRI’s ArcGIS suite

THANK YOU! QUESTIONS?

9/17/201934

Please feel free to contact me at michael.murphy@terraphase.com if you have

any questions we cannot get to, or find me around the conference.

CASE STUDY: EDA TO VISUALIZE METALS DISTRIBUTIONS WITH DEPTH

9/17/201936

• Metals data from soil samples– Potentially contaminated site

– Arsenic is primary COC

– Examine distribution of As with depth to determine possible outliers

• EDA with ‘Seaborn’ Python library, Jupyter Notebooks

– Seaborn: High-level statistical visualization tools built on Matplotlib

– Works seamlessly with pandas DataFrames

– Allows rapid EDA

– Produce publication-quality visualizations

– Jupyter Notebooks: Edit and compile code, view inline plots

9/17/201937

Wide-format tableof metals data:

9/17/201938

Plot all metals against depth:

9/17/201939

9/17/201940

Highlight arsenic results:

9/17/201941

9/17/201942

Box and scatter plot of arsenic vs. depth:

9/17/201943

9/17/201944

Violin and scatter plot of arsenic vs. depth:

9/17/201945

open–source python tools for environmental data processing

Documents

python tools for optimization ted ralphs

building an image processing pipeline with python

python: developing geoprocessing tools · python. using...

software carpentry - python€¦ · software carpentry:...

overview of python web scraping tools

xml processing with perl, python, and php

processing oceanographic data by python libraries numpy

nltk - natural language processing in python

python tools for visual studio: python na microsoftovom...

natural language processing and python

new python-based methods for data processing

python programming - xii. file processing

python xml processing with lxml

ersion technicalspecification - orcina•pipelay and riser...

natural language processing (python)

distributed processing of large graphs in python

why you should use python 3 for text processing · why you...

using python for signal processing and...

activity recognition and energy expenditure estimation ·...

python learning for natural language processing