delivering usable and scalable gis data and services for ...•current joint agricultural weather...

Delivering Usable and Scalable GIS Data and Services for Weather and Climate InformationSudhir Shrestha (National Government Science and Civilian)

Matthew Rosencrans (NOAA Climate Prediction Center)

Adam Hartman (Innovim/ NOAA Climate Prediction Center)

Matt Tisdale (Booz Allen/ NASA Atmospheric Science Data Center)

Dan Pisut (Esri Living Atlas)

Exploiting scalable Imagery and Scientific Data: Weather and Climate Domain: How we go about it with ArcGIS Platform

What are the best practices on working with the scientific data and model output that can be enabled from Research to Operations and what are the current works, challenges and opportunities?

Web Services, Web enabling your data and services?

Current way to serve the data and information product that are consumed by large community of scientists, policy makers and industries?

Enabling the innovation through enabling the Scientific Data?

Interoperability ?

Build once and reuse?

GIS Technology at the Climate Prediction Center

Adam Hartman

Innovim, LLC / NOAA – NWS – NCEP Climate Prediction Center

Matthew Rosencrans

NOAA – NWS – NCEP Climate Prediction Center

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Products with ESRI TechU.S. Hazards OutlookDrought MonitoringDrought PredictionTropical Cyclone forecastsClimate Monitoring

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Extremes in Short-term Climate

https://www.cpc.ncep.noaa.gov/products/predictions/threats/extremesTool.php?var=tmin

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Drought MonitoringWeekly product – USMonthly – North America

Established in 1999, Western Governors Association helped

Range of timescales used5 year impacts out west

Impact of climatological wet and dry seasons also play a role

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Data Consolidation

0.453

0.2260.151

0.1130.057

0.000

0.100

0.200

0.300

0.400

0.500

3 6 9 12 24

Weights

Weight

• How do we consolidate the data?• Scripting is done with Python using

ArcGIS Spatial Analyst

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Drought Prediction

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Applications for Agriculture

Station Blended Satellite & StationStation data Satellite data QC’d station data

Blended gridded data1. Download and parse data

2. Import data into GDB as tables

3. QC data tables

4. Create point feature classes

5. Interpolate point feature classes

6. Contour raster datasets

7. Create maps

8. Forecasters QC and edit maps

9. Send maps to world

1. Download data and import to

GDB

2. QC gridded data

3. Import to GDB and distribute

4. Create maps

5. Forecasters QC/edit data

6. Republish maps to world

• Current Joint Agricultural Weather Facility (JAWF) maps use interpolated station data

- Long processing times required to parse and interpolate station data (20-25 minutes)- Inaccurate representations of meteorological parameters in data-sparse regions- QC of data is done after map creation

- Requires 2-3 forecasters per week- 2-3 days per week editing maps

• New gridded product suite for JAWF- Shorter processing times since data is already in gridded format (12-15 minutes)- Blended satellite and station products- QC of data is done prior to creation of maps

- Requires only one forecaster- A couple hours of editing per week (maybe)

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Imperfect Data Collection

www.fiedler.company

Swift and Cavalieri 1985

HighReflectivity

LowEmissivity

Anomalously LowTemperatures

Human Factor:• Missed/incorrect reports• Reuse of station IDs• Degradation of stations

Satellites:

• Erroneous precipitation measurements

• Unphysicalgradients (e.g. bullseyes)

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Incorporation of Noisy Data

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Challenges

• Forecaster familiarity with ArcMap- Level of familiarity with ArcMap?- How to design a tool to fit that familiarity?

• Workflow and tool design- Performing QC of gridded data

- How to directly interact with raster layers via GUI.- Workarounds - Used polygon and point feature layers.

- Big data = slow/problematic rendering.- To vector (point) from raster (GDB) to save RAM.- Created buffers for point feature layers using Python.

- Accounting for editors’ changes or mistakes.- Centering points added by editors within grid cells of the underlying raster layer.- Creating a form of version control.

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Examples

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Using Points

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Using Polygons

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Conclusion

• Blended satellite and station gridded data improves overall accuracy of measurements

• Upstream data QC reduces processing and editing time- Further reduced by new processing scripts and workflow practices

• New JAWF Gridded Editing Tool facilitates editing of new gridded products

• USDA receives higher-resolution maps with greater accuracy

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

• Reduce the number of MXDs used during map creation.

• Work with ESRI to determine feasibility of developing more direct methods for editing raster layers.

• Add functionality to editing tool that automates calculation of gradients for nested polygons.

N A T I O N A L O C E A N I C A N D A T M O S P H E R I C A D M I N I S T R A T I O N

Improvements

Delivering Usable and Scalable GIS Data and Services for Weather and Climate

Information

NASA Atmospheric Science Data Center (ASDC)

Matthew Tisdale, Booz Allen Hamilton (BAH), matthew.s.tisdale@nasa.gov

DAACs

Earthdata Search

https://search.earthdata.nasa.gov

The NASA Atmospheric Science Data Center (ASDC) at a Glance

https://eosweb.larc.nasa.gov

Hierarchical Data Format (HDF)

Preferred container for NASA EOS data

• Curates more than 300 unique science products and provides data services for over 44 science projects

• 4.65 Petabytes of data, over 58 million files, are in the archive as of October 2017

• Data distributed to over 165,000 customers in 158 countries

Radiation Budget - The radiation budget takes into account the sum of all radiation, transferred in all directions, through the Earth's atmosphere and to and from space. Instruments: CERES

Clouds - A visible aggregate of minute water droplets and/or ice crystals in the atmosphere above the Earth's surface. Instruments: CALIPSO, MISR

Aerosols - Suspension of particles of condensed matter (liquid, solid, or mixed) in a carrier gas (usually air). Instruments: CALIPSO, MISR, SAGE III

Tropospheric Composition - Measurements of chemical constituents in the atmosphere including the major (non-H2O) greenhouse gases (CO2, CH4, O3, N2O). Instruments: MOPITT, TES

Tools Used with NASA Data Products

• Significant increase in the number of users interested in using NASA Earth Science data in a GIS • Per the ACSI survey results for “Top tools used to work with

data”: ArcGIS ranked number 1 at 64%

• GIS is utilized to support the delivery of priority data products, experiment with various geospatial technologies, and expand geospatial capabilities.

2017 ACSI RESULTS – All DAACs

Visualization• Visualize temporal change of a variable• Visualize a variable at any vertical dimension• Visualize flow direction and magnitude variables

ArcGIS Multidimensional Mosaic Dataset IndexingHDF/netCDF/GRIB Data

Warehouses

Publish ArcGIS Image Service Usable by ArcGIS Platform

Utilizing the ArcGIS Platform as an End-to-End Solution for Processing, Analyzing, and Visualizing NASA’s Scientific Data

Raster Functions• On-the-fly Computing

• Image Processing (NDVI, pansharpen, image classification, etc.)

• Raster Calculator (Convert Celsius to Fahrenheit)

• Processes the pixels that are requested• Can be chained and avoid intermediate

results

Mosaic Index

• Aggregate (mosaic) spatial, time, and vertical dimensions

Prediction Of Worldwide Energy Resources (POWER) GIS - current

29

SSE GIS has merged into POWER GIS• SSE-GIS capabilities merged into POWER GIS

(https://power.larc.nasa.gov/)• Includes all available parameters for climatological values• Interactive maps with geospatial visualization tools

• Time series slider and swiping tools, etc.

Mission Partner Driving Enhancements• RETScreen currently developing new web-based software using

new POWER API with geoJSON response. (User base 500k+)

Graphical Data Access• ½ x ½ deg; within 5-7 days of observations• Multiple parameters from FLASHFlux, GMAO, etc. available• Parameters arranged by application community (i.e., renewable

energy, buildings, agroclimatology)• Multiple data output formats

Three Applications (initial version):• Time series at a single point (daily, monthly, up to 30 years*)• Regional times series (limited area)• Global climatology (30 year*)

NASA ASDC ArcGIS Portal

Examples of Variables Available for Initial Release (Daily and Long Term Averages over a 22 year Period):

• Global Horizontal Radiation• Diffuse Radiation• Direct Normal Radiation• Latitude Tilt Radiation• Clear Sky Insolation• Top-of-Atmosphere Insolation• NO-SUN or BLACK Days• Air Temperature• Relative Humidity• Atmospheric Pressure• Earth Skin Temperature• Heating Degree Days Below 18C• Cooling Degree Days Above 18C

ArcGIS Living Atlas

Increasing the discoverability of data, services, maps, and apps

https://asdc-arcgis.larc.nasa.gov/portal

Data Rates Drive System Evolution

Moving to the Cloud

EOSDIS CUMULUS in AWSASDC OpenShift and OpenStack On-Premise

Next Steps

• Determining Best Formats for Cloud Storage (S3/CEPH):• Archive - HDF, NetCDF• Analysis - MRF, other?

• Deploying ArcGIS Enterprise in OpenStack• Integrating services into NASA Common Metadata Repository (CMR), Earthdata Search, and ArcGIS Living Atlas• Adding more data products to ArcGIS• Additional Esri Story Maps and Apps

Delivering Usable and Scalable GIS Data and Services for Weather and Climate

Information

NASA Atmospheric Science Data Center (ASDC)

Matthew Tisdale, Booz Allen Hamilton (BAH), matthew.s.tisdale@nasa.gov

DISCOVER-AQ Proof of Concept

• Deriving Information on Surface conditions from Column and Vertically Resolved Observations Relevant to Air Quality (DISCOVER-AQ)

• NASA began a multi-year airborne field campaign in 2011 to distinguish between pollution high in the atmosphere and that near the surface where people live and breathe.

• Detailed observations of air pollution from the surface up into the atmosphere will help improve the capability of future satellites to monitor air quality around the world.

Story Map Journal

Comparison Template

Web Scene

Addressing ASDC Geospatial User Driven Requirements

Leveraging ArcGIS Platform [Server, Portal Desktop, Pro] to meet guidelines of the White House’s draft Common Framework for Earth-Observation Data (CFEOD) Data-Access Services

Developed one-off “Data Recipes” for how

to extract, transform and load data into

traditional GIS formats (Shapefiles and

GeoTIFFs)

Esri announces support for

HDF/NetCDF/GRIB

Developed proof of concept to determine if

NASA EOS data, in its native formats, can be

used in ArcGIS Platform

Continuously receive

request from users for

data in “GIS format”

• OGC Web Map Service (WMS), OGC Web Map Tile Service (WMTS), OGC Web Coverage Service (WCS), OGC Web Feature Service (WFS)

• Data Access Protocol (DAP), Web Processing Service (WPS)

BEFORE

Geospatial Application Support for Multidimensional DataProof of Concept

• Until recently, GIS applications were frequently unable to read EOS data product files or unable to properly interpret the internal data structures necessary to be visualized or analyzed.

• Many geospatial tools, including ArcGIS, GeoServer, MapServer, and Quantum GIS (QGIS), rely on GDAL, open source translator library, to present a single raster abstract data model to the calling application.

• Developing an extensible GDAL augmentation framework, that can be leveraged by data consumers and producers, to properly interpret EOS data products in GIS applications.

• Project known as GDAL Enhancements for ESDIS (GEE), in support of the Big Earth Data Initiative (BEDI)

EXAMPLEImage Displayed Inverted

MOP03TM.005 (HDF4): Retrieved Surface Temperature Night

Before

After

POWER/SSE GIS Lessons Learned Were Leveraged by New GIS Projects

38

Prediction Of Worldwide Energy Resources (POWER) (https://power.larc.nasa.gov , https://eosweb.nasa.gov/sse )

• COMMUNITIES: Renewable Energy including Solar, Building Energy Efficiency and Sustainability, Agricultural

• OBJECTIVE: Integrate latest NASA-derived environmental data, analysis and modeling for enhanced management of energy production and energy efficiency systems.• Target datasets for Electric Power, Renewable Energy, Energy-Efficient Building Design and Performance, and Biomass Crop Development Sectors

• GOALS: Through partnerships, derive/validate/provide parameters relevant to industry needs, link to decision support, and transition capabilities when possible.

• POWER/SSE continues project, collaborating with the ASDC, to provide direct access to source data sets, add GIS capabilities and web services for updated data products including low latency time series.

• LaRC ASDC stood up “Web services” used to make the application platform and technology independent by following standards (i.e. DAP, WCS, WMS, etc.), promoting interoperability

… - 2014 2014-2016 2016 - ...SSE GIS - ACCESS 36

• HTML Based web applications with manual latitude/longitude input, copy/paste HTML Tables, ASCII Downloads.

• Surface Radiation Budget (SRB) – Solar; GMAO GEOS 4 – Meteorology; 1x1 Spatial Resolution, 1983-2005; 22 Year Climatologies

• End Users requesting higher resolution, near real time daily time series and longer climatological products in formats other than ASCII

SSE Web Apps• Implementing lessons learned and ArcGIS architecture put in

place by A36 to geospatially enable entire POWER data archive for access to growing Applied Science users.

• Increased spatial/temporal resolutions: SRB, CERES FLASHFLUX – Solar, GMAO MERRA-2/GEOS 5.12.4; ½ x ½ Spatial resolution, Near Real Time Daily Time Series, 30 Year Climatological Averages

• (Phase 1) Beta product providing data in ASCII, CSV, geoJSON, NetCDF4, ICASA, GeoTiff and (Phase 2) ArcGIS Image Services

POWER GIS (beta)

MERRA 2

GEOS 5.12.4

SRB 3.0

CERES FLASHFlux2H-3C

Data NotDistributed*

DataStored

DataDistributed

RecurringPython Scripts

Native GriddedModel Data

Server withEnterprise GeoDatabase

WebServer

Users Access OnResponsive Platform

* Data may become available in future depending on resources and user requirements. Icons Source: http://www.flaticon.com (Creative Commons 3.0)

1 32

Native meteorological model data is ingested and processed on a recurring cycle and stored.

A

B

A

Native solar model data is ingested on a recurring cycle and averages greater than daily are processed and stored.

B1

2Data is served to users via OPeNDAP and Esri ArcGIS Server (Image Services, Geoprocesing Services).

3Users can access the data in commercial and custom applications via Desktop, Tablet and smartphone technologies.

NASA POWER Data Processing & Distribution

Delivering Usable and Scalable GIS Data and Services for Weather and Climate Information: Esri Living Atlas of the WorldDan Pisut

dpisut@esri.com

Living Atlas of

the World

Living Atlas Team

Esri Contributor

s

Hosted on ArcGIS Online

Hosted Externally

The Living Atlas of the World receives content and nominations from a variety of sources, including Esri, those hosting services in ArcGIS Online, and those running their own ArcGIS servers.

Contributing data the Living Atlas not only improves the findability of your data, but also creates added value by allowing it to be merged with related data from other sources.

• Live Feeds: real-time updating - National Weather Service layers- Earthquakes- Wildfire

• National Water Model• GLDAS monthly hydrology reanalysis• National Drought Monitor• Sea Surface Temperature• HYCOM• Stream Gauges

• CMIP5 temperature and precipitation• GHCN-M

How?

Analysis

Raster

Image Service

Vector

Map Service

Feature Service

Display

Tiles

Plan out the How, Where, and Who of each service to optimize the output

You may need more than one output to satisfy the demands

Feature Service Map Service Tile Service

Multiple Layers

Pop-Ups

Change Symbology

Geoprocessing

Editing

Hosted

Increasing Performance

Vector

But are you really ready to publish the service?

• How optimized are the fields? Is floating point necessary? Can you convert to an integer? Realistic resolution/tolerance?

• Are there meaningful field aliases?• Did you enable query?• Did you enable editing symbology?• Did you enable copying?

- Copy Feature or Copy Raster geoprocessing tools

• Is the symbology intuitive?

Raster

Increasing Performance

Image Service Tile Service

Multiple Layers

Pop-Ups

Change Symbology

Geoprocessing

Editing

Hosted

Optimizing the Raster

• Does the intended use match the projection?• Is the projected resolution and tolerance realistic?• Enable query or just use a tile service

- Scientific (value) vs cartographic renderers (RGB)

• Have you tried using Raster Templates?- Multiple variables- Pre-build time aggregation- Multiple symbologies

Exploiting scalable Imagery and Scientific Data: Weather and Climate Domain: How we go about it with ArcGIS Platform

What are the best practices on working with the scientific data and model output that can be enabled from Research to Operations and what are the current works, challenges and opportunities?

Web Services, Web enabling your data and services?

Current way to serve the data and information product that are consumed by large community of scientists, policy makers and industries?

Enabling the innovation through enabling the Scientific Data?

Interoperability ?

Build once and reuse?