Download - Combined Precipitation Algorithms – IMERG
Combined Precipitation Algorithms – IMERGGeorge J. Huffman
NASA/GSFC Laboratory for AtmospheresContact: [email protected]
Combined Precipitation Algorithms – IMERGGeorge J. Huffman
NASA/GSFC Laboratory for AtmospheresContact: [email protected]
Animation located in Hurricane Matthew 10-2-16.mp4
Thanks to:Bob AdlerDavid BolvinEric Nelkin
èSensors, algorithms, and archivesè IMERG algorithm and accessèHow good?èFuture prospectsèFinal pointsèAdditional material
Combined Precipitation Algorithms – IMERGGeorge J. Huffman
NASA/GSFC Laboratory for AtmospheresContact: [email protected]
Data Sources (1/3)
The international constellation of “precipitation relevant”• some sensor on the satellite is useful for estimating precipitation
“satellites of opportunity”• the satellites are flown by an agency for its own purposes, and they contribute their
data to the constellation archive
Huffman 10/16
Data Sources (2/3)
Passive microwave p.m. overpass times
• DMSP F08 SSMI was the first “modern” PMW
• we’re now in the “golden age”
• what’s the future hold?
• some satellites drifta lot
• shading indicates precessing TRMM, GPM, Megha-Tropiques
• persistent gap at 00/12 LT
Huffman 10/16http://precip.gsfc.nasa.gov/times_allsat.jpg
Data Sources (3/3)
Sensor types• radar – very
expensive• PMW imagers –
expensive• PMW sounders –
less expensive• geosynchronous
IR (and multi-spectral) – plentiful data
• precipitation gauges – gold standard, but gaps and only over land
“warm”ocean
“warm”land
complexterrain
“cold”ocean
snowy/frozen
“easy” Surface Type “hard”
“w
orst”
Qua
lity
“be
st”
DPR,PRGMI,TMI
AMSR
SSMIMHS
AMSU
TOVS,AIRS
IR
SSMIS
Huffman 10/16
Algorithms
Sensors have specific strengths and weaknessesAlgorithms might or might not do the best job at extracting the information in a sensor’s readingsYou’ve already seen discussions of the algorithms for individual sensors
What is the best way to combine individual sensors?• it’s almost always done using precipitation datasets
– except IR is usually converted from Tb to precip as part of the combination• averages in time/space that use unevenly spaced or sparse data run the risk
of biasesMulti-satellite precip can be
• Climate Data Record (CDR)– homogeneous, generally by neglecting some data– CMAP, GPCP
• High-Resolution Precipitation Product (HRPP)– use “all” the data, homogenous or not– CMORPH, GSMaP, IMERG, TMPA
• although each strives for the other goal as well
Huffman 10/16
Example family tree for Goddard products
The Adjusted GPI (early ‘90’s) led to GPCPGPCP concepts were first used in TRMM, then blended with new multi-satellite conceptsIMERG adds morphing, Kalman smoother, and neural-network IR estimates
TRMM V4,5 3B42
TRMM 3B42RT
Time
thin arrowsdenote heritage TRMM 3B42RTTRMM 3B42RT
CMORPH
IMERG lateIMERG final
IMERG early
GPCP V1SGMAGPI
TRMM V4,5 3B43
GPCP V2,2.1 SGGPCP V1,1.1 1DDGPCP V1,1.1 Pentad
V6,7 TRMM 3B42V6,7 TRMM 3B43
TRMM 3B42RT
GPCP V3
PERSIANN PERSIANN-CCS
KF-CMORPH
Huffman 10/16
Notes on combination algorithms
We consider elimination of bias in the individual input estimates to be critical
• PMM-calibrate all microwave to a single standard • use merged microwave to calibrate IR• calibrate multi-satellite to monthly gauge using large-area averagesCalibrations must be done with both data sets averaged to the same spatial scalefor consistencyData are merged after calibrationCombination schemes are now at the maturity that microwave schemes had about 10 years agoOperational estimates are just starting to be useful over frozen/icy surfacesPassive microwave data too sparse to completely populate a fine time/space grid• need Lagrangian time interpolation • “morphing” is a linear fade algorithm• IR beats interpolated microwave if the nearest orbit is > 90 min. away• actually need “cloud development” algorithm
Huffman 10/16
Archive sites
Once computed, the data sets are held at archive sites• primary – responsible for providing the original data• mirror – archives at other locations to satisfy local requirementsBoth the primary and mirror sites may create value-added products
• reformatted, aggregated or subsetted, etc.Data provenance is always important, particularly for mirror sites and value-added productsIPWG posts 4 tables of primary archive sites for • quasi-global• freely available• long-termprecipitation data sets• organized according to input data types• maintained by George Huffman
see “additional material”
Huffman 10/16
IMERG processing
IMERG is a unified U.S. algorithm that takes advantage of
• Kalman Filter CMORPH (lagrangian time interpolation) – NOAA• PERSIANN with Cloud Classification System (IR) – U.C. Irvine• TMPA (inter-satellite calibration, gauge combination) – NASA• PPS (input data assembly, processing environment) – NASA
The Japanese counterpart is GSMaPInstitutions are shown for module origins, but• package will be an
integrated system• goal is single code system
appropriate for near-realand post-real time
• “the devil is in the details”
Huffman 10/16
GSFC CPCUC Irvine
prototype6
Receive/storeeven-odd IR
files
Import PMW data;grid; calibrate;
combine
Compute even-odd IR files(at CPC)
Compute IRdisplacement vectors
Build IR-PMW precip calibration
IR Image segmentationfeature extraction
patch classificationprecip estimation
Apply Kalman
filter
Build Kalman
filter weights
Forward/backward
propagation
Import mon. gauge; mon. sat.-gauge
combo.;rescale short-intervaldatasets to monthly
Apply climo. cal.RT
Post
-RT
Recalibrateprecip rate
IMERG Data Sets
Multiple runs accommodate different user requirements for latency and accuracy
• “Early” – 5(4) hours (flash flooding)• “Late” – 15(12) hours (crop
forecasting)• “Final” – 3.5(2.5) months (research)Time intervals are half-hourly and monthly (Final only)
0.1° global CED grid • PPS will provide subsetting by
parameter and location• 60°N-S for now
User-oriented services• interactive analysis (Giovanni)• alternate formats (KMZ, KML, TIFF
World files, …)• area averages
Huffman 10/16
Half-hourly data file (Early, Late, Final)
1 [multi-sat.] precipitationCal2 [multi-sat.] precipitationUncal3 [multi-sat. precip] randomError4 [PMW] HQprecipitation5 [PMW] HQprecipSource [identifier]6 [PMW] HQobservationTime7 IRprecipitation8 IRkalmanFilterWeight9 probabilityLiquidPrecipitation [phase]
Monthly data file (Final)
1 [sat.-gauge] precipitation2 [sat.-gauge precip] randomError3 GaugeRelativeWeighting4 probabilityLiquidPrecipitation [phase]
Data Fields from IMERG Test Data (1/4)1430-1500Z 3 April 2014
Huffman 10/16
PMW data collected in the half hour
PMW sensor contributing the data,selected as imager first, then sounder, then closest to center time [PMW] HQprecipSource [identifier]
[PMW] HQprecipSource [identifier] (mm/hr)
GMI
TMIAMSR2
MHSSSMIS
Data Fields from IMERG Test Data (2/4)1430-1500Z 3 April 2014
Huffman 10/16
[PMW] HQobservationTime (min)
IRprecipitation (mm/hr)
PMW sensor observation time after start of half hour
precip from merged geo-IR data
Data Fields from IMERG Test Data (3/4)1430-1500Z 3 April 2014
Huffman 10/16
[multi-sat.] precipitationCal (mm/hr)
“Final” IMERG field: forward and backwardmorphedmicrowave, Kalman filter with IR data; monthly gauge
[multi-sat. precip] randomError (mm/hr)
estimated random error for the multi-satellite precip
Data Fields from IMERG Test Data (4/4)1430-1500Z 3 April 2014
Huffman 10/16
probabilityLiquidPrecipitation [phase] (%)
probabilitythat precipitation phase is liquid; diagnosticcomputed from ancillary data
IRkalmanFilterWeight (%)
weighting of IR in the Kalman filter step
Core Observatory – constellation – “best estimate”
Huffman 10/16
Animation located in GPM_Fleet_IMERG_globe.mp4
PMM home page
New features to improve “discovery” of data and documentation
Huffman 10/16
“get data” hot links all lead to the data access page:
http://pmm.nasa.gov/data-access
GPM data access page
The actual download page lists sources available at GSFC
http://pmm.nasa.gov/data-access/downloads/GPM
Level 0: instrument countsLevel 1: instrument units (Tb) in original swaths
Level 2: geophysical units (precip) in original swaths
Level 3: griddedLevel 4: assimilated data
Huffman 10/16
other Level 3 products (click on arrowhead)
IMERG description, link to documentation
access is by Levels
headers have mouse-over descriptions of column headers
currently available data have the orange arrow
derived datasets are listed separately from re-formatted datasets
To summarize so far,
The inventory of precipitation-relevant satellites varies significantly with time
There are a number of estimates available from both IR and microwave dataCombined-sensor schemes are generally assumed to yield the best overall performance and come in two approaches:• CDR• HRPP
IMERG is a U.S. GPM algorithmThe PMM data access pages demonstrate that data producers are increasingly working to put out data sets in formats that users want
Huffman 10/16
Examples
GPCP V.2.2 SG climatology for 1979-2010 (CDR)
Note ITCZ, dry subtropical highs, mid-latitude storm tracks
Precipitation is concentrated around maritime continent
GPCP V2.2 Precipitation 1979-2010 (mm/d)
Huffman 10/16
Regionally coherent trends do exist• >0.7 mm/d/decade linear trend over 29 years, locally• the pattern appears to be driven by increases in ENSO frequency • data set inhomogeneities require careful examination
Examples (cont.)
Local linear trend in GPCP V.2.1 SG, 1979-2007 (CDR)
Huffman 10/16
Adapted from Kidd; Huffman 11/14
Louisiana, USA, 8-15 August 2016
Examples (cont.)
Near-real-time monitoring for extreme events (1/2)
Adapted from Kidd; Huffman 11/14
3-day heavy rains > 250 mm related to Hurricane Noel produces
• flooding (deduced by hydrologic model running globally in real time)
• landslides (estimated from real-time landslide potential algorithm)
Hispaniola, 1 November 2007, analyzed in real time by Global Hazard System (GHS), Adler and others, NASA funded research
Examples (cont.)
Near-real-time monitoring for extreme events (2/2)
How good?
Precipitation is easy to “measure,” but hard to analyze
• precip is intermittent and non-negative• precip is generated on the microscale• the decorrelation distances and times are short• point values only represent a small area; snapshots only represent a short time• a finite number of samples is a problem
Huffman 10/16
How good? (cont.)
Rainfall for Washington, DC area, July 1994 (in inches)
Convective rain has very short correlation distances - even for a month
Huffman 10/16
3-hoursnapshots
day
5-day
Month
IR vs. PMW (mm/hr) Feb. 2002 30°N-S
How good? (cont.)
At full resolution the correlation of estimated rain is low; averaging over time and spaceimproves the pictureWe provide the fine-scale data sousers get to decide on averaging strategy
Huffman 10/16
How good? (cont.)
Nearly coincident views by 5 sensors southeast of Sri LankaThe offset times from 00Z are given below the algorithm nameThe estimates are related, but differ due to
• time of observation• resolution• sensor/algorithm limitations
Huffman 10/16
How good? (cont.)
The problem of estimating errors for individual grid boxes is challenging
• the theory is not well-developed• necessary data are not always available from the input data sets• a first approximation is available for GPCP, TMPA, and IMERG monthly estimates• a legitimate daily and shorter error scheme is still under development• a full treatment must address the algorithm and sampling sources of error• goals:
- relate random errors across ranges of time/space scales- estimate bias
Huffman 10/16
Huffman 10/16
How good? IMERG V03 Final Run vs. 3B43 for June 2014
Same input satellites, different algorithms, different calibrator
Similar features, but not identical
• features (SPCZ)
• bias (ITCZ)
IMERG Final (mm/d) June 2014
TMPA 3B43 (mm/d) June 2014
MRMS = NOAA Multi-Radar Multi-SensorIMERG better• Wisconsin to
Nebraska • Idaho, Nevada
IMERG worse
• Northern MinnesotaRadar stops just off-shore; satellite doesn’t
How good? Daily 0.25° IMERG V03, 3B42 V7, MRMS for 15 June 2014
Huffman 10/16
[Courtesy J. Wang(SSAI; NASA/GSFC 612)]
Daily IMERG and Pocamoke Fine-Scale Grid, April-August 2014
23 surface gauges in a 6x5 km region near Wallops Island, Virginia
Excellent correlation for most events (warm season)Both over- and under-estimates for largest events
After Kidd; Huffman 10/16[Courtesy J. Tan (UMBC; WFF)] Days after 1 April 2014
Half-hourly IMERG V03 by source vs Pocamoke grid, April 2014 – March 2015
“Violin diagram” for individual sources of the half-hourly IMERG estimates
• width shows relative contribution for each difference bin
GMI is best; AMSR and SSMIS less so
The extra scatter for no-PMW (interpolated) is partly driven by the large number of cases
No-PMW (interpolated) data are competitive with the skill for most of the sensorsThis is pre-launch calibration! the shift to Version 4 should give more consistency
Number of cases
[Courtesy J. Tan (UMBC; GSFC)]
Huffman 10/16
This diagram focusessolely on heavy rain
All sensors are positively biased
• MHS is particularlybiased due to an IMERGerror
• “no PMW” (morphed andIR) is better
• again, low number ofsamples
This is pre-launchcalibration! the shift to Version 4 should givemore consistency
Half-Hourly IMERG V03 vs MRMS Radar/Gauge Product, 2-4 October 2015, South Carolina Floods
Actual accumulations of rain were up to 24”, but IMERG was high by a factor of 2
Huffman 10/16
Number of cases
[Courtesy J. Tan (UMBC; GSFC)]
Future prospects
Combination approaches undergoing vigorous development• to what degree will these be appropriate for or applied to historical data?• gauge data are critical as tie points for best accuracy
Varying amounts of microwave data will be available• despite GPM, LEO constellation will be sparser in the next 5 years• development will continue on GEO-IR schemes to fill gaps
- “cloud development” schemes try to capture pattern changes in GEO- GEO-multi-spectral is getting renewed interest
GPM is focusing attention on the difficult retrievals over icy/frozen surfaces• current proxies will be refined• “high-frequency” algorithms are being pushed
Error estimates will become increasingly important• we have a possible general framework for estimating random errors at arbitrary
time/space averaging volumes• estimates of bias error might fall out of the general framework
The long-term record must be improved to Climate Data Record standards• discontinuities and algorithm uncertainty hamper global change research
Huffman 10/16
Transitioning from Version 3 to Version 4
Version 3 IMERG is available• Final Run from mid-March 2014 to February 2016• Late Run from 7 March 2015• Early Run from 1 April 2015
Early November 2016: Version 4, first-generation GPM-based IMERG archive, March 2014–present
Mid-2017: Version 5 IMERG, March 2014–present
Late 2017: TRMM V.8/GPM V.5 TRMM/GPM-based IMERG archive, 1998–present
Winter 2017-18: Legacy TMPA products retired
Huffman 10/16
Future prospects (cont.)
Bigger data sets keep showing up• subsetting by parameter, location is becoming more standard• major archive sites are driving toward accessibility from anywhere
More non-expert users keep showing up• archives and developers need to provide more support
Publications need to reference data sets (in addition to algorithms)• authors have to be clear on the provenance of the data• Digital Object Identifier (DOI) is definitive• lacking that, identify
- developer / algorithm / version- producer / original dataset / version- archive / any reformatting- any “value-added” originators
Huffman 10/16
Final points
A variety of global precipitation estimates is available
Combination schemes are favored, but no single data set answers every need• trade-off between latency and accuracy
The next five years will see intense development, mostly of combinations• combination schemes have unfinished business• snowfall is still a work in progress
Archives are working to make data more accessible• some datasets are being put out in multiple formats• subsetting is starting to be offeredVersion 4 IMERG addresses a number of issues uncovered in Version 3
Versions will move quickly over the next 18 months
• Currently Version 3• GPM era reprocessed soon in Version 4, then a year from now in Version 5• TRMM-GPM eras reprocessed in Version 5 in mid-2017• TMPA to be run until Winter 2017-18
The future holds some “interesting” challenges, technical and institutionalHuffman 10/16
NASA/GSFC PMM Web Site: http://pmm.nasa.gov/
Contact: [email protected]
Slides: ftp://meso.gsfc.nasa.gov/agnes/huffman/IPWG8_Huffman_training-IMERG_slides.pptx
Movie: http://svs.gsfc.nasa.gov/cgi-bin/details.cgi?aid=4285Huffman 10/16
Animation located in IMERG-LE_1610020230.mp4
IPWG Dataset Tables
Organized according to input data types
• Table 1 – combinations of satellite data, with gauge data• Table 2 – combinations of satellite data• Table 3 – individual satellite data• Table 4 – precipitation gauge analyses
Columns give basic information about the datasets and a pointer to the archive
• algorithm name (with version, where possible)• a high-level list of the input data• the space/time grid on which the data are carried• the areal coverage and start date (and end date where updates are not routine)• how often updates occur• how close to observation time the data set represents• institution and person responsible and a (footnoted) link to the data
Huffman 10/16
APPENDIX: Acronyms and JargonAGPI Adjusted GPIAIRS Advanced IR SounderAMSR Advanced Microwave Scanning Radiometer (Japan)AMSR2 AMSR – 2AMSU Advanced Microwave Sounding UnitCDR Climate Data RecordCMAP CPC Merged Analysis of PrecipitationCMORPH Climate Prediction Center (CPC) Morphing AlgorithmDOI Digital Object IdentifierDMSP U.S. Defense Meteorological Satellite ProgramDPR Dual-frequency Precipitation RadarENSO El Niño/Southern OscillationGEO Geosynchronous Earth Orbit (also, a satellite in GEO)GHS Global Hazard SystemGMI GPM Microwave ImagerGPI Geosynchronous Operational Environmental Satellite (GOES) Precipitation Index GPM Global Precipitation Measurement missionGPCP Global Precipitation Climatology ProjectGSMaP Global Satellite Mapping of Precipitation (Japan)HRPP High-Resolution Precipitation ProductIMERG Integrated Multi-satellitE Retrievals for GPM (used to compute GPM datasets 3IMERGHH and
3IMERGM; runs include Early, Late, and Final)IPWG WMO/CGMS International Precipitation Working GroupIR InfraredITCZ Intertropical Convergence ZoneKF-CMORPH Kalman Filter CMORPHKML, KMZ Keyhole Markup Language, KML with Zip (compression)LEO Low Earth Orbit (also, a satellite in LEO)Level 0,1,2,3,4 0: sensor units, 1: instrument units, 2, 3: geophysical units, 4: assimilated data; 0-2: sensor
footprint locations, 3-4: griddedLT Local Time
Huffman 10/16
APPENDIX: Acronyms and Jargon (cont.)MERRA NASA Modern-Era Retrospective Analysis for Research and ApplicationsMHS Microwave Humidity SounderMRMS Multi-Radar Multi-Sensor precipitation product (NOAA)NASA National Aeronautics and Space AdministrationNOAA National Oceanic and Atmospheric AdministrationPERSIANN Precipitation Estimation from Remotely Sensed Information using Artificial Neural NetworksPERSIANN-CCS PERSIANN-Cloud Classification SystemPMM U.S. NASA Precipitation Measurement MissionsPMW Passive MicrowavePPS U.S. PMM Precipitation Processing SystemPR TRMM Precipitation RadarSG Satellite-Gauge combined precipitation estimate (usually refers to GPCP)SPCZ South Pacific Convergence ZoneSSMI Special Sensor Microwave/ImagerSSMIS Special Sensor Microwave Imager/SounderTb Brightness temperature (usually measured in Kelvin)TIFF Tagged Image File FormatTMI TRMM Microwave ImagerTMPA TRMM Multi-satellite Precipitation Analysis (used to compute TRMM datasets 3B42, 3B43,
3B42RT)TOVS Television-Infrared Operational Satellite (TIROS) Operational Vertical SounderTRMM Tropical Rainfall Measuring Mission V.1, V.2, … Version 1, Version 2, and so on1DD GPCP One-Degree Daily precipitation product3B42 TRMM Plus Other Satellite precipitation product3B42RT TRMM Real-Time VARHQ3B43 TRMM Plus Other Data precipitation product
Huffman 10/16