Grib2

Wesley Ebisuzaki (wesley.ebisuzaki@noaa.gov)

wgrib2: Wesley Ebisuzaki, Reinoud Bokhorst, Jaakko Hyvätti, Kristian Nilssen, Karl Pfeiffer, Pablo Romero, Manfred Schwarb, Arlindo da Silva, Niklas Sondell, Sergey Varlamov

Outline

what is grib2, new featuresGrADS & grib2converting from wgrib to wgrib2inventory grib2 filesgeolocationread grib2 with wgrib2write grib2 with wgrib2

What is grib2 WMO replacement for grib version 1 Transfer of gridded data between met centers Software to read/write grib2

NCO fortran/C libraries, used by many codes NCAR library, NCL (NCAR command language) ECMWF grib api, used by many codes wgrib2 others: UK? Japan?

Why grib1 -> grib2 Address the limitations of grib1 Time stamp

Only two bytes to store numbers (255 is ok) can combine 2 bytes for 16 bit integer (64k is ok)

Files size limitation Only 128 standard variables, all other are

locally defined Grids only defined to 1/1000 of a degree

New features in grib2 Submessages Improved metadata: time stamps, level

information, variable information Default grid resolution is 1e-6 degrees but

can be specified to be higher *** Selectable compression schemes ***

Gone: kpds(5) .. kpds(7) Grib1: variable, type of level, level # Grib2:

variable: table/version, discipline, parameter category, parameter number

New tables to modify variables, ensemble, bias, error, chemical type, statistical processing, etc

level: most used templates require several #s. wgrib2, GrADS: hide complexity, similar to

grib1

Forecast timing information Grib1: 4 bytes: type, time units, 2 bytes Grib2: size of time stamp is not fixed

No overflows, precise descriptions Means are handled is systematic manner

in wgrib2 notation, monthly means look like “124@6 hour ave(6 hour fcst)” “124@6 hour ave(0-6 hour ave fcst)”

Statistical operations: ave, acc, min, max, rms, etc

Submessagesbash-2.05a$ wgrib2 wafsgfs44.t18z.gribf84.grib21:0:d=2011011218:HGT:70 mb:84 hour fcst:2:5409:d=2011011218:TMP:70 mb:84 hour fcst:3.1:8037:d=2011011218:UGRD:70 mb:84 hour fcst:3.2:8037:d=2011011218:VGRD:70 mb:84 hour fcst:4:16727:d=2011011218:HGT:100 mb:84 hour fcst:5:22349:d=2011011218:TMP:100 mb:84 hour fcst:

Note: 3.1 and 3.2, UGRD and VGRD are in same grib message, share grid description and metadata.GrADS, wgrib2: process by “fields” not by messages.Fortran: extra complexity

Submessages are good for irregular grids. Only need to store one copy of the latitude/longitude values.

Bitmaps & Undefined Values Grib1: uses bitmap for undefined grid value

overhead = 1 bit per grid point Grib2: use bitmap or special value(s)

All packing methods can use bitmap Complex packing can use special values better to use special values if available

GrADS, wgrib2 hides this complexity fortran code: extra complexity

Scan order Decode a grib field – get a list of grid pt

values “scan order” specifies how you fill the grid Grib1: we:sn and we:ns are common Grib2: 16 different orders but 3 are common,

we:sn, we:ns and “Glahn” (NDFD) wgrib2: default is to convert data to we:sn

but other options include we:ns and raw but best to keep in we:sn order

Grib1, Grib2 – storing grid data Store scaled integers Value = (i * 2^E + ref) * 10^D ref = min value of the field so i >= 0 E, D are scaling factors for the field Global model: E = 0, D is specified Mesoscale: E and D are specified ECMWF/wgrib2(default): D = 0, E

determined by # bits wanted

Packing schemes: IEEE IEEE

store as IEEE real numbers Allows full precision files (ex. restart files) No compression Easy to write (no grib library needed) Not supported by NCO software

Simple packing Similar to grib1 packing

Store scaled (positive) integers If biggest scaled integer < 2**N Store integer as N bits No compression relative to grib1 Fast to write and read common

PNG Compress file by “PNG” compression

same png compression as .png files Better packings are available Not recommended

JPEG

Uses JPEG2000 to compress the grid data Best compression for data without

undefineds Slow to write and read Used by NCO

Complex Packing Start with scaled integer, 3 variants

C1: store values (Vi) C2: store increments (Vi - Vi-1) C3: store increments of increments

Split into multiple regions each region stores packed integer with

varying minimum value and size

Complex Packing

Heuristic methods to determine regions Some programs pack more efficiently Best to use special values rather than bitmap

Undefined encoded as a run of N “special values” Reading complex packed files is fast. Similar

to grib1/simple packing.

Complex Packing: file size

C1/C2/C3 is best for noisy/somewhat noisy/smooth fields

Can produce better complex packing by making C1,C2,C3 and choosing the best (C0)

C0 produces smaller files than JPEG for files with undefined values

C0 typically 10%-20% bigger than jpeg

Examples: file size

1/14/2011 gfs.t00z.sfluxgrbf06.grib2 grib1: 213 MB jpeg: 65.1 MB C0: 65.9 MB *

50.7 MB 1/14/2001 gfs.t00z.prgb2f06

jpeg 55.1 MB C0: 56.5 MB * 36.0 MB 1/14/2011 nam.t00z.edasf06.tm

grib1: 114 MB jpeg: 38.1 MB C0: 34.1 * 24.8 MB * use best of jpeg, C1, C2, C3

Compression: summary JPEG slow to write and read Complex packing is fast to read Complex packing is smaller for files with

undefineds otherwise usually 10-20% bigger

Tapes: best compression (jpeg+C0) Speed: C0 Multitasking routines: jpeg <-> C0..C3

Grib2: common tasks Display with GrADS Inventory Slicing and dicing files Exporting data Averaging Making regional subsets Making grib2 content: conversion, new data

Grib2 & GrADS

Use GrADS v2, g2ctl Supports ensemble Supports minutes Supports 3 time mapping schemes

-0, -b, -verf (use “”)

GrADS: time dimension Mapping grib time to GrADS time dimension.

:d=2009060500:APCP:surface:324-330 hour acc fcst:d=2009060500 analysis time (-0)d=2009060500+324 hours start of verf period

(-b)d=2009060500+330 hours end of verf period

(“”)

d=2009060500:UGRD:1000 mb:330 hour fcstd=2009060500 analysis time (-0)d=2009060500+330 hours verf time (-b or “”)

GrADS examples

Reference, analysis time: Use -0g2ctl -0 ipvhnl.gdas.%y4%m2%d2%h2.grb2

>anl.ctlgribmap -0 -i anl.ctl

GrADS examples

start of verification time period: Use -b optiong2ctl -b gfs.t12z.pgrb2.f%f3 >fcst.ctlgribmap -b -i fcst.ctl must be gribmap v2

end of verification time period: No extra optionsg2ctl gfs.t12z.pgrb2.f%f3 >fcst.ctlgribmap -i fcst.ctl must be gribmap v2

GrADS examples(forecast) ensemble:

files: gep01.t00z.pgrb2cf*gep02.t00z.pgrb2cf*gep03.t00z.pgrb2cf*

g2ctl -ens “01,02,03” gep%e.t00z.pgrb2cf%f3>ens.ctl

gribmap -i ens.ctl

Can use -b in the above example.

Introduction to wgrib2 wgrib2 was planned

Easy to add modules People donated modules

Geolocation Name <-> parameter routines Update of tables from NCO web pages mysql export netcdf export

wgrib2 changes Each option corresponds to a subroutine call

Order is important command line can look like a small program

Many options, to view wgrib2 will list main options wgrib2 -help all will list all options

Functionality Geolocation, if, for, conversion to other formats,

writing grib2, interpolation, multitasking

Inventory Simple inventory:

wgrib2 -s GRB_FILE wgrib2 GRB_FILE -s is assumed, no need -

4yr Verbose inventory

wgrib2 -V GRB_FILE -s and -V are inventory options

Inventory Inventory has changed

old names are mostly the same new levels, old levels are expanded years are 4 digits long time stamps can be more complicated NO KPDS5, KPDS6, KPDS7 or equiv g2lib info

Slicing and Dicing Grib1: wgrib IN -s -4yr | egrep (...) | wgrib IN -i -grib -o

OUT Grib2: wgrib2 IN | egrep (...) | wgrib2 IN -i -grib OUT -4yr, -s are gone, now 4 digit year codes Notice: -grib -o OUT becomes -grib OUT Minor changes to scripts but will need to debug Watch out for scripts that write binary/ieee/text files

default is we:sn rather than raw. (we:ns in global)

Efficiency: -match, -not wgrib2 IN | egrep A | egrep -v B | wgrib2 -i

IN ... A, B are regular expressions 4 processes, read file up to 2x wgrib2 IN -match A -not B ... multiple -match and -not options More efficient: 1 read and 1 process

Efficiency: multiple write wgrib2 IN -match ':TMP:' -grib ht.grb wgrib2 IN -match ':HGT:' -append -grib ht.grb wgrib2 IN -match ':RH:' -grib rh.grb OR with one command line

wgrib2 IN -if ':TMP:' -grib ht.grb \ -if ':HGT:' -grib ht.grb \ -if ':RH:' -grib rh.grb

-if is ended by i/o operation or -fi

Geolocation

A need to know the values at specified lat-lon.A need to know the lat-lon of the grid points.

-lon LON LAT grid value nearest to lat-lon -ijlat I J values and lat-lon of grid point (i,j)-lola ARGS makes lola file with NN interpolation-csv FILE CSV with lat-lon values-spread FILE CSV with lat-lon values

wgrib2: exporting data Many output formats

ieee -ieee OUTPUT.bin big endian IEEE bin -bin OUTPUT.bin machine (IEEE) text -text OUTPUT.txt csv -spread OUTPUT.txt comma sep. values csv -csv OUTPUT.txt another CSV mysql -mysql ... to mysql netcdf -netcdf ... write netcdf file

Exporting to fortran Use -bin OUT.bin (binary)

Default is -header open(...,form='unformatted',access='sequential') If you use -no_header option, get direct access file. open(..,recl=nx*ny*4,form='unformatted',access='di

rect') Scan order is we:sn by default, use -order raw for

original order

Exporting to fortran Order and number of the records can change! Be prepared for potential changes Specify the records that you want (and count) Sort the inventory before decoding to order wgrib2 IN | egrep “:(UGRD|VGRD):(2|3)00 mb”

| sort -t: -k3,3 -k4,4 -k5,5 -k6,6 | wgrib2 IN -i -ieee sorted.bin

Only 1 record to read?

call system('grep ":DPT:surface:60 hour fcst:" ds.td.bin.inv | wgrib2 -i ds.td.bin -bin fort.20',status) if (status /= 0) stop 8 open(unit=20,file='fort.20',form='unformatted',access='sequential') read(20) rdata close(20)

AveragingUse wgrib2: -ave dt GRIB.OUT dt = N(time unit), ex 6hr, 1dyThe records have to be in sequence

d=2011010100:UGRD:1000 mb:anl:d=2011010106:UGRD:1000 mb:anl:d=2011010112:UGRD:1000 mb:anl:d=2011010118:UGRD:1000 mb:anl:

wgrib2 IN | sort -t: -k4,4 -k5,5 -k6,6 -k3,3 | wgrib2 -i IN -set_grib_type c3 -ave 6hr AVE_OUT

Regional Subsets -small_grib lonE:lonW latS:latN OUT

Cookie cutter of existing grid -ijsmall_grib i1:i2 j1:j2 OUT

Cookie cutter of existing grid (i,j) coordinates -new_grid A B C OUT

Uses IPOLATES like copygb A, B, C are new grid definition

-new_grid

subset of copygb calls NCEP iplib library not all functionality of iplib is supported -new_grid is optional, not in opn version

using -new_gridType of interpolation-new_grid_interpolation TYPE bilinear, neighbor, budget etcInterpolation parameters-new_grid_iopts I:J:K iopts(i) arrayType of winds (a must)-new_grid_winds TYPE earth, grid

Notes: bilinear interpolation is defaultMust set the type of winds (earth/grid)

-new_grid: example-new_grid latlon lon0:nlon:dlon lat0:nlat:dlat OUT.grb

wgrib2 IN -new_grid_winds earth -new_grid latlon 0:360:1 90:181:-1 OUT

Some NCEP grids are built-in:wgrib2 IN -new_grid_winds -new_grid ncep grid 3 OUT

input projections: latlon, mercator, polar, lambert, gaussianoutput projections: latlon, gaussian, lambert, nps, sps

RPN Calculator

Why a calculator? CSV: change units from Kelvin to Celcius Net flux = downward flux – upward flux wind speed = sqrt(u*u + v*v) (3-6 hour prate) = 2*(0-6 hour prate) – (0-3

hour prate) growing degree days 500-1000 mb thickness = hgt(500mb) –

hgt(1000mb)

RPN calculator old fashioned, stack based calculator 10 registers 0..9, stack is 10 deep works on grids operators

+,-,*,/,pop,exc,trig,sqrt,log,exp,etc ex. convert from K to C: -rpn “273.15:+”

Saving calculator resultsThe calculation replaces the grid data read from the grib file. The new data can be written out with the standard wgrib2 output operators except for grib data, you have to use -grib_out.

Trivial example: changing sign of LHTFL

wgrib2 -match “:LHTFL:” -rpn “-” -grib_out neg_lhtfl.grb

calculating wind speed

wgrib2 IN.grb -match GRD \ -if “:UGRD:500 mb:” -rpn sto_1 -fi \ -if “:VGRD:500 mb:” -rpn sto_2 -fi \ -if_reg 1:2 -rpn “rcl_1:sq:rcl_2:sq:+:sqrt:clr_1:” \ -set_var WIND -grib_type jpeg \ -grib_out out.grb

Changing metadata previous slide: -set_var WIND many options to alter the metadata -set_var HGT -set_date 2011010212 -set_ftime “12 hour fcst” -set_lev “200 mb” -set_metadata FILE

Writing out new grib2 file Select compression -set_grib_type (jpeg|s|c1|c2|c3) Compress data and write new grib message -grib_out NEW.grb Examples of combining -rpn, -set_* and -

grib_out to write a new grib message are on web site

Writing grib: -import_bin -import_bin allows over write the grid data

wgrib2 T.GRB -import_bin NEW.bin -grib_out NEW.grb -import_bin, reads a binary file and overwrite data -grib_out writes out a new grib2 file useless example – same variable/time/level change metadata by: -set_date, -set_var, -set_lev, -

set_ftime, etc

Conversion Scripts GrADS can read many file formats such as

binary, grib, netcdf, hdf If GrADS can read the file, can convert the

data into grib2 by the g2grb.gs script Used in converting MERRA to grib2 Limitations: only lat-lon grids supported,

Lambert/polar converted to lat-lon grid

Summary Grib2 is a more complicated format than grib1

Compression Fixes limitations of grib1 Some grib2 fields have no grib1 versions

Conversion from wgrib to wgrib2 is straight forward Take advantage of the new features in wgrib2 export data to csv, netcdf, mysql

Summary: can use wgrib2 to write grib2 files

-rpn calculator -import_bin import binary data use -set options to modify metadata -grib_out to write grib files

multi-processing can be done (see web site)

RLE: run length encoding

store as N < 255 discrete IEEE values radar data (Japan) Works well in this application Limited use

Grib1 Conversion Scripts Everybody has a grib1->grib2 program

cnvgrib, ECMWF api, grib1to2.pl Grib1 -> grib2 (used for CFSR conversion)

wgrib -> ieee file wgrib + perl -> metadata file cnvgrib -> grib2 template wgrib2 + ieee + metadata + template = grib2 file 228 lines (1 cpu version), 278 lines (multi-cpu)

Multiprocessing: simple example

Parallel operations wgrib2 can be used for cpu intensive

operations such as compression and regional subsetting.

On the CCS, a 30 sec job only took 8 seconds when run on 4 copies of wgrib2

-for_n I:J:K option uses do-loop syntax to select the fields to process.

-for_n/-for can also be used to select blocks to process -for 10:20 will process msgs 10-20

Example: multicpu -> jpeg# 1-cpu: wgrib2 IN -set_grib_type jpeg -grib_out OUTmkfifo pipe1 pipe2 pipe3wgrib2 IN -for_n 1::3 -set_grib_type jpeg -grib_out pipe1 &wgrib2 IN -for_n 2::3 -set_grib_type jpeg -grib_out pipe2 &wgrib2 IN -for_n 3::3 -set_grib_type jpeg -grib_out pipe3 &gmerge OUT pipe1 pipe2 pipe3IN = input grib file, OUT = output grib file