Climate Change

C o p e r n i c u s C l i m a t e C h a n g e S e r v i c e

G l o b a l L a n d a n d M a r i n e O b s e r v a t i o n s

D a t a b a s e P e t e r T h o r n e

ClimateChange • Provision of global land and marine surface meteorological holdings

• Multiple meteorological parameters served together• Data consistency across timescales (sub-daily to monthly)• Data IPR clearly articulated and flagged• Provided under a common data model via the CDS to users

• 4 year service (2017-2021 – so be kind in your expectations!) – Aim is not to solve all issues, but rather to make a significant dent in the

problem.– Service brings together European parties and NOAA NCEI

S e r v i c e r e m i t

Climate Change

Inventory

ClimateChange • Need firstly to ascertain what data exists

• For ICOADS we have archived the sources and individual data decks that underlie

• For land we are sourcing from a range of international, regional and nationally available sources– Underlying data sources at NOAA NCEI have been shared– Several national repositories have been downloaded– Several institutional repositories under consideration– ECMWF MARS archive (basis for reanalysis)

• Only using publically facing sources.• Key: Building upon national, regional and international capabilities

W o r k o u t w h a t d a t a t h e r e i s

ClimateChange

I C O A D S s o u r c e i d e n t i f i e r a n d d e c k l e v e l

Number per 1x1 degree over POR (log scale)

Number by source type over time

Number by variable over time

Similar maps for rejects files

c.300 unique source ids / deck ids inventoried in this manner

Shall be made available online

ClimateChange

1 7 1 , 2 1 4 a p p a r e n t s u b - d a i l y r e c o r d s

Primarily USAF re-release to NOAA NCEI with many more records than in ISD. Huge volume of complex ASCII files

ClimateChange

1 6 2 , 8 9 2 d a i l y ( m a n y d u p l i c a t e s + d a t a i s s u e s )

ClimateChange • Detailed inventory information will go up on service website shortly

(service website will be part of broader climate.copernicus.eudomain)

• We shall be building the data holdings throughout the contract

• If you have data or know of data you think we should consider send us an email …

incoming_data@surfacetemperatures.org

W e m a y n o t h a v e a l l s o u r c e s

Climate Change

H a r m o n i s a t i o n

ClimateChange • Under continual development with three remaining lots under the

same ITT• Should contain all relevant data and metadata and be flexible and

extendable• Table driven EAV model being developed• Consistent with ODB, BUFR, WMDS and ISO 19115 to extent possible• We shall convert sources into a format that is compliant with the

CDM and work with these converted sources

C o m m o n d a t a m o d e l

ClimateChange • Sequential merge of data sources

– Location metadata similarity– Data similarity– Backfill ECVs and periods– Cannot be completely automated

• Start from daily data – build monthly averages from dailies by preference (assures consistency)– Ensure sub-daily consistent

• Use of a consistent identifier approach (investigating use of OSCAR surface ids with WMO – ongoing)

L a n d h a r m o n i s a t i o n

ClimateChange

E x a m p l e o f m e r g i n g s o u r c e s

ClimateChange • Apply quality control

checks (flag don’t delete)

• Assess homogeneity of records to extent possible

L a n d d a t a h a r m o n i s a t i o n

ClimateChange • Harmonisation work builds upon substantive recent work by NOC and

MO– Improved duplicate identification and flagging– Improved handling of mis-positioned reports– Improved tracking handling– Better quality control / quality assurance– Improved metadata

• Revisit and retrieve data to extent possible that was not considered in early releases and may add value for subsequent inclusion

• More details in next talk by Liz Kent

M a r i n e h a r m o n i s a t i o n

Climate Change

D a t a d e l i v e r y

ClimateChange

D i s s e m i n a t i o n

Data shall be served via the C3S CDS to users

User support shall be available

Three supporting one-stop shop documents:• Technical service guide• Land user guide• Marine user guide

ClimateChange • Observed datasets and derived products (incl. individual reanalyses)

come and go– New techniques– New computational capabilities– New insights

All mean that its all but guaranteed that todays state-of-the-art products shall be deprecated 30-50 years hence.• Whereas the fundamental record is forever

– If we muck up stewardship of the fundamental record, it and the associated knowledge disappears forever.

– Robust international sustained stewardship program required– Keep a deep audit trail, flag but don’t remove, do no harm!– Resource for today, tomorrow and many tomorrows to come.

F i n a l l y : t h e p o i n t o f a l l t h i s ?