earth system grid (esg) - dkrz · myscript.sh) run with: $ bash myscript.sh or: $ chmod +x...
TRANSCRIPT
Earth System Grid (ESG)
24.02.2012 Estanislao Gonzalez MPI/DKRZ
Usage – a brief introduction
• What is ESG? – ESG@DKRZ
• Creating an Account
• Logging in
• Finding Data/Metadata
• ESG Security – CMIP5
• Downloading Data
• Getting Help
Summary
24.02.2012 2 Estanislao Gonzalez MPI/DKRZ
raw model output (temporary)
/work/ik0555/cmip5/… …/experiments/outdata/…
Data Flow and Archives
• System for managing data from environmental sciences
• Previous version: system serving CMIP3 data @PCMDI (size 36TB) – still running
• Now: federation of systems serving CMIP5 data (Current archive size 940TB, expected 2PB)
• MPI data in ESG 60TB (currently ~26TB) All of it has gone through QC.
• All ESG CMIP5 data is freely accessible to researchers. Some even to all, e.g. MPI data
What is ESG?
24.02.2012 4 Estanislao Gonzalez MPI/DKRZ
What is ESG?
24.02.2012 5 Estanislao Gonzalez MPI/DKRZ
• All Gateways have access to all the metadata i.e. search results are the same across the federation
• All Gateways trust users from other Gateways i.e. only one account required federation-wide
• Each Gateway manages their own data nodes i.e. expect to jump from one to others
• Data is gathered directly from the data nodes i.e. the download URL will be different from that of the Gateway and could be situated in different continents.
What is ESG?
24.02.2012 6 Estanislao Gonzalez MPI/DKRZ
What is ESG? – ESG@DKRZ
24.02.2012 7 Estanislao Gonzalez MPI/DKRZ
Full Text Search
Faceted Search
Help Links
Other Gateways
Summary
Info Area
Navigation Bar
http://ipcc-ar5.dkrz.de
Creating an Account
24.02.2012 8 Estanislao Gonzalez MPI/DKRZ
Creating an Account
24.02.2012 9 Estanislao Gonzalez MPI/DKRZ
Creating an Account
24.02.2012 10 Estanislao Gonzalez MPI/DKRZ
Creating an Account
24.02.2012 11 Estanislao Gonzalez MPI/DKRZ
https://ipcc-ar5.dkrz.de/myopenid/egonzalez
Creating an Account
24.02.2012 12 Estanislao Gonzalez MPI/DKRZ
Logging in
24.02.2012 13 Estanislao Gonzalez MPI/DKRZ
Logging in
24.02.2012 14 Estanislao Gonzalez MPI/DKRZ
Logging in
24.02.2012 15 Estanislao Gonzalez MPI/DKRZ
This means you haven’t follow the link in the confirmation mail. Check your Spam folder!
Finding Data
24.02.2012 16 Estanislao Gonzalez MPI/DKRZ
Finding Data
24.02.2012 17 Estanislao Gonzalez MPI/DKRZ
Finding Data
24.02.2012 18 Estanislao Gonzalez MPI/DKRZ
File Selection
Dataset Title
Finding Data
24.02.2012 19 Estanislao Gonzalez MPI/DKRZ
DRS - Dataset ID
24.02.2012 20 Estanislao Gonzalez MPI/DKRZ
cmip5.output2.MPI-M.MPI-ESM-LR.rcp45.yr.ocnBgchem.Oyr.r1i1p
<project>.<product>.<institution>.<model>. <experiment>.<realm>.<cmor-table>.<ensemble member>
Finding Data
24.02.2012 21 Estanislao Gonzalez MPI/DKRZ
Finding Data
24.02.2012 22 Estanislao Gonzalez MPI/DKRZ
Finding Data
24.02.2012 23 Estanislao Gonzalez MPI/DKRZ
Finding Data
24.02.2012 24 Estanislao Gonzalez MPI/DKRZ
Finding Metadata (CIM)
24.02.2012 25 Estanislao Gonzalez MPI/DKRZ
Finding Metadata (CIM)
24.02.2012 26 Estanislao Gonzalez MPI/DKRZ
Finding Metadata (CIM)
24.02.2012 27 Estanislao Gonzalez MPI/DKRZ
Finding Metadata (CIM)
24.02.2012 28 Estanislao Gonzalez MPI/DKRZ
ESG Security – CMIP5
24.02.2012 29 Estanislao Gonzalez MPI/DKRZ
ESG Security – CMIP5
24.02.2012 30 Estanislao Gonzalez MPI/DKRZ
ESG Security – CMIP5
24.02.2012 31 Estanislao Gonzalez MPI/DKRZ
ESG Security – CMIP5
24.02.2012 32 Estanislao Gonzalez MPI/DKRZ
ESG Security – CMIP5
24.02.2012 33 Estanislao Gonzalez MPI/DKRZ
ESG Security – CMIP5
24.02.2012 34 Estanislao Gonzalez MPI/DKRZ
ESG Security – CMIP5
24.02.2012 35 Estanislao Gonzalez MPI/DKRZ
At this time CMIP5 groups memberships can only be viewed at PCMDI Gateway!
Downloading Data
24.02.2012 36 Estanislao Gonzalez MPI/DKRZ
Downloading Data
24.02.2012 37 Estanislao Gonzalez MPI/DKRZ
File sub-selection
Downloading Data
24.02.2012 38 Estanislao Gonzalez MPI/DKRZ
Downloading Data
24.02.2012 39 Estanislao Gonzalez MPI/DKRZ
To retain the version number it’s better to store the wget script or the dml file!
It’s a Bash script Store as text (e.g. myscript.sh) Run with: $ bash myscript.sh Or: $ chmod +x myscript.sh $ ./myscript.sh NEVER: sh myscript.sh (it’s not an sh script)
It’s a list that can be used with the DML (Data Mover Lite)
It’s a download manager, requires a GUI and at this time it will only be displayed if there are GridFTP endpoints to download from.
Certificates are valid for 72Hs
Downloading Data - alternative
24.02.2012 40 Estanislao Gonzalez MPI/DKRZ
https://verc.enes.org/data/direct-data-access/data/query
• Security status is stored in:
• If accessing via browser – cookie
• If accessing via script – X509 Certificate
•CMIP5 memberships are stored centrally at PCMDI
•It’s common to be moved from one Gateway to another. You might have to input your OpenID more than once, though never your password.
ESG Security
24.02.2012 41 Estanislao Gonzalez MPI/DKRZ
• To get a X509 Certificate: • Download the MyProxyLogon app (needs a gui):
http://ipcc-ar5.dkrz.de/webstart/myProxyLogon/MyProxyLogon-ESG.jar
• Run using the proper parameters as defined in: http://ipcc-ar5.dkrz.de/help/download-help.htm
• Retrieved certificate will be in $HOME/.esg • In case of troubles, just wipe $HOME/.esg and
start anew.
ESG Security
24.02.2012 42 Estanislao Gonzalez MPI/DKRZ
• The IS-ENES vERC page: https://verc.enes.org
• Downloading from CMIP5 Gateways: https://verc.enes.org/help/how-to-./esgf-gateways
• IS-ENES Search Service: https://verc.enes.org/help/how-to-./the-is-enes-data-search
Getting Help
24.02.2012 43 Estanislao Gonzalez MPI/DKRZ
• Getting Started Guide: http://cmip-pcmdi.llnl.gov/cmip5/data_getting_started.html?submenuheader=3
• Everything about downloading data: http://ipcc-ar5.dkrz.de/help/download-help.htm
• Data availability: http://cmip-pcmdi.llnl.gov/cmip5/availability.html
• General Info on CMIP5:http://cmip-pcmdi.llnl.gov/cmip5/
Getting Help
24.02.2012 44 Estanislao Gonzalez MPI/DKRZ
• CMIP5 Helpdesk:
Getting Help
24.02.2012 45 Estanislao Gonzalez MPI/DKRZ
Thank you.
24.02.2012 46 Estanislao Gonzalez MPI/DKRZ