new workflow: getting started with sms & ned · 2020. 5. 12. · basic workflow terms sms - the...
TRANSCRIPT
![Page 1: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/1.jpg)
Workflow: Getting Startedwith SMS & NED Developer Training
NASA SIVO Code 610.3
With GMAO
May 2008
![Page 2: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/2.jpg)
Outline
• Review of Workflow System Concepts
• Example Workflow
• GEOS GCM Workflow
• Hands-on Example
• Troubleshooting
• Resources
![Page 3: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/3.jpg)
Basic Workflow Terms
SMS - the workflow engine that runs the scriptsCDP - command line shell for SMSTrigger - an event that kicks off a taskDependency - usually synonymous with trigger
SMS Components• Suite - an instance of an SMS workflow
• Described by a .def file• Also is the top-level directory
• Family - a group of related Workflow tasks or other families• Described within a .def file• Represented as a directory
• Task - a unit of work in the Workflow• Described within a .def file• Defined as a .sms file
![Page 4: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/4.jpg)
Files Making up a Workflow
• Workflow definition - SMS file thatdefines overall structure of workflow tasks& their dependencies
• Workflow scripts - SMS files thatrepresent tasks within a workflow; Oursgenerally call shell scripts
• Run scripts - any shell scripts thatperform an action for a workflow task
• Configuration files (.ned) - any sort offile generated from the variables definedin NED
![Page 5: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/5.jpg)
Submitting a workflow to runin NED1. Local install directory created
• Fresh install of a workflow will go here
2. Workflow & run scripts are created• Currently under source control• Checked out from source
repository, has tags & branches• Added to the install directory
3. Configuration files arecreated
• Defines all the variables shownin NED for user to configure
• Generated according to theconfiguration settings
• Added to the install directory
4. Calls an initial “submission” shellscript
• Script, generally, will submit theworkflow to SMS
![Page 6: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/6.jpg)
Workflow Example
Configuration
Running workflow
![Page 7: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/7.jpg)
Workflow Example
Configuration in NED
Configuration in XML
![Page 8: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/8.jpg)
Workflow ExampleWorkflow script - SMS file(runModelSegment.sms)
Run script - shell script(runModelSegment.bash)
Definition file(workflow.def)
![Page 9: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/9.jpg)
Some SMS commands
![Page 10: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/10.jpg)
GEOS Workflow
gcm_setup script(Larry’s script)
991 lines
~20 GEOS5 run scripts (coupledModel/GEOSgcm/bin)
Decompose into
setup_wf_env.bash
![Page 11: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/11.jpg)
GEOS Workflow
• Access the GEOS5 workflow scripts from mapmeby typing at the prompt: >> getgeos
• coupledModel/GEOSgcm/bin contains the coreGEOS5 run scripts:
archive_restarts.bashcreate_exp_dirs.bashcreate_live_rcs.bashcreate_rc_templates.bashgetgcm_rst_exe.bashget_restarts.bashgsi_analysis.bashlinkbcs.bashload_batch_parms.bashmove_history.bash
post_processing.bashpull_back_data.bashregress_test.bashremove_wf_env.bashrename_chk_to_rst.bashrename_rst_for_ana.bashresubmit.bashrun_model.bashsetup_model_env.bashsetup_wf_env.bash
![Page 12: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/12.jpg)
Hands-On Demo
![Page 13: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/13.jpg)
Simple Demo to Modify
• Check out a demo workflow
• Make a simple modification
• Check it back in
• Run it
![Page 14: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/14.jpg)
Requirements for RunningWorkflows
• Required for this demo:• Some experience with CVS or Subversion
• Account on mapme
• Member of wf_dev group on mapme
• Account on discover and/or explore
• A dirac account is not required for this demo
• User’s must have a dirac account to run theGEOS5 workflow
![Page 15: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/15.jpg)
Setup a New Workflow
• Log-in to mapme• First time developers must switch to the “wf_dev”
group:newgrp wf_dev
• Create a working directory and cd to it.• Run developer startup script to add command-line
shortcuts:. /home/workflow/bin/script/wfdev.bash
• Execute the following scripts to create a new “demo”workflow in your working directory
branchWorkflow <username> checkoutWorkflow <username>
![Page 16: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/16.jpg)
Enhance the Workflow
• Goal here is to add a new variable to theworkflow• cd to working directory/<username>/bin and edit
script runModelSegment.bash to add the variable“expName”.• Enter ‘echo “Variable: $expName”’.
• Check in the Demo workflow checkinWorkflow
![Page 17: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/17.jpg)
Enhance the Workflow (cont.)
• Bring up NED• Select File -> Open from File. Click OK.• Navigate to working directory/<username>
• open baseWorkflow.ned .• Go to File -> Client Mode and select “Design Mode”• Under Configuration Tree select Variable Group. Execute
<ctl>right click and navigate to Add a Variable.• Enter expName in the information field of the Name property (hit
return to enter the string in NED)• Expand the Variable Group and note the variable named
expName listed with the existing variables.• Under Configuration Tree select Base Workflow.
• Execute <ctl>right click then navigate to Edit Workflow Properties.• In the Repository Branch workflow property enter:/baseWorkflow/branches/<username>
![Page 18: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/18.jpg)
Enhance the Workflow (cont.)
• Save and run NED workflow• Select File -> Save to File. Choose a name for your new
NED workflow.• Select Run -> Submit Workflow
• Navigate to the tab named <Unique ID> and watch theworkflow execute.
![Page 19: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/19.jpg)
Tips: Testing Workflows
• Write the workflow for easier testing• Be generous with echoes/prints to track problems faster
• Check for error codes in scripts
• Run the workflow to test it• New workflows get installed and run under:
/home/workflow/expWorkDir
• Check the install directory when problems occur
• Test the workflow• Incrementally test to fix problems as you develop
• Use multiple user accounts to avoid hard-wired solutions
![Page 20: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/20.jpg)
Tips: TroubleshootingWhen a workflow fails…
• Determine where the problem is coming from• Files may be missing - did you commit?• SMS scripting errors - look at SMS job files• User script errors
• Check the log file• Edit and re-run the task to see if it passes
• Remote script errors - must have a log file
• Visit Modeling Guru for questions and answers• NASA website:
http://modelingguru.nasa.gov/clearspace/community/mapmewkflow
• SIVO staff regularly monitors Modeling Guru• Knowledge base for Earth System modeling
• Contact SIVO workflow staff for more assistance
![Page 21: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/21.jpg)
Other Resources
• Modeling Guru communityThis points to a number of resources for workflow developers, such
as discussion forum that you can use to post questions to us andothers.User tutorialsWorkflow group documentation
https://modelingguru.nasa.gov/clearspace/community/mapmewkflow
• SMS DocumentationThe SMS software was built by ECMWF, so while it is helpful, it may
refer to things specific to their environment, like MARS orprepIFS.
http://www.ecmwf.int/publications/manuals/sms/documentation/index.html
![Page 22: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/22.jpg)
BACKUP SLIDES
![Page 23: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/23.jpg)
![Page 24: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/24.jpg)
Steps to integrating the GEOSrun scripts
1. GEOS run.script is componentized or decomposed• Identify portions of the script that are generally
• Independent,• Perform a basic task
• Pull out that portion into a new script
2. Associate with SMS• New script becomes an SMS task• Task is defined in the definition file• A .sms file is created that invokes the shell script
3. Finally add variables from the workflowconfiguration
• Most scripts will require user-configurable variables• Simply source the file generated by NED at the top of the
script
![Page 25: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/25.jpg)
Special cases
• Scripts that finish running are considered“complete” by SMS• So submitting a job to a queue requires you to
wait for the job to complete before exiting
• Scripts that get extracted may have variabledependencies• Shared variables between scripts can be included
• Breaking down tasks changes how jobs enterthe queue• Lots of small jobs vs. single large job
![Page 26: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/26.jpg)
Workflow Config Management
Ideally GEOS source and workflow’s run scriptsshould be kept in sync in the same repository
![Page 27: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/27.jpg)
Workflow System in NCCS
• Run from singlelocation withinNCCS: “mapme”
• Connects to HPCresources toperform tasks• palm• discover• dirac• Others
![Page 28: New Workflow: Getting Started with SMS & NED · 2020. 5. 12. · Basic Workflow Terms SMS - the workflow engine that runs the scripts CDP - command line shell for SMS Trigger - an](https://reader036.vdocuments.net/reader036/viewer/2022071215/60463e8b4f74f22c59744a80/html5/thumbnails/28.jpg)
Workflow System in NCCS