cÉci/cism hpc training sessions use of matlab on the clusters · 2016. 11. 23. · calcul intensif...
TRANSCRIPT
![Page 1: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/1.jpg)
Calcul intensif et Stockage de Masse
CÉCI/CISM HPC training sessionsUse of Matlab on the clusters
![Page 2: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/2.jpg)
CISM Typical usage...
InteractiveType in and get an answer
SequentialPerform tasks one after theother
BatchSubmit job and fetch results
ParallelPerform multiple tasks at thesame time
![Page 3: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/3.jpg)
CISM One more obstacle: Matlab Licensing
![Page 4: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/4.jpg)
Calcul intensif et Stockage de Masse
Using Matlab in batch modeWith Matlab (e.g. your computer or Lagrange)Without Matlab (e.g. the clusters)
Using Matlab in parallelWith no effortWith little effortWith a lot of effort
November 2015 [email protected]
Parallel Matlab on the cluster
![Page 5: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/5.jpg)
CISM Using Matlab in Batch mode
You might be used to ...
![Page 6: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/6.jpg)
CISM Using Matlab in Batch mode
... but no GUI in batch mode !
![Page 7: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/7.jpg)
CISM Using Matlab in Batch mode
Two methods for launching a script:
You have a script myscript.m:
You have a function "function a = myfun(x,y) ..."
![Page 8: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/8.jpg)
CISM Using Matlab in Batch mode
You need to adapt your Matlab script
Paths
Data
Prompts
Figures
GUI
Make sure to automate any setup that you usually do byhand: adding paths, loading data, saving results, etc.
No more clicks!
![Page 9: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/9.jpg)
CISM Using Matlab in Batch mode
You need to adapt your Matlab script
Paths
Data
Prompts
Figures
GUI
Put all 'configuration' values in a file and load it
![Page 10: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/10.jpg)
CISM Using Matlab in Batch mode
You need to adapt your Matlab script
Paths
Data
Prompts
Figures
GUIUse 'print' to put your
graphics to a file.
Or avoid plotting at all and dothat interactively atpostprocess time
![Page 11: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/11.jpg)
CISM Using Matlab in Batch mode
You need to adapt your Matlab script
Paths
Data
Prompts
Figures
GUI
Simplest solution: avoid GUI'sand use command line version
![Page 12: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/12.jpg)
CISM Using Matlab in Batch mode
And then launch it !
Options
Launch
-nodisplay: do not try to display plots-nodesktop: do not launch full GUI-nojvm: do not launch Java support
(do not use in recent versions of Matlab)-nosplash: do not display splashscreen
And then launch it !
![Page 13: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/13.jpg)
CISM Using Matlab in Batch mode
Use 'screen' for unattended execution
As soon as you pressENTER, the screen is
cleared and you are in a 'screen' session
![Page 14: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/14.jpg)
CISM Using Matlab in Batch mode
Use 'screen' for unattended execution
We are now in a 'screen' session
![Page 15: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/15.jpg)
CISM Using Matlab in Batch mode
Use 'screen' for unattended execution
We launch Matlaband we 'detach' with
CTRL-a d
![Page 16: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/16.jpg)
CISM Using Matlab in Batch mode
Use 'screen' for unattended execution
We are now detachedand disconnected.
![Page 17: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/17.jpg)
CISM Using Matlab in Batch mode
Use 'screen' for unattended execution
We connect backto lm9 and 'reattach'
to the 'screen' session
![Page 18: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/18.jpg)
CISM Using Matlab in Batch mode
Use 'screen' for unattended execution
And we find our result
![Page 19: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/19.jpg)
CISM Your turn..1. Connect to lm9 “ssh -X lm9.cism.ucl.ac.be”
Login: tutoXX where XX is a number between 01 and 19Password: xxxxxxxx
2. Copy directory ~dfr/matlab/batch to your directory and “cd” there
3. Load module matlab/R2010a3. Launch Matlab
4. Run 'go' to see what it does (you will have to set the paths correctly: File > Set Paths myTools)
5. Edit go.m so as to be able to run it in batch6. Quit Matlab
7. Test your Matlab script in 'batch mode'
8. Make a longer test with “screen”
![Page 20: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/20.jpg)
CISM Your turn..
Step 2 : create the SGE script
Step 3 & 4 : submit & wait
Step 5 : get the results
Tell SGE you need Matlabuse -nosplash -nodesktop -nodisplay -nojvmuse redirection < or option -r
![Page 21: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/21.jpg)
CISM Your turn..
![Page 22: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/22.jpg)
Calcul intensif et Stockage de Masse
Using Matlab in batch modeWith Matlab (e.g. your computer or lagrange)Without Matlab (e.g. the clusters)
Using Matlab in parallelWith no effortWith little effortWith a lot of effort
November 2015 [email protected]
Parallel matlab on the cluster
![Page 23: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/23.jpg)
CISM Dealing with the license
Number of licenses is limited!
![Page 24: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/24.jpg)
CISM Dealing with the license
Option 1 : Compile Matlab to C...
Why
How
Issues
![Page 25: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/25.jpg)
CISM Dealing with the license
Option 1 : Compile Matlab to C...
Why
How
IssuesUse -a to add resources (additional code or mat files)
Addpath are forbidden in compiled code. Protect them with if ~isdeployed addpath(...)end
Within Matlab:
myfunction must be a function, not a script
![Page 26: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/26.jpg)
CISM Dealing with the license
Option 1 : Compile Matlab to C...
![Page 27: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/27.jpg)
CISM Dealing with the license
Option 1 : Compile Matlab to C...
![Page 28: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/28.jpg)
CISM Dealing with the license
Option 1 : Compile Matlab to C...
![Page 29: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/29.jpg)
CISM Dealing with the license
Option 1 : Compile Matlab to C...
Why
How
Issues
![Page 30: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/30.jpg)
CISM Dealing with the license
Option 1 : ... and deploy with MCR
Why
How
Issues
![Page 31: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/31.jpg)
CISM Dealing with the license
Option 1 : ... and deploy with MCR
![Page 32: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/32.jpg)
CISM Dealing with the license
Option 1 : ... and deploy with MCR
![Page 33: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/33.jpg)
CISM Dealing with the license
Option 1 : ... and deploy with MCR
![Page 34: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/34.jpg)
CISM Dealing with the license
Option 1 : ... and deploy with MCR
![Page 35: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/35.jpg)
CISM Dealing with the license
Option 1 : ... and deploy with MCR
![Page 36: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/36.jpg)
CISM Dealing with the license
Option 1 : ... and deploy with MCR
![Page 37: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/37.jpg)
CISM Dealing with the license
Option 1 : ... and deploy with MCR
![Page 38: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/38.jpg)
CISM Your turn..
1. Connect to lm9 “ssh -X lm9.cism.ucl.ac.be”Login: tutoXX where XX is a number between 01 and 19Password: *******
2. Copy directory ~dfr/matlab/compile to your directory and “cd” there
3. Load module matlab/R2014a and launch Matlab 4. Compile go_f.m (note if ~isdeployed ) mcc -a myTools/ -m go_f.m
5. Connect to Hmem with your CÉCI login
6. Copy your 'compile' directory from lm97. Load MCR module version 713 (no need to copy/install MCR)
9. Run go_f (no need for run_go_f.sh)
![Page 39: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/39.jpg)
CISM Dealing with the license
Option 1 : ... and deploy with MCR
Matlab is not installed on Manneback but the MCR is
![Page 40: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/40.jpg)
CISM Dealing with the license
Option 2 : Develop with Matlab, run with Octave
“a language that is mostly compatible with Matlab”GPL license, free
Why
How
Issues
![Page 41: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/41.jpg)
CISM Dealing with the license
Option 2 : Develop with Matlab, run with Octave
Why
How
Issues
You have a script myscript.m. Rather than:
Simply write
The other option '-r' becomes '--eval '
![Page 42: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/42.jpg)
CISM Dealing with the license
Option 2 : Develop with Matlab, run with Octave
Why
How
Issues
![Page 43: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/43.jpg)
CISM Dealing with the license
Option 2 : Develop with Matlab, run with Octave
Why
How
Issues
Plots
Toolboxes
Java
Multithreading
Speed
Not as good as Matlab's
![Page 44: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/44.jpg)
CISM Your turn..
1. Connect to lm9 “ssh -X lm9.cism.ucl.ac.be”Login: tutoXX where XX is a number between 01 and 19Password: ???????
2. Copy your 'batch' directory into 'octave' : cp -r batch octave
3. Go to your octave directory 4. Load the octave module 5. Launch octave
6. Run go_f
7. exit octave
8 Launch “octave –eval go_f”
![Page 45: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/45.jpg)
CISM
![Page 46: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/46.jpg)
Calcul intensif et Stockage de Masse
Using Matlab in batch modeLaunch a script, get resultsDeal with licenses
Using Matlab in parallelWith no effortWith little effortWith a lot of effort
November 2015 [email protected]
Parallel matlab on the cluster
![Page 47: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/47.jpg)
CISM No effort: Multithreading
![Page 48: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/48.jpg)
CISM No effort: MultithreadingmaxNumCompThreads
exists since R2007b, defaults to one core
defaults to available number of cores from R2008a
marked deprecated since R2009b warning off MATLAB:maxNumCompThreads:Deprecated to suppress warning
![Page 49: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/49.jpg)
CISM No effort: Multithreading
Ask for several coresin the submission script
Alone on the node With 2 Matlab running concurrently
![Page 50: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/50.jpg)
CISM No effort: MultithreadingElement wise operations and linear algebra
![Page 51: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/51.jpg)
CISM No effort: MultithreadingMore with the Parallel Computing Toolbox
![Page 52: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/52.jpg)
Calcul intensif et Stockage de Masse
Matlab on the cluster
Using Matlab in batch modeLaunch a script, get resultsDeal with licenses
Using Matlab in parallelWith no effortWith little effortWith a lot of effort
November 2015 [email protected]
![Page 53: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/53.jpg)
CISM Little effort: embarrassingly parallel
Embarrassingly parallel programs (fork-join without communication / synchronization)
e.g. Parameter sweep, Monte carlo, Multiple data sets
Outer-loop splitting:
e.g. Loop over parameter
Program-level parallelism
Inner-loop splitting :
e.g. Loop over a vector index
Data-level parallelism
![Page 54: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/54.jpg)
CISM Little effort: embarrassingly parallel
Outer-loop splitting with Slurm's srun
![Page 55: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/55.jpg)
CISM Little effort: embarrassingly parallel
Outer-loop splitting with Slurm's srun
![Page 56: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/56.jpg)
CISM Little effort: embarrassingly parallel
Outer-loop splitting with Slurm's srun
![Page 57: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/57.jpg)
CISM Little effort: embarrassingly parallel
Outer-loop splitting with Slurm's srun
Requires one license per task !
But we know what to do, don't we ?
![Page 58: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/58.jpg)
CISM Little effort: embarrassingly parallel
Outer-loop splitting with Slurm's srun
Try it yourself! : ~dfr/matlab/embarrassingly*
![Page 59: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/59.jpg)
CISM Little effort: embarrassingly parallel
![Page 60: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/60.jpg)
CISM Little effort: embarrassingly parallel
Parallel Computing Toolbox
Parfor, spmd and distributed arrays
![Page 61: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/61.jpg)
CISM Little effort: embarrassingly parallel
Parallel Computing Toolbox
Parfor, spmd and distributed arrays
![Page 62: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/62.jpg)
CISM Little effort: embarrassingly parallel
Parallel Computing Toolbox
Parfor, spmd and distributed arrays
Can be compiled !
![Page 63: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/63.jpg)
CISM Little effort: embarrassingly parallel
Parallel Computing Toolbox
Parfor, spmd and distributed arrays
![Page 64: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/64.jpg)
CISM Little effort: embarrassingly parallel
Parallel Computing Toolbox
Parfor, spmd and distributed arrays
Compilation can fail :(
![Page 65: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/65.jpg)
CISM Little effort: embarrassingly parallel
Matlab 3rd party peval : jPar
On the solvers: >> jpar_solver(['hostname']);
On the server: >> a = rand(100,100,10) + i*rand(100,100,10);
>> [V,D] = jpar_client('eig', a)In
tere
sting
but n
ot
com
pilab
le
![Page 66: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/66.jpg)
CISM Little effort: embarrassingly parallel
Matlab 3rd party peval : Multicore
On the slaves: >> startmulticoreslave
On the master: >> for i=1:10; a{i} = rand(100,100) ;end
>> cellRes = multicoremaster(@eig, a)
Compil
able
uses
file
syste
m
mult
inode
![Page 67: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/67.jpg)
CISM Little effort: embarrassingly parallel
![Page 68: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/68.jpg)
CISM Little effort: embarrassingly parallel
Matlab 3rd party peval : Multicore
Compile (e.g. lm9):
matlab -r "mcc -a myTools/ -a multicore/ -m go_adapted_function_compile_peval;"
matlab -r "cd multicore/;mcc -a ../myTools/ -a ../ -m startmulticoreslave;exit"
Submit (e.g. Manneback):
![Page 69: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/69.jpg)
CISM Little effort: embarrassingly parallel
Matlab 3rd party peval : Multicore
Try it yourself! : ~dfr/matlab/multicore_compiled
![Page 70: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/70.jpg)
CISM Little effort: embarrassingly parallel
Octave peval: parcellfun, pararrayfun
A = {rand(100,100), rand(100,100), rand(100,100)}
Res = parcellfun(2, @eig, A)Mult
icore
![Page 71: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/71.jpg)
CISM Little effort: embarrassingly parallel
Octave peval: parcellfun, pararrayfun
srun --ntasks=1 --cpus-per-task=4 octave < go_f_parcellfun.mTry it ! ~dfr/matlab/parcellfun_octave
![Page 72: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/72.jpg)
CISM Little effort: embarrassingly parallel
Octave peval: parcellfun, pararrayfun
A = rand(100,100,3)
Res = pararrayfun(2, @eig, A)
![Page 73: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/73.jpg)
CISM Little effort: embarrassingly parallel
Octave peval: multicore
On the slaves: >> startmulticoreslaves
On the master: >> for i=1:10; a{i} = rand(100,100) ;end
>> cellRes = multicoremaster(@eig, a)m
ultino
de
![Page 74: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/74.jpg)
CISM Little effort: embarrassingly parallel
Octave peval: multicore
![Page 75: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/75.jpg)
CISM Little effort: embarrassingly parallel
Octave peval: multicore
Try it ! ~dfr/matlab/multicore_octave
![Page 76: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/76.jpg)
CISM Little effort: embarrassingly parallel
Octave peval: multicore
![Page 77: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/77.jpg)
Calcul intensif et Stockage de Masse
Matlab on the cluster
Using Matlab in batch modeLaunch a script, get resultsDeal with licenses
Using Matlab in parallelWith no effortWith little effortWith a lot of effort
November 2015 [email protected]
![Page 78: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/78.jpg)
CISM A lot of effort: explicitly parallel
Explicitly parallel programs
SPMD & MPMD
Communications handled explicitly by the user
Matlab not specifically good at it..
![Page 79: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/79.jpg)
CISM A lot of effort: explicitly parallel
Parallel Computing Toolbox
Parfor, spmd and distributed arrays
OpenMP-like construct based on MPI for distributed memory
![Page 80: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/80.jpg)
CISM A lot of effort: explicitly parallel
Lincoln Laboratory MatlabMPI
![Page 81: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/81.jpg)
CISM A lot of effort: explicitly parallel
Lincoln Laboratory pMatlab
![Page 82: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/82.jpg)
CISM A lot of effort: explicitly parallel
http://www.walkingrandomly.com/?p=1795
OpenMP Mex files
![Page 83: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/83.jpg)
CISM A lot of effort: explicitly parallel
Octave: reval (parallel package)
3rd party toolboxes
Compile
Octave
![Page 84: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/84.jpg)
CISM A lot of effort: explicitly parallel
Octave and MPI
![Page 85: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/85.jpg)
CISM Batch Matlab/Octave
● Scripts need adjustment● Batch processing with Matlab
● Use 'screen'
● Batch processing without Matlab● Compile with mcc● Dev. with Matlab, Prod. with Octave
![Page 86: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/86.jpg)
CISM Parallel Matlab/Octave
● No effort: Matlab Multithreading● Some effort: embarrassingly parallel
● Matlab: Jpar, multicore● Octave: parcelleval, multicore
● More (too much?) effort● Matlab: SPMD, MPI toolboxes● Octave: parallel, openmpi_ext
Coming
to
Octave
![Page 87: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/87.jpg)
CISM
![Page 88: CÉCI/CISM HPC training sessions Use of Matlab on the clusters · 2016. 11. 23. · Calcul intensif et Stockage de Masse Using Matlab in batch mode With Matlab (e.g. your computer](https://reader035.vdocuments.net/reader035/viewer/2022071213/6041b67c6792d6560333dd11/html5/thumbnails/88.jpg)
CISM
----------------------------------------------MATLAB | MATLAB | MATLAB | Release | Component | Compiler | | Runtime (MCR) | Version | ----------------------------------------------R14 (7.0) | 7.0 | 4.0 | R14SP1 (7.0.1) | 7.1 | 4.1 | R14SP2 (7.0.4) | 7.2 | 4.2 | R14SP3 (7.1) | 7.3 | 4.3 | R2006a (7.2) | 7.4 | 4.4 | R2006b (7.3) | 7.5 | 4.5 | R2007a (7.4) | 7.6 | 4.6 | R2007b (7.5) | 7.7 | 4.7 | R2008a (7.6) | 7.8 | 4.8 | R2008b (7.7) | 7.9 | 4.9 | R2009a (7.8) | 7.10 | 4.10 | R2009b (7.9) | 7.11 | 4.11 | R2009bSP1 (7.9.1) | 7.12 | 4.12 | R2010a (7.10) | 7.13 | 4.13 | R2010b(7.11) | 7.14 | 4.14 |R2010bSP1 (7.11.1)| 7.14.1 | 4.14.1 |R2011a(7.12) | 7.15 | 4.15 |R2011b(7.13) | 7.16 | 4.16 |R2012a(7.14) | 7.17 | 4.17 |R2012b(8.0) | 8.0 | 4.18 |----------------------------------------------