ComputingRefresher(Day1)TimMoonICMERefreshersSeptember19,2016

Introduction

AboutMe•  Third-yearICMEPhD•  Researchinterests:•  High-performancecompuFng•  Numericallinearalgebra

•  Coursewebsite:hJps://stanford.edu/~tym1/refreshers•  Personalwebsite:hJps://stanford.edu/~tym1•  Email:tym1@stanford.edu

Goals•  AccessStanford’scompuFngresources•  NavigatewiththeUnixcommandline•  EdittextfileswithVim•  UseMATLABasacalculator

AdditionalResources•  hJps://uit.stanford.edu/service/sharedcompuFng/loggingin(Stanford’stutorialtologintoFarmshare)

•  hJps://linuxjourney.com(verygoodonlinecourseonLinux)•  hJps://danielmiessler.com/study/vim/(tutorialtobecomeaVimguru)

•  hJp://tnerual.eriogerg.free.fr/vimqrc.pdf(Vimcheatsheet)•  hJp://vim-adventures.com(silly,Legend-of-Zelda-styleVimtutorial)

•  hJp://stanford.edu/class/cme192/(ICMEMATLABcourse)

WhatisUnix?

FarmShare•  SharedcomputersopentoStanfordstudents•  corn:generalpurpose•  rye:GPUs•  cardinal:long-term,low-intensityjobs•  barley:high-intensityjobs(usingjobmanager)

•  FilesstoredonAndrewFileSystem(AFS)•  ManagefilesathJps://afs.stanford.edu•  WebhosFngwithWWWdirectory

•  Availableso`ware:•  R,abinit,acml,allinea,ampl,amplgurobi,ansys,biopython,bowFe2,caffe,

celltool,cplex,cuda,cudasamples,cudnn,cufflinks,farmvnc,fiji,gams,gaussian,gurobi,hisat2,intel,java,lemon,lifelines,maple,mathemaFca,matlab,meep,meta,openfoam,parallel,pyroseJa,r,relion,roseJa,rstudio,sagemath,sas,seaborn,sentaurus,sicstus,spectra,sratoolkit,statamp,statase,staJransfer,terraform,tophat,torch,xmipp

SSH•  “SecureShell”•  Standardmethodtologintoaremotecomputer•  GraphicaluserinterfacewithX11•  XWindowSystem,version11

LoggingintoFarmShare•  Linux•  OpenTerminalandtype:

ssh -X <SUNetID>@corn.stanford.edu

•  Mac•  InstallXquartz(hJps://www.xquartz.org)•  OpenTerminal(ApplicaFons/UFliFes/Terminal)andtype:

ssh -X <SUNetID>@corn.stanford.edu

•  Windows•  InstallXming(hJps://sourceforge.net/projects/xming/)•  DownloadandopenPuTTY(hJp://www.puJy.org)

•  Set“HostName(orIPAddress)”to“corn.stanford.edu”(inSession)•  Select“EnableX11forwarding”(inConnecFon/SSH/X11)

•  UseyourSUNetIDasyourusernameandfollowthestepsfortwo-factorlogin

CommandLineEssentials

PracticeFiles•  Downloadfilesbytyping:

git clone https://github.com/timmoon10/icme_computing_refresher_2016

cd icme_computing_refresher_2016

GettingStarted

•  AbortcommandsbypressingCtrl-c

Command Purposeecho <arg> Printargumentstodisplayexit Exitshell

Directories

•  RelaFvepathvs.absolutepath•  Pathshortcuts•  .(currentdirectory)•  ..(parentdirectory)•  ~(homedirectory)

Command Purpose UsefulOp2onsls <dir> Listcontentsindirectory -l,-apwd Printcurrentdirectorycd <dir> Changedirectorymkdir<dir> Makedirectoryrmdir<dir> Removedirectory

ReadingFilesCommand Purpose Useful

Op2onstouch <file> Createemptyfilecat <file> Printfilehead <file> Printfirstfewlines -n <lines>tail <file> Printlastfewlines -n <lines>less <file> Readfilein“less”

•  UseVimnavigaFoninless•  lessisbeJerthanmore

ManipulatingFilesCommand Purpose UsefulOp2onscp <old> <new> Copyfile -r,-imv <old> <new> Movefile -i

rm <file> Removefile -r,-f,-i

•  Becareful,Unixdoesnotforgive•  Wildcards•  *(anynumberofcharacters)•  ?(exactlyonecharacter)

Help

•  Manycommandscanalsoprintahelppage•  Put“-h”or“--help”asacommand-lineargument

•  Googleisyourfriend

Command Purposeman <command> Manualpage

AdvancedUnixTopics

UsefulCommandsCommand Purpose UsefulOp2onshistory Printpreviouscommandswc <file> Countwords -w,-l,-m,-csort <file> Sorttextlines -r,-n

grep

•  Youcanusearegularexpression(regex)•  UsefulforsearchingthroughmulFplefiles

Command Purpose UsefulOp2onsgrep <string> <file> Searchforstringinfile -i,-r

RedirectionandPiping•  RedirecFon(>)putsstandardoutputinfile•  Use >>toappendtofile•  ls > file_list.txt

•  Pipe(|)givesoutputtoanotherprogram•  Unixphilosophyusesmodularprograms•  ls | echo

ProjectTrees•  Commonstructure:•  README:helpfile•  Makefile:Makefileformake•  src:sourcefiledirectory•  doc:documentaFondirectory•  include:headerfiledirectory•  ext:directoryforexternalprojects•  bin:directoryforcompiledexecutables•  lib:directoryforcompiledlibraries•  build:directoryforintermediatebuildfiles

Make•  ManyprogramsrequirecompilaFonfromsourcecode•  MayalsorequireconfiguraFon,installingdependencies,andinstallaFon

•  Makeisacommontoolforbuildingprojects•  LooksatMakefileincurrentdirectory•  ManyprojectsuseCMake,whichgeneratesMakefiles

•  Usage:•  Changetoprojectdirectoryandtype“make”•  “make clean”iso`enusedtoresetdirectory•  MayrequireconfiguraFonwith“./configure”•  MayrequireinstallaFonwith“make install”

VersionControl•  CollaboraFvecodingcanbeapain•  Simultaneouschanges•  Findingwhereabugwasadded•  Seeingwhowrotewhat

•  Commonprograms:Git,Subversion,Perforce,Mercurial•  HighlyrecommendmakingaGitHubaccount

GitCommand Purposegit clone <repo> Downloadrepositorygit pull Updatefromremotegit status Getrepositorystatusgit add <file> Preparetoaddfilegit rm <file> Removefilegit commit –m “<message>” Commitlocallygit push Submitchangestoremote

SimpleGitWorkQlow

Makechangestocode

git pull

git add <file>

git commit -m “<message>”

git push

Tarballs

•  “Tapearchive”•  Compressionwithgzip•  .tar.gzor.tgzfileextension•  TrydownloadinganddecompressinghJps://stanford.edu/~tym1/refresher/materials/example_tarball.tgz

Command Purposewget <url> Downloadfiletar –czvf <tarfile> <files> Compresstotarballtar -xzvf <tarfile> Decompresstarball

ComputingRefresher(Day2)TimMoonICMERefreshersSeptember20,2016

LoggingintoFarmShare•  Linux•  OpenTerminalandtype:

ssh -X <SUNetID>@corn.stanford.edu

•  Mac•  InstallXquartz(hJps://www.xquartz.org)•  OpenTerminal(ApplicaFons/UFliFes/Terminal)andtype:

ssh -X <SUNetID>@corn.stanford.edu

•  Windows•  InstallXming(hJps://sourceforge.net/projects/xming/)•  DownloadandopenPuTTY(hJp://www.puJy.org)

•  Set“HostName(orIPAddress)”to“corn.stanford.edu”(inSession)•  Select“EnableX11forwarding”(inConnecFon/SSH/X11)

•  UseyourSUNetIDasyourusernameandfollowthestepsfortwo-factorlogin

PracticeFiles•  TryupdaFngtherepo:•  cd icme_computing_refresher_2016•  git pull

•  Ifyougetanerror,deletetherepoanddownloadagain:•  cd ~•  rm –rf icme_computing_refresher_2016•  git clone https://github.com/timmoon10/icme_computing_refresher_2016

Vim

WhatisVim?•  “ViIMproved”•  viisasimilar,oldertexteditor

•  Mostpopularcommand-linetexteditor•  FoundonalmostallUnixsystems

•  ModalediFng•  Powerful,butsteeplearningcurve

•  Veryusefultoprintoutacheatsheet:•  hJp://tnerual.eriogerg.free.fr/vimqrc.pdf

•  StartVimbytyping:vim <file>

OtherTextEditors•  Emacs•  SublimeText•  nano•  Notepad++

Navigation

•  DefaultVimmodeisNormalMode•  Navigatewitharrowkeysorhjklkeys

Command PurposeEsc ReturntoNormalModeh Le`j Downk Upl Right

InsertMode

•  Thisiswhereyouactuallytype•  ReturntonormalmodewithEsc

Command Purposei EnterInsertModebeforecursor(“insert”)a EnterInsertModea`ercursor(“append”)

SavingandExitingCommand Purpose:w <file> Savetofile(“write”):e <file> Openanotherfile(“edit”):q Quit(requiresnochanges):wq Saveandquit:q! Quitwithoutsaving

SearchingCommand Purpose/<string> Forwardsearchforstring?<string> Backwardsearchforstringn NextsearchresultN Previoussearchresult

VisualModeCommand Purposev EnterVisualMode(character-based)V EnterVisualMode(line-based)y CopyselecFon(“yank”)x Cut/deleteselecFonp Pastea`ercursorP Pastebeforecursor

•  VisualModeisusedtoselectregionsoftext•  NavigationissameasNormalMode

MoreUsefulCommandsCommand Purpose0 Gotobeginningofline$ Gotoendoflinegg GotobeginningoffileG Gotoendoffile:<number> Gotospecificlinedd Deletelineu Undo. Repeatlastcommandorredoundo

ScreenSplittingCommand PurposeCtrl-w s SplitwindowCtrl-w n CreateemptywindowCtrl-w j MovetonextwindowCtrl-w k MovetopreviouswindowCtrl-w o Closeallbutcurrentwindow

•  EachwindowislikeanindependentsessionofVim

MATLAB

WhatisMATLAB?•  “MatrixLaboratory”•  Proprietaryso`warebyMathWorks•  VerygoodforexperimentaFon•  High-levelprogramminglanguage•  FastmatrixoperaFons•  Feature-rich

•  StarFngMATLABonFarmShare:module load matlabmatlab

•  Towriteascript,putmulFplecommandsina.mfile•  Torunmy_script.m,typemy_scriptinMATLAB

Alternatives•  Octave•  Python(withNumPyandSciPy)•  Julia•  R

Variables•  a = 4•  b = -5.3•  third_variable = a * b•  d = sin(a) + 1.1•  a > b•  Comparisons:<,<=,>,>=,==,~=

•  Suppressoutputwithsemicolon:•  f = 3.1 – 4.3 * i;

•  Specialvalues:pi,i,Inf,NaN•  FuncFons:sin,cos,exp,sqrt,round,floor,ceil

Matrices•  A = [1 2 3; 4 5 6]•  MathemaFcalresult:

•  B = exp(A)•  A + B•  [1; 2; 3]•  [4 5 6]

A = 1 2 34 5 6

⎡

⎣⎢

⎤

⎦⎥

MatrixOperations•  A = [1 2; 3 4]•  B = [5 6; 7 8]•  [A; B]•  [A B]•  A * B(matrixmulFplicaFon)•  A \ B(matrixinverse)•  Entry-wiseoperaFons:.*or./or.\or.^

UsefulFunctions•  zeros(2,3)•  ones(2,3)•  eye(2,3)•  rand(2,3) (uniformdistribuFon)•  randn(2,3) (normaldistribuFon)•  1:4•  5:-2:1•  linspace(2,4,5)

Indexing•  A = [1 2 3 4; 5 6 7 8; 9 10 11 12]•  A(2,3)•  A(1:2,2:4)•  A(1:2,2)•  A(2,2:end)•  A(1:2,:)•  A(:)•  A(3:5)

Plotting•  x = linspace(1,10,50);•  y = sin(x) + 1.1;•  z = sin(2*x) + 1.1;•  plot(x,y)•  plot(x,z)•  hold on•  close

Help•  help<function>•  doc <function>•  Google

Scripting•  JustputmulFplecommandsinafile•  .mfileextension

•  Runbytypingfilename,excluding.m•  OpenMATLAB’sfileeditorwith“edit <funcFon>”•  Commentwith%•  Controlstatements:if,else,elseif,while,for•  Closeblockwithend

Exercise•  Let’ssolvetheheatequaFon:

•  ForanapproximatesoluFonatthenthFmestep:

•  IniFalizeuasarandomcolumnvectorandusec=0.25•  ComputeuforseveralFmesteps.Plota`ereveryFmestep(uselinspace(0,1,5)forx-dimension).Whatishappening?

∂u∂t−∇2u = 0

un ≈ un−1 + cAun−1

A =

1 −1−1 2 −1

−1 2 −1−1 2 −1

−1 1

⎡

⎣

⎢⎢⎢⎢⎢⎢

⎤

⎦

⎥⎥⎥⎥⎥⎥

ComputingPotpourri

VirtualPrivateNetwork•  FarmSharerequiresuserstobeinStanfordnetwork•  VPNclientscreatesecureconnecFontoStanfordnetwork•  Veryusefulwhenoff-campus

•  SeeStanford’sVPNtutorial•  hJps://uit.stanford.edu/service/vpn

LaTeX•  LaTeXisadocumentpreparaFonsystem•  LiJleeffortfornice-lookingdocuments

•  Verygoodformathandtechnicaldocuments•  MakeslideshowswithBeamer•  LaTeXmarkuplanguage•  .texfileextension

•  ForaWYSIWYGeditor,tryLyX(hJps://www.lyx.org)

Jupyter•  VeryniceenvironmentforJulia,Python,andR•  Showsupinwebbrowser•  Workbooksareconvenientwaytosharecodesnippets•  SeeinstallaFoninstrucFons:•  hJp://jupyter.readthedocs.io/en/latest/install.html

LinksCommand Purposeln -s <target><link> Createsymboliclinktotarget

•  Symboliclinkpointstoatargetpath•  Leavingoutthe“-s”opFoncreatesahardlink•  Besttoavoidsincehardlinksarenotassafe

FilePermissions•  “ls -l”showsfilepermissions•  Filetype•  Ownerpermissions(read,write,execute)•  Grouppermissions(read,write,execute)•  Otheruserpermissions(read,write,execute)

•  Changefilepermissionswithchmod <number> <file>

•  Numberhasthreedigits:owner,group,andotherusers•  Eachdigit(inbinary)givesread,write,executepermissions•  Commonnumbersare500(fortext)and700(forexecutables)

BashScripting•  “BourneAgainShell”•  Writescriptsbypusngcommandsasifincommand-line•  Commentwith#•  Setvariableswith=•  MY_VARIABLE=<value>

•  Referencevariableswith$•  echo $MY_VARIABLE

•  Runwithsourceorbysesngfilepermissionstoexecute•  Commontoput“#!/bin/bash”infirstline

SuperUser

•  Defaultstosuperuser•  Alsoknownasroot,administrator,supervisor

•  Usermustbeinsudoerlisttousesudo•  Commonforinstallingpackagesandconfiguringsystem•  Requiresmodifyingsystemfiles

Command Purposesu <users> Switchusersudo <command> Runcommandasanotheruser

StanfordComputingResources•  FarmShare:compuFngclusterforstudents,faculty,staff•  ICMEclusters•  icme-sharedmem•  icme-hadoop1•  icme-mpi1•  icme-gpu•  VirtualdesktopsforICMEstudents•  ContactBrianTempero(btempero@stanford.edu)foraccess

•  Sherlock:high-performancecompuFngcluster•  Forresearch,requiressponsorshipfromfacultymember

•  JobmanagementwithSLURM

ICMEComputingClasses•  Fullcourses:•  CME211-compuFngbasicswithPythonandC++•  CME212-scienFficcompuFngwithC++•  CME213-parallelcompuFng(mostlyGPUs)•  CME214-Fortran•  CME323-distributedcompuFngwithSpark•  CME342-theoreFcalparallelcompuFng

•  Shortcourses:•  CME192-MATLAB•  CME193-Python•  CME195-R•  CME292-AdvancedMATLAB