Chemistry Packages at CHPC

Anita M. OrendtCenter for High Performance Computing

anita.orendt@utah.eduFall 2012

Purpose of Presentation• Identify the computational chemistry software

and related tools currently available at CHPC• Present brief overview of these packages• Present how to access packages on CHPC• Next presentation – Gaussian09 & GaussView

– 18 October 2012

http://www.chpc.utah.edu

Brief Overview CHPC Resources• Computational Clusters – sanddunearch, updraft and ember • Home directory – NFS mounted on all clusters

– /uufs/chpc.utah.edu/common/home/<uNID>– generally not backed up (there are exceptions)

• Scratch systems – /scratch/serial – all clusters – 15 TB NFS– /scratch/general – updraft only – 3.5TB NFS– /scratch/ibrix/chpc_gen – updraft and ember – 55TB– /scratch/local on compute nodes (60GB sda, 200GB updraft, 400GB

ember)• Applications

– /uufs/arches/sys/pkg– /uufs/chpc.utah.edu/sys/pkg – /uufs/cluster.arches/sys/pkg where cluster = sanddunearch, updraft or

emberhttp://www.chpc.utah.edu

http://www.chpc.utah.edu

Switch

Ember about 382 nodes/4584 cores

Infiniband and GigE67 nodes general usage

/scratch/ibrix/chpc_genember, updraft

Administrative Nodes NFS

IBRIX

Owner nodes

Home Directories

Sanddunearch156 nodes/624 coresInfiniband and GigE

Telluride

Turret Arch

Meteorology

Updraft256 nodes/2048 cores Infiniband and GigE

85 nodes general usage

scratch systemsserial – all clustersgeneral – updraft

NFS

12 GPU nodes (6 CHPC)

Getting Started at CHPC• Account application – an online process

– https://www.chpc.utah.edu/apps/profile/account_request.php • Username is your unid with password administrated by campus• Interactive nodes

– two CHPC owned nodes per cluster (cluster.chpc.utah.edu) with round-robin access to divide load

• CHPC environment scripts – in account when created– www.chpc.utah.edu/docs/manuals/getting_started/codes/chpc.tcshrc– www.chpc.utah.edu/docs/manuals/getting_started/codes/chpc.bashrc

• Getting started guide– www.chpc.utah.edu/docs/manuals/getting_started

• Problem reporting system– http://jira.chpc.utah.edu or email to issues@chpc.utah.edu

http://www.chpc.utah.edu

Interactive Node Usage• Interactive nodes for prepping/testing of input files,

analyzing results, compilations, debugging, data transfer, etc – no running of jobs – 15 min MAX cpu– no jobs of any time length that negatively impact ability of

other users to get work done (e.g., heavy cpu, memory usage and/or i/o)

• https://wiki.chpc.utah.edu/display/policy/2.1.1+Cluster+Interactive+Node+Policy

http://www.chpc.utah.edu

Batch System• All use of compute nodes go through a batch system using

Torque (PBS) and Maui (Moab) for scheduling#PBS -S /bin/csh#PBS – A account#PBS –l walltime=24:00:00,nodes=2:ppn=8

• qsub –I to get interactive use of a compute node (-X if X11 forwarding needed)

• Walltime limits – 72 hours on sanddunearch– 24 hours on updraft– 72 hours on ember (long qos by request)

http://www.chpc.utah.edu

Allocation Procedure• Allocation process now within jira

https://jira.chpc.utah.edu/secure/CreateIssue.jspa?pid=10130&issuetype=12

• Allocations on a calendar quarter basis – Next requests due about Dec 1 for Jan-Mar 2013– Can request for up to 4 quarters at a time

• Allocations for different clusters are kept separate• Sanddunearch is no longer being allocated – all freecycle • One allocation per research group; accessible by all members

of group http://www.chpc.utah.edu

Out of Allocation Options• If your group did not request any allocation – can get a quick

allocation (limited to 5000 units) on one of the clusters– https://

jira.chpc.utah.edu/secure/CreateIssue.jspa?pid=10130&issuetype=13

• Can run in freecycle mode on sanddunearch; once job starts it will run until done or walltime requested is reached

• Can run on smithp-guest on smithp nodes ember or freecycle on CHPC nodes on either ember or updraft (jobs are preemptable)

http://www.chpc.utah.edu

Security Policies (1)• No clear text passwords - use ssh and scp• Do not share your account under any circumstances• Don’t leave your terminal unattended while logged into

your account• Do not introduce classified or sensitive work onto

CHPC systems• Use a good password and protect it – see

gate.acs.utah.edu for tips on good passwords

http://www.chpc.utah.edu

Security Policies (2)• Do not try to break passwords, tamper with files, look

into anyone else’s directory, etc. – your privileges do not extend beyond your own directory

• Do not distribute or copy privileged data or software• Report suspicions to CHPC (security@chpc.utah.edu)• Please see

http://www.chpc.utah.edu/docs/policies/security.html for more details

http://www.chpc.utah.edu

Access to Interactive Nodes• From Windows machine:

– Need ssh client• PuTTY: http://www.chiark.greenend.org.uk/~sgtatham/putty/ • Xshell 4: http://www.netsarang.com/products/xsh_overview.html

– For Xwindowing – need tool to display graphical interfaces such as Gaussview, ECCE, AutoDockTools• Xming (use Mesa version) -

http://www.straightrunning.com/XmingNotes

http://www.chpc.utah.edu

Default login scripts• CHPC maintains default login scripts that will set up

necessary environment for batch commands and many of the programs to work– http://www.chpc.utah.edu/docs/manuals/getting_started/code/chpc.tcshrc – http://www.chpc.utah.edu/docs/manuals/getting_started/code/chpc.bashrc

• Located in your home directory as .tcshrc or .bashrc• Can comment out setups for packages not used • Default ones provided have chemistry package setups

commented out – need to remove # at start of line• Can customize by creating .aliases file that is sourced at

end of the CHPC script – do not customize .tcshrc/.bashrc file as you will need to occasionally get a new version

http://www.chpc.utah.edu

Location of Installations• Have a three layer system for installations:

– /uufs/chpc.utah.edu/sys/pkg• Accessible on all clusters and CHPC supported linux

desktops– /uufs/arches/sys/pkg

• Accessible on all clusters, but not desktops– /uufs/cluster.arches/sys/pkg

• Single cluster access• For builds of packages customized to some feature

(usually for the MPI) of the given cluster

10/21/2010 http://www.chpc.utah.edu Slide 13

Software - General Information• Available on CHPC web pages/wiki

– Can get to from www.chpc.utah.edu → Software documentation → More (for complete list) http://www.chpc.utah.edu/docs/manuals/software

– Available for most packages – Has information on licensing restrictions, example

batch scripts, where to get more information on a specific package

– Also has useful information on running of jobs (scaling, issues, etc) for some applications

http://www.chpc.utah.edu

Types of Chemistry Packages• Molecular Mechanics/Dynamics• Electronic Structure Calculations• Plane Wave Pseudopotential based packages

– allows for periodic boundary conditions• Support Packages

• Integrated Modeling Platforms

http://www.chpc.utah.edu

Molecular Mechanics/Dynamics• Amber• Gromacs• NAMD• LAMMPS• Charmm (licensed by research group)

Note – Gaussian and NWChem also have MM capabilities

http://www.chpc.utah.edu

Quantum (Mainly) Packages• Gaussian03/09• NWChem• GAMESS• Molpro – serial only• Dalton – serial only• Orca• Psi• SIESTA – licensed by individual research group but can add

others to license at no charge• CFOUR

http://www.chpc.utah.edu

Plane Wave Pseudopotential(good for periodic systems)

• Quantum Espresso• CPMD• VASP – licensed by group

• BigDFT (wavelet basis set)

http://www.chpc.utah.edu

Integrated Molecular Platforms • Schrodinger – owned by individual groups

http://www.chpc.utah.edu

Support Packages• Gaussview• Molden• VMD• NBOView• Babel (Openbabel)• Dock • AutoDock (and Autodock Vina)• Cambridge Structural Database• ECCE

• Special case – installed on carbon.chpc.utah.edu• Talk to me first if you want to use

http://www.chpc.utah.edu

GaussView• Molecular builder and viewer for Gaussian input/output files • CHPC provides campus license for linux version

– For Gaussian03 – standard is version 4– For Gaussian09 – standard is version 5

• Chemistry Department has campus license for GV5 for windows; can buy into license

• Access with gv & – provided you have uncommented the Gaussian setup from the standard .tcshrc

• DO NOT submit jobs from within GaussView – instead create and save input file and use batch system

http://www.chpc.utah.edu

NBOView• Package to view NBO orbitals• Works only with G03 (not G09)• For information see

http://www.chem.wisc.edu/~nbo5/v_manual.htm• How to use at CHPC:

– Add to your path: /uufs/chpc.utah.edu/sys/pkg/nboview/nboview1.1

– nboview

http://www.chpc.utah.edu

Molden• Another program for viewing molecular/electronic

structures; version 4.7 • Works with Gamess, Gaussian, Molpro• Supports plots of electronic density, MOs, etc• More information at

http://www.cmbi.ru.nl/molden/molden.html• How to use at CHPC:

– Make sure your path includes /uufs/chpc.utah.edu/sys/pkg/molden/std

– molden &

http://www.chpc.utah.edu

VMD• Visualization, mainly for MM/MD• latest version (1.9.1) installed• Reads a number of different file formats• Information at http://www.ks.uiuc.edu/Research/vmd • Can install on own desktop (windows/mac/linux

versions available)• /uufs/chpc.utah.edu/sys/pkg/vmd/vmd-std/bin/vmd

(can uncomment setup in .tcshrc then vmd)• Can use for 3D viewing on CHPC’s vis wall

http://www.chpc.utah.edu

ECCE• Extensible Computational Chemistry Environment • Package developed at EMSL at PNNL

– See ecce.pnl.gov for info• Set of modules to manage computational chemistry

computer jobs (Gaussian, NWChem) from start (including building system) to finish (analyzing output) all from your desktop system

• Installed on carbon.chpc.utah.edu• User accounts need setup – see me if interested as

carbon is not always kept running

http://www.chpc.utah.edu

OpenBabel• Tool to interconvert structure files between a

number of formats used in molecular modeling• See openbabel.org for more information• To run:

– source /uufs/chpc.utah.edu/sys/openbabel/etc/babel.csh (or uncomment in your .tcshrc)

– babel -i < input-type > < infile > -o < output-type > < outfile >

– babel –H to see format for usage, options, and input/output file-types

http://www.chpc.utah.edu

Dock/AutoDock• Programs to look at binding of a small molecule within the active site of a

receptor, usually a macromolecule• Dock

– version 6.5 installed– get info at: http://dock.compbio.ucsf.edu – source /uufs/chpc.utah.edu/sys/pkg/dock/etc/dock.csh– dock6.mpi to start (needs arguments)

• Autodock – version 4.2– info available at http://autodock.scripps.edu – source /uufs/chpc.utah.edu/sys/pkg/autodock/etc/autodock.csh– autodock4 (with proper arguments) or autogrid4– autodocktools, a GUI interface, installed – start with adt &– Autodock Vina – multicore performance and enhanced accuracy – start

with vina &

http://www.chpc.utah.edu

Cambridge Structural Database• Moved from library to CHPC summer 2006 – additions to the

database are made every 3-4 months; package updated annually• www.ccdc.cam.ac.uk for information• Need CHPC account to use• From PC need Xterm/Xwindowing software (Putty/XMing work well)

to start session on any of the interactive nodes– source (or uncomment line in .tchsrc)

/uufs/chpc.utah.edu/sys/pkg/CSD/std/cambridge/etc/csd.csh– cq & <- to start conquest (search engine)– mercury & <- to start crystal structure viewer– The first time you use it on a given computer you will be asked to

confirm licensing • need to provide site/license codes (840/7E92A7)

http://www.chpc.utah.edu

Amber• Molecular mechanics/dynamics package Current version is

Amber 12 (Amber 11 still active)• Basic information on getting started:

http://www.chpc.utah.edu/docs/manuals/software/amber.html• For more information see http://ambermd.org

– The Amber Mail Reflector Archive and mailing list is very useful!• Different parallel builds available for each cluster to make use

of the different interconnects• Have GPU version – initial tests show impressive performance

gains

http://www.chpc.utah.edu

Other MM/MD packages-- All have builds optimized for individual clusters --• Gromacs - http://www.gromacs.org/

– version 4.5.4 installed with builds of both single and double precision

• LAMMPS - http://lammps.sandia.gov/ • NAMD: http://www.ks.uiuc.edu/Research/namd/

– version 2.6 on sanddunearch (2.7 current)• NWChem also has MM/MD capabilities and

Gaussian09 also has limited MM capabilities

http://www.chpc.utah.edu

CHARMM

• Package is licensed per research group and for a specific version

• If interested – get license and we can do installation such that only your group has access

• See www.charmm.org for details

http://www.chpc.utah.edu

Gaussian03/09• Commercial electronic structure package

– http://www.gaussian.com for information and User’s Guide• Last installed revision of G03 is E.01

– /uufs/chpc.utah.edu/sys/pkg/gaussian03/std– Has been updated to include latest NBO5

• Version C.01 of G09 installed– /uufs/chpc.utah.edu/sys/pkg/gaussian09/AMD64– /uufs/chpc.utah.edu/sys/pkg/gaussian09/EM64T

• For information on accessing the CHPC installation – http://www.chpc.utah.edu/docs/manuals/software/g03.html

http://www.chpc.utah.edu

NWChem• Package developed at PNNL to work on massively parallel systems• http://www.nwchem-sw.org• Goal: Computational chemistry solutions that are scalable with

respect to both chemical system size and MPP hardware size • Has quantum mechanics, molecular mechanics/dynamics, and

quantum molecular dynamics, plane waves for periodic systems • Version 6.1 (with Python) – 5.1.1 still available in chpc.utah.edu

– /uufs/chpc.utah.edu/sys/pkg/nwchem/nwchem-6.1/bin/LINUX64– /uufs/ember.arches/sys/pkg/nwchem/nwchem-6.1/bin/LINUX64 (not tested)– /uufs/updraft.arches/sys/pkg/nwchem/nwchem-6.1/bin/LINUX64 (not tested)

• To run:– source appropriate etc/nwchem.csh– must have $HOME/pdir directory if running parallel

• More information and example batch script at– http://www.chpc.utah.edu/docs/manuals/software/nwchem.html

http://www.chpc.utah.edu

GAMESS• General Atomic and Molecular Electronic Structure System• Another option for most ab initio quantum calculations• On 22 February 2006 version – can be updated if needed• /uufs/chpc.utah.edu/sys/pkg/GAMESS/gamess • http://www.msg.ameslab.gov/GAMESS for information on

usage and capabilities• Can run both parallel or serial• For information on accessing the CHPC installation see

– http://www.chpc.utah.edu/docs/manuals/software/gamess.html

http://www.chpc.utah.edu

Other Quantum Codes-- Mainly codes that are specialized in terms of functionality --• Dalton – only serial build

– Focus on property calculations at HF, DFT, MCSCF and CC levels of theory

– version 2.0 installed; Dalton2011 recently released– http://dirac.chem.sdu.dk/daltonprogram.org/

• Molpro – only serial build– Emphasis on highly accurate calculation methods– Version 2006.1 installed; new 2012.1 available – http://www.molpro.net/

• Psi - http://www.psicode.org/ • Orca - http://www.thch.uni-bonn.de/tc/orca/ • CFOUR - http://www.cfour.de/

http://www.chpc.utah.edu

Plane Wave CodesPlane wave codes for the study of systems under periodic boundary conditions (PBC) • NWChem has some functionality• VASP – http://cms.mpi.univie.ac.at/vasp/

– Ab initio QM/MD – Licensed per research group (have 1 group with an old version)

• Quantum Espresso - http://www.quantum-espresso.org/ – Understands crystal space groups– Has GIPAW module to do NMR calculations– J-ICE and XCrysDen to view

• CPMD - http://www.cpmd.org/ • BigDFT - http://inac.cea.fr/L_Sim/BigDFT/

http://www.chpc.utah.edu

Schrodinger Suite• Commercial package geared for chemical modeling and simulation in

the pharmaceutical field, specifically drug discovery• http://www.schrodinger.com/ • Interface – Maestro (free for academia)• Calculation code includes Jaguar (Quantum), Macromodel (MM),

Qsite (QM/MM)• Interfaces with Desmond for MD (installed)• Tools for structure based drug design

– Docking, ligand design, binding affinities, screening libraries• Year to Year licensing

– Token based, so limited in number of concurrent uses– Purchased by two groups in Medicinal Chemistry– If interested in using – come talk to me

http://www.chpc.utah.edu

Finally…..• Let us know if there is some other package that does

something that our current packages do not; we can look into the possibility of getting it.– Factors: cost, hardware/OS requirements, licensing issues,

usage needs• Any questions – contact me

– anita.orendt@utah.edu – Phone: 801-231-2762– Office: 422 INSCC

http://www.chpc.utah.edu